But Can You Use It? Design Recommendations for Differentially Private Interactive Systems

Aaron R. Williams; Claire McKay Bowen; Erika Tyagi; Joshua Snoke; Liudas Panavas

REVIEW 4 major objections 5 minor 13 references

But Can You Use It? Design Recommendations for Differentially Private Interactive Systems

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Differentially private validation servers can become usable by hiding the privacy dial and letting researchers request accuracy instead.

desk verdict A thoughtful synthesis of usable-DP recommendations for federal validation servers, but the load-bearing accuracy-to-privacy translation is explicitly future work. read the letter →

arxiv 2412.11794 v1 pith:SH54AS4I submitted 2024-12-16 cs.HC cs.CRstat.AP

Liudas Panavas , Joshua Snoke , Erika Tyagi , Claire McKay Bowen , Aaron R. Williams This is my paper · ORCID

classification cs.HCcs.CRstat.AP

keywords differentialprivacyvalidationserversinteractivequerysystemsusabilityhuman-computerinteractionfederalstatisticalsystemassuranceutility

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Differentially private interactive systems—validation servers that let researchers query confidential government data—have been studied for years but rarely deployed, and this paper argues the main blocker is not the privacy math but the neglect of usability. The authors propose treating privacy assurance, statistical utility, and system usability as three equally important design goals, and they derive five recommendations for balancing them: synthetic data for exploration, removing privacy-parameter language from the user interface, per-research-question privacy allocation, light human review, and automatically generated output documentation. A worked example architecture shows a policy researcher querying IRS-style data without ever seeing an $\epsilon$ value: she explores on synthetic data, requests a desired accuracy, and receives approved noisy results with publication-ready text. If the recommendations hold, federal agencies could offer a middle tier of data access that researchers can use without differential-privacy expertise, expanding who can do policy research on restricted data.

What carries the argument

The argument is carried by a three-way design frame—privacy assurance, statistical utility, and system usability—used to define success for an interactive query system, and by the five recommendations keyed to that frame (Sections 5.1–5.5). The load-bearing mechanism is the proposal in Section 5.2 to replace user-set privacy parameters with an automatic translation from user-specified accuracy requirements to privacy parameters; the paper acknowledges that this translation still needs methodological work and is the step on which the framework's usability promise depends. The companion infrastructure sketch in Section 6 ties the recommendations together into a concrete front-end-to-review workflow, with a human reviewer deciding whether a proposal's requested accuracy costs too much privacy.

What would settle it

Run a benchmark on a standard query, such as a confidence interval for a population mean or a linear-regression coefficient, and check whether the privacy parameter implied by a user-specified accuracy target can be computed from public metadata alone while still delivering intervals with the requested width and nominal coverage; if common queries fail this test, the remove-privacy-parameter language recommendation collapses.

Watch

Extended reading notes

Core claim

The paper's central claim is that a viable differentially private validation server must be redesigned around the people who will actually use it, and that this is achievable by moving all privacy-budget decisions out of the user's hands and into an administrative pipeline. Concretely, the authors claim that users should explore on synthetic data, request statistics by stating the accuracy they need rather than setting parameters like $\epsilon$ and $\delta$, and submit a short proposal for per-project approval by a human reviewer; outputs should come with automatically computed uncertainty measures and example language for publications. They do not claim to have built and tested the system; they claim that these five recommendations, taken together, resolve the known incompatibilities between DP and statistical practice well enough to justify building such a system, and they supply a reference architecture and a set of user-research questions to test it.

Load-bearing premise

Everything hinges on whether a researcher's requested accuracy level can be translated into a correct differential-privacy parameter without using confidential data and without making the answers too noisy to be useful; the paper itself concedes this translation is not yet reliable.

Editorial extensions

If this is right

Federal statistical agencies could deploy validation servers that researchers use without any training in differential privacy, with privacy parameters handled internally.
Researchers who cannot currently pass full clearance—non-citizens, remote researchers, or those at under-resourced institutions—would gain a middle access tier between public-use files and restricted data.
Publishing noisy outputs with automatically generated uncertainty and example language could reduce the public distrust that followed the 2020 Census's noisy statistics.
Privacy budgeting would shift from fixed per-user caps to per-project approval, removing the 'budget runs out' problem and making the process resemble IRB review.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: if the accuracy-to-privacy translation becomes reliable, the same 'request accuracy, not epsilon' pattern could be applied to non-interactive DP releases, letting agencies report error bounds instead of privacy parameters in data documentation.
Beyond the paper: the per-project review model turns privacy budgeting into a governance process, which suggests a testable prediction—agency adoption will hinge more on reviewer consistency and trust than on any single technical parameter choice.
Beyond the paper: a quick empirical check of the synthetic-data recommendation would compare analysts' error rates on a mock system with and without a synthetic exploration tier; if synthetic data does not reduce invalid queries, the usability benefit would need rethinking.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know this is a position paper, not an empirical study. It argues that interactive DP systems for federal statistical agencies need to prioritize usability — synthetic data for exploration, removing privacy-parameter language, per-project budgets, light human review, and output documentation. The five recommendations are not new individually; they synthesize prior work, including the authors' own. What's new is the explicit framing around three design considerations (privacy assurance, statistical utility, system usability) and a concrete high-level architecture that ties them together. The paper is clearly written, honest about limitations, and gives a fair account of the barriers.

The strongest part is the articulation of why current DP systems fail real users: EDA conflict, parameter setting, fixed budgets, and result interpretation. The recommendations follow logically from those barriers, and the example workflow with Emily is helpful. The authors also outline a credible HCI research agenda (Q1–Q4) to test the recommendations. That's a legitimate contribution to the DP-practice literature.

The main soft spot is exactly what the stress-test flags: Recommendation 5.2 removes privacy-parameter language and lets users request accuracy, which must be translated into an epsilon/delta without looking at confidential data. The paper explicitly says this translation is unresolved. That's not a hidden flaw — they state it — but it means the core usability recommendation is a research program, not a deployable design. The architecture in §6 assumes the translation works. Also, the paper leans heavily on the authors' own prior work (Snoke, Tyagi, Williams), which is fine when the work is relevant, but it gives the synthesis a somewhat self-referential feel. The abstract overstates novelty by claiming prior efforts didn't consider usability, when Sarathy et al. and others clearly did; the paper itself cites them. That is a minor overreach.

Who is this for? People designing or evaluating DP interactive systems, especially at statistical agencies, get a useful checklist and a sensible way to think about trade-offs. It is also a good starting point for HCI researchers looking for open problems. It deserves a serious referee: the arguments are coherent, the literature is well-covered, and the recommendations are actionable even if unvalidated. I'd accept it for peer review with a request to soften the novelty claim and to be more explicit that the accuracy-translation step is the critical open problem.

Send it to review. It's not ready to inform actual deployments, but it's a legitimately useful framework paper.

Referee Report

4 major / 5 minor

Summary. The paper argues that prior work on differentially private (DP) interactive query systems has focused on privacy and statistical accuracy while neglecting system usability, and that this neglect is a key reason why such systems have not been deployed in the federal statistical system. It proposes three design considerations—privacy assurance, statistical utility, and system usability—and derives five recommendations: synthetic data for exploration, removal of privacy parameter language from the user interface, per-research-question privacy allocation, light human review, and output documentation. It then sketches a high-level architecture embodying these recommendations and lists four HCI research questions for future user testing. The paper is explicitly a design framework and research agenda rather than an implemented or empirically validated system.

Significance. If adopted, the framework could provide a common vocabulary and a structured checklist for designers of government validation servers, and it reframes DP deployment as a sociotechnical problem in a useful way. The paper's strengths are its broad synthesis of the DP-usability literature, its explicit identification of unresolved research questions, its concrete architecture sketch, and its transparency about the lack of empirical validation. It does not overclaim to have built or tested a system, and it correctly identifies user research as the necessary next step. The main risk is that the central recommendation—replacing privacy parameters with user-specified accuracy requirements—depends on an accuracy-to-privacy translation that the paper itself concedes is not yet solved.

major comments (4)

[5.2, 6] The accuracy-to-privacy translation is the load-bearing assumption of Recommendations 2, 3, and 4 and of the architecture in Section 6. The paper itself states in Section 5.2 that 'there is still methodological work that needs to be done in the field to ensure accurate transformations from the utility metrics to the privacy parameters is possible.' Yet Section 6 presents the translation as operational ('an algorithm determines the privacy loss parameter required for each query') and the abstract claims the recommendations will make DP interactive systems 'work in practice.' The manuscript should either specify a concrete method for the translation that does not use confidential data (e.g., using public metadata, known sensitivity bounds, and precomputed simulation tables) or explicitly frame the central claim as conditional on the translation being solved. As written, the paper is a research agenda rather than a deployable design, and the framing should match that.
[5.2, Privacy Assurance] The statement 'The privacy assurance depends on the ability to compute the privacy parameters based on the user inputs without using the confidential data' is the crux, but the conditions under which this is possible are never specified. The cited works (Ligett et al., 2017; Rogers et al., 2024) address accuracy-first composition for particular mechanisms, but not a general mapping from user-specified confidence-interval widths or error bounds to (epsilon, delta) for counts, means, quantiles, and regressions. The paper should give at least a concrete example of a mechanism and a calibration procedure that would work without confidential data, or state explicitly that the recommendation applies only to statistics for which such a calibration is already known.
[5.3] The per-research-question allocation recommendation removes the fixed cap on the total privacy budget, but the paper does not analyze how privacy loss is composed across research projects or how the system can continue to 'measure and track the privacy expenditure over time' (Section 3.1) without such a cap. If the same individuals appear in multiple approved projects, the total disclosure risk depends on all projects; without a global cap, the system cannot offer a DP guarantee over the lifetime of the data. The recommendation should either explain how cross-project composition is bounded (e.g., with a global odometer) or explicitly accept that the system provides a weaker, non-compositional assurance.
[5.4] The human review step is proposed as necessary, but the criteria for approval are unspecified beyond the consensus-based process in Q3 (Section 7). The paper should clarify how the reviewer is expected to judge whether a requested accuracy level corresponds to 'too much privacy loss' (Section 5.4), since the privacy parameters are computed by the system and the reviewer is not described as having DP expertise. This is particularly important because Section 5.2 removes privacy parameter language from the user interface, so the only place where the privacy implications are examined is the human review.

minor comments (5)

[Abstract] The sentence 'We propose balancing three design considerations: privacy assurance, statistical utility, and system usability, we develop recommendations...' has a comma splice; it should be split into two sentences or joined with 'and.'
[Title] The title contains 'Y ou' instead of 'You.'
[3.1] The phrase 'three underlying tenants' should be 'three underlying tenets.'
[5.2, Table 1] The Table 1 entry 'Doesn't affect privacy if privacy parameters are pre-computed' is confusing because the recommendation computes privacy parameters from user accuracy requests rather than precomputing them; the table should use consistent language.
[Figure 1] The figure caption is present, but the figure itself is not included in the manuscript text; if it is intended to be an actual figure, it should be inserted.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: the recommendations are design proposals, and the one load-bearing technical step (accuracy-to-privacy translation) is explicitly left as future work.

full rationale

The paper makes no formal derivation claims; it proposes design recommendations and an illustrative architecture. I checked each load-bearing step. (1) The central recommendation to remove privacy-parameter language and let users request accuracy (Section 5.2) does not fit or predict anything; the paper explicitly states 'there is still methodological work that needs to be done in the field to ensure accurate transformations from the utility metrics to the privacy parameters is possible,' so the accuracy-to-privacy translation is presented as an open problem, not as a result derived from inputs. (2) The recommendations (synthetic data, per-project budgets, human review, output documentation) are justified by qualitative arguments and citations to external and self-authored prior work; none reduces by construction to a definition in the paper. (3) Self-citations such as 'we take the approach recommended in Snoke et al. (2024)' and 'currently DP does not provide the means of performing common EDA tasks (Snoke et al., 2024)' are present, but the same claims are also supported by independent sources (Sarathy et al., 2023; Nanayakkara et al., 2024) and are not uniqueness theorems or fitted parameters. No equation in the paper is equivalent to another by construction, and no fitted parameter is renamed as a prediction. Therefore no circular step is exhibited; the low score reflects minor non-load-bearing self-citation only.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The paper's recommendations rest on several untested socio-technical assumptions: that accuracy requests can be converted to privacy parameters, that synthetic data can support exploration without leaking confidential information, that human review can be fair and private, and that agencies would adopt a per-project budget model. None of these are validated in the paper, though the authors flag some as open questions.

assumptions (6)

domain assumption Differential privacy is the appropriate privacy framework for interactive statistical query systems.
Section 2 frames the paper around DP; Section 8 discusses limitations but does not consider alternative formal privacy definitions.
domain assumption User-specified accuracy requirements can be translated into DP privacy parameters without relying on confidential data.
Section 5.2 recommends removing privacy parameters and using accuracy requests, but the paper states this translation remains an open methodological problem.
domain assumption Synthetic data resembling the confidential data can be generated and released without unacceptable privacy loss while supporting query development.
Section 5.1 recommends synthetic data for exploration and notes privacy trade-offs; it assumes the risk can be managed, e.g., by using data from prior years.
domain assumption Human review of research proposals can be fair, privacy-preserving, and administratively feasible.
Section 5.4 recommends a light human review; the paper acknowledges possible privacy leakage and inequity but concludes the process can be designed to be simple and quick.
domain assumption Federal statistical agencies would adopt a system with per-research-project privacy budgets and a human approval process.
Sections 5.3 and 6 describe agencies such as the IRS implementing this; the paper uses an illustrative example but provides no evidence of agency willingness.
domain assumption Users prefer and can reason about accuracy metrics (e.g., standard errors, confidence interval widths) rather than DP parameters.
Section 5.2 and Q2 in Section 7 assume users are familiar with statistical accuracy language, citing prior usability findings that DP parameters confuse users.

how reviews work

0 comments

Cite this review

Pith. "Pith review of But Can You Use It? Design Recommendations for Differentially Private Interactive Systems." pith.science (2026). https://pith.science/paper/SH54AS4I

@misc{pith2026241211794,
  author       = {Pith},
  title        = {Pith review of: But Can You Use It? Design Recommendations for Differentially Private Interactive Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SH54AS4I}},
  note         = {Machine review of arXiv:2412.11794}
}

read the original abstract

Accessing data collected by federal statistical agencies is essential for public policy research and improving evidence-based decision making, such as evaluating the effectiveness of social programs, understanding demographic shifts, or addressing public health challenges. Differentially private interactive systems, or validation servers, can form a crucial part of the data-sharing infrastructure. They may allow researchers to query targeted statistics, providing flexible, efficient access to specific insights, reducing the need for broad data releases and supporting timely, focused research. However, they have not yet been practically implemented. While substantial theoretical work has been conducted on the privacy and accuracy guarantees of differentially private mechanisms, prior efforts have not considered usability as an explicit goal of interactive systems. This work outlines and considers the barriers to developing differentially private interactive systems for informing public policy and offers an alternative way forward. We propose balancing three design considerations: privacy assurance, statistical utility, and system usability, we develop recommendations for making differentially private interactive systems work in practice, we present an example architecture based on these recommendations, and we provide an outline of how to conduct the necessary user-testing. Our work seeks to move the practical development of differentially private interactive systems forward to better aid public policy making and spark future research.

Figures

Figures reproduced from arXiv: 2412.11794 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 9 canonical work pages

[1]

Abowd, J. M. (2018). The us census bureau adopts differential privacy. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 2867–2867. Abowd, J. M., R. Ashmead, R. Cumings-Menon, S. Garfinkel, M. Heineck, C. Heiss, R. Johns, D. Kifer, P. Leclerc, A. Machanavajjhala, et al. (2022). The 2020 census disclos...

work page 2018
[2]

Berghel, D

Adeleye, T., S. Berghel, D. Desfontaines, M. Hay, I. Johnson, C. Lemoisson, A. Machanavajjhala, T. Magerlein, G. Modena, D. Pujol, et al. (2023). Publishing wikipedia usage data with strong privacy guarantees. arXiv preprint arXiv:2308.16298. 3See website to learn more about the CRediT taxonomy, https://credit.niso.org 27 Advisory Committee on Data for Ev...

arXiv 2023
[4]

Hoffmann, T. C., P. P. Glasziou, I. Boutron, R. Milne, R. Perera, D. Moher, D. G. Altman, V. Barbour, H. Macdonald, M. Johnston, et al. (2014). Better reporting of interventions: template for intervention description and replication (tidier) checklist and guide. Bmj

work page 2014
[8]

Oberski, D. L. and F. Kreuter (2020). Differential privacy and social science: An urgent puzzle. Harvard Data Science Review 2(1), 1–21. of Sciences Engineering, N. A., Medicine, et al. (2024). Toward a 21st century national data infrastructure: Managing privacy and confidentiality risks with blended data. Panavas, L., T. Crnovrsanin, J. L. Adams, J. Ullm...

work page 2020
[12]

Seeman, J. and D. Susser (2024). Between privacy and utility: On differential privacy in theory and practice. ACM Journal on Responsible Computing 1(1), 1–18. Shackel, B. (2009). Usability–context, framework, definition, design and evaluation. Interacting with computers 21(5-6), 339–346. Shoemate, M., A. Vyrros, C. McCallum, R. Prasad, P. Durbin, S. Casac...

arXiv 2024
[16]

Subramaniam, S

Rogers, R., S. Subramaniam, S. Peng, D. Durfee, S. Lee, S. K. Kancha, S. Sahay, and P. Ahammad (2021). Linkedin’s audience engagements api: A privacy preserving data analytics system at scale. Journal of Privacy and Confidentiality 11(3). 33 Rogers, R. M., A. Roth, J. Ullman, and S. V adhan (2016). Privacy odometers and filters: Pay-as-you-go composition....

work page 2021
[23]

Harris-Kojetin, B. A., R. M. Groves, N. A. of Sciences Engineering, Medicine, et al. (2017). Protecting privacy and confidentiality while providing access to data for research use. Innovations in Federal Statistics: Combining Data Sources While Protecting Privacy. Hart, S. G. (2006). Nasa-task load index (nasa-tlx); 20 years later. In Proceedings of the h...

work page 2017
[29]

Rogers, R. M., G. Samorodnitsk, S. Z. Wu, and A. Ramdas (2024). Adaptive privacy composition for accuracy-first mechanisms. Advances in Neural Information Processing Systems

work page 2024

Show all 13 references

[30]

McSherry, F. D. (2009). Privacy integrated queries: An extensible platform for privacy-preserving data analysis. In Proceedings of the 2009 Association for Computing Machinery’s Special Interest Group on Management of Data International Conference on Management of Data, pp. 19...

2009 arXiv
[36]

Sarathy, J. (2023). Bridging Sociotechnical Gaps for Privacy-Preserving Data Science. Ph. D. thesis, Harvard University. Sarathy, J., S. Song, A. Haque, T. Schlatter, and S. Vadhan (2023). Don’t look at the data! how differential privacy reconfigures the practices of data scie...

2023
[209]

Yu, R. (2021). Automated consort flow diagram generation by sas® programming. In PharmaSUG 2021 Conference Proceedings. PharmaSUG. 35

2021
[348]

Hotz, V. J., C. R. Bollinger, T. Komarova, C. F. Manski, R. A. Moffitt, D. Nekipelov, A. Sojourner, and B. D. Spencer (2022). Balancing data privacy and usability in the federal statistical system. Proceedings of the National Academy of Sciences 119(31), e2104906119. Johnson, ...

2022 arXiv
[2024]

Drechsler, J. (2023). Differential privacy for government agencies—are we there yet? Journal of the American Statistical Association 118(541), 761–773. Dumas, J. S. and J. E. Fox (2007). Usability testing: Current practice and future directions. In The human-computer interacti...

2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Abowd, J. M. (2018). The us census bureau adopts differential privacy. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 2867–2867. Abowd, J. M., R. Ashmead, R. Cumings-Menon, S. Garfinkel, M. Heineck, C. Heiss, R. Johns, D. Kifer, P. Leclerc, A. Machanavajjhala, et al. (2022). The 2020 census disclos...

work page 2018

[2] [2]

Berghel, D

Adeleye, T., S. Berghel, D. Desfontaines, M. Hay, I. Johnson, C. Lemoisson, A. Machanavajjhala, T. Magerlein, G. Modena, D. Pujol, et al. (2023). Publishing wikipedia usage data with strong privacy guarantees. arXiv preprint arXiv:2308.16298. 3See website to learn more about the CRediT taxonomy, https://credit.niso.org 27 Advisory Committee on Data for Ev...

arXiv 2023

[3] [4]

Hoffmann, T. C., P. P. Glasziou, I. Boutron, R. Milne, R. Perera, D. Moher, D. G. Altman, V. Barbour, H. Macdonald, M. Johnston, et al. (2014). Better reporting of interventions: template for intervention description and replication (tidier) checklist and guide. Bmj

work page 2014

[4] [8]

Oberski, D. L. and F. Kreuter (2020). Differential privacy and social science: An urgent puzzle. Harvard Data Science Review 2(1), 1–21. of Sciences Engineering, N. A., Medicine, et al. (2024). Toward a 21st century national data infrastructure: Managing privacy and confidentiality risks with blended data. Panavas, L., T. Crnovrsanin, J. L. Adams, J. Ullm...

work page 2020

[5] [12]

Seeman, J. and D. Susser (2024). Between privacy and utility: On differential privacy in theory and practice. ACM Journal on Responsible Computing 1(1), 1–18. Shackel, B. (2009). Usability–context, framework, definition, design and evaluation. Interacting with computers 21(5-6), 339–346. Shoemate, M., A. Vyrros, C. McCallum, R. Prasad, P. Durbin, S. Casac...

arXiv 2024

[6] [16]

Subramaniam, S

Rogers, R., S. Subramaniam, S. Peng, D. Durfee, S. Lee, S. K. Kancha, S. Sahay, and P. Ahammad (2021). Linkedin’s audience engagements api: A privacy preserving data analytics system at scale. Journal of Privacy and Confidentiality 11(3). 33 Rogers, R. M., A. Roth, J. Ullman, and S. V adhan (2016). Privacy odometers and filters: Pay-as-you-go composition....

work page 2021

[7] [23]

Harris-Kojetin, B. A., R. M. Groves, N. A. of Sciences Engineering, Medicine, et al. (2017). Protecting privacy and confidentiality while providing access to data for research use. Innovations in Federal Statistics: Combining Data Sources While Protecting Privacy. Hart, S. G. (2006). Nasa-task load index (nasa-tlx); 20 years later. In Proceedings of the h...

work page 2017

[8] [29]

Rogers, R. M., G. Samorodnitsk, S. Z. Wu, and A. Ramdas (2024). Adaptive privacy composition for accuracy-first mechanisms. Advances in Neural Information Processing Systems

work page 2024