Enabling External Scrutiny of AI Systems with Privacy-Enhancing Technologies

Helen Toner; Kendrea Beers

REVIEW 4 major objections 4 minor 21 references

Enabling External Scrutiny of AI Systems with Privacy-Enhancing Technologies

T0 review · 4 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Privacy-enhancing technologies can make external scrutiny of AI systems feasible without exposing user data, model weights, or secret evaluation sets.

desk verdict A clear policy synthesis of two PET-based audit pilots, but the stronger privacy and independence claims outrun the self-reported evidence. read the letter →

arxiv 2502.05219 v1 pith:FBFA32GD submitted 2025-02-05 cs.CY cs.AIcs.CR

Kendrea Beers , Helen Toner This is my paper · ORCID

classification cs.CYcs.AIcs.CR

keywords AIgovernanceexternalscrutinyprivacy-enhancingtechnologiessecureenclavesmutualsecrecyauditingmodelevaluationdifferentialprivacy

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

External scrutiny of AI systems has been blocked by companies' legitimate worries about user privacy, system security, and intellectual property. This paper argues those worries no longer need to be decisive, because privacy-enhancing technologies have matured into end-to-end infrastructure that lets outside researchers run approved analyses without seeing raw user data, model weights, or secret evaluation sets. It supports the argument with two real-world governance pilots: one in which external researchers studied a video platform's recommendation algorithm through a privacy-preserving access tool, and one in which a government safety agency evaluated a frontier model inside a secure enclave while both the model weights and the agency's test dataset remained mutually hidden. If the paper is right, routine external audits of consequential AI systems become technically feasible, and the main remaining task for policymakers is legal: safe harbor for auditors and legal backing for access to systems whose owners decline to cooperate.

What carries the argument

The load-bearing mechanism is a secure enclave, a protected execution environment that can hold an AI model and a sensitive dataset at the same time and emit signed certificates attesting that only computation code approved by both the model owner and the evaluator ran. An open-source Python library orchestrates a workflow in which the researcher proposes questions (and, in the advanced setup, brings code and a private dataset), the model owner approves the computation, and the researcher receives only the outputs. This arrangement realizes 'mutual secrecy': the model owner's weights stay private and the evaluator's dataset, such as a national-security-relevant question set, stays private, while each party retains the power to approve or deny every computation.

What would settle it

A direct test would be to run an adversarial red-team exercise against the enclave setup using deliberately malicious evaluation code or an attempt to forge attestation certificates; a second test would compare the findings of an owner-approved audit with those of an unrestricted audit of the same public model and look for systematic blind spots. Either a successfully forged certificate or an owner-approved audit that misses problems the unrestricted audit finds would refute the paper's central claim.

Watch

Extended reading notes

Core claim

The paper's central claim is that a nonprofit's open-source infrastructure, built around a Python library, combines several privacy-enhancing technologies—secure enclaves, secure multi-party computation, zero-knowledge proofs, federated learning, and differential privacy—into setups that implement a single core workflow: a researcher proposes questions, the model owner approves them, and the researcher receives answers without learning anything else. The simpler setup lets researchers analyze a proprietary dataset held on a secure server through owner-approved code, as demonstrated when external researchers tested whether a video platform's recommender amplified suggestive content and found that it did not disproportionately do so. The more advanced setup uses a secure enclave to host both the model and a secret evaluation dataset simultaneously, producing certificates that only the evaluation code approved by both parties ran; in the second pilot a government evaluator used exactly this arrangement to run a frontier-model safety evaluation while the model owner never saw the test data and the evaluator never saw the weights. The paper's conclusion is that these privacy-preserving technical solutions have succeeded in real-world governance scenarios, so privacy, security, and intellectual-property concerns are no longer conclusive excuses for refusing external access.

Load-bearing premise

The load-bearing premise is that the secure enclave's attestation certificates genuinely guarantee confidentiality and integrity in a real high-stakes audit, and that an audit in which the model owner approves every research question still provides meaningful external scrutiny; if either of these gives way, the paper's privacy and independence claims are overstated.

Editorial extensions

If this is right

External researchers could routinely study deployed systems' real behavior, such as whether a recommender system has a partisan lean or a chatbot produces toxic replies, without seeing raw user logs.
Government agencies could evaluate frontier models against secret test sets that they never reveal, while the model owner's weights remain confidential.
Auditors could operate as private businesses because their audit prompts remain proprietary, and model owners would be unable to 'teach to the test' by training on those prompts.
Future setups that keep researcher code private, add verification, and prevent deletion or swapping of models would further strengthen the audit regime.
The decisive policy moves become legal safe harbor for auditors and legislation that backs researcher access, such as the proposed platform transparency law the paper cites.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A likely next test is scaling the enclave setup from public demonstration assets to truly proprietary frontier models; the paper acknowledges significant engineering work remains, and the trust model of enclave vendors has not yet been stress-tested at that scale.
The owner-approval workflow makes these audits 'supervised scrutiny' rather than fully independent oversight; one could test whether approved research questions systematically avoid adversarial or reputationally sensitive topics by comparing approved audits with unrestricted audits on a public model.
If privacy-preserving access becomes standard, data-sharing regulation may shift from compelling disclosure of raw data to compelling computation-with-approval protocols, which would change what protections regulators need to specify.
The same infrastructure could generalize beyond AI to digital-platform research and cross-border statistics, areas the paper notes are already being piloted, suggesting a broader governance template for private data analysis.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should read this as a policy synthesis, not a technical paper. It compiles two real deployment pilots—OpenMined with Dailymotion and with UK AISI/Anthropic—into a readable argument that PETs now make external audits technically feasible. The descriptive core is accurate as far as I can tell, and the authors are honest that the workflow still gives model owners approval rights. The legal recommendations (safe harbor, mandated compliance) are sensible and appropriately grounded.

The soft spots are real but not fatal to the policy argument. First, all the evidence comes from OpenMined's own blog posts and the Christchurch Call write-up by the same developer community. That is a circularity burden the paper never addresses. Second, the strong conclusion that 'trustworthy privacy-preserving technical solutions have succeeded' is undercut by the details: the UK AISI case used public GPT-2 and public CAMEL data, so it was a trial run with no actual secrets at stake. The stress-test note about enclave attestation is on point—certificates prove what code was loaded, not that a hostile model cannot exfiltrate secret data through its outputs. The paper conflates load-time integrity with runtime confidentiality. That matters if you generalize from these pilots to high-stakes evaluations.

I also think the Dailymotion finding, that the algorithm does not amplify suggestive content, is stated without any effect sizes or statistical analysis. For a governance audience that may be okay, but it is thinner evidence than the prose implies.

Who benefits? Policymakers and governance researchers who want a compact overview of how PETs could fit into audit regimes. It will not teach a security researcher anything new, and the future-directions section is admittedly speculative. The authors are clear thinkers—they identify the owner-approval problem themselves and propose legal backstops. I would not call this a groundbreaking contribution, but it deserves a serious referee because it addresses an important question and could be strengthened with relatively modest revisions: independent security assessment, more methodological detail, and a softened conclusion.

Referee Report

4 major / 4 minor

Summary. The paper argues that privacy-enhancing technologies (PETs), as operationalized by OpenMined in the open-source PySyft library, now make it feasible for external researchers to audit AI systems without learning sensitive user data, proprietary model weights, or secret evaluation datasets. It describes two deployments: a Christchurch Call–Dailymotion collaboration in which external researchers analyzed recommendation-algorithm behavior without seeing impression data, and a UK AI Safety Institute–Anthropic trial in which a secure enclave hosted both a model (GPT-2) and an evaluation dataset (CAMEL Biology) under claimed mutual secrecy. The paper concludes that these technical solutions have succeeded in real-world governance scenarios and that policymakers should now focus on legal empowerment of researchers, such as safe harbor and mandated access.

Significance. If the central claim is substantiated, the paper would describe a meaningful advance in AI governance: routine external scrutiny of consequential AI systems could become technically feasible without forcing companies to expose user data, weights, or audit datasets. The paper is valuable as a descriptive account of deployed PET infrastructure and as a policy-oriented synthesis, and it names concrete organizations and public assets. However, the evidence base is thin and largely self-reported by the developer community: the Dailymotion result is stated without effect sizes or methodology, and the secure-enclave trial used public assets as an exercise rather than sensitive ones. The contribution is therefore primarily programmatic and descriptive rather than a demonstrated or independently verified security result.

major comments (4)

[Sec. 4] The claimed 'mutual secrecy' between Anthropic and UK AISI rests on secure-enclave certificates that 'proved that it only ran the evaluation code that both Anthropic and UK AISI approved.' This conflates load-time integrity with runtime confidentiality: attestation can certify which code was loaded into a genuine enclave, but it does not by itself show that the approved evaluation code prevents a malicious model from exfiltrating the secret dataset through its generated outputs, timing, or other side channels. The paper provides no evidence of output filtering, side-channel mitigation, or formal verification of the evaluation code, so the confidentiality guarantee for the UK AISI biology dataset is unsupported as stated.
[Sec. 3] The Dailymotion case study reports that 'the researchers found that Dailymotion's recommendation algorithms do not disproportionately amplify highly suggestive content,' but the paper gives no effect sizes, statistical analysis, or methodological details behind that finding. Without any quantitative or qualitative analysis of the data, the reader cannot assess whether the finding is robust, which is load-bearing for the general claim that privacy-preserving external audits 'have succeeded in real-world governance scenarios.'
[Secs. 3-4 and refs [1, 19, 20]] The central case-study evidence consists of OpenMined's own blog posts and the Christchurch Call write-up authored by the same developer community. There is no independent security evaluation, no third-party verification of the privacy guarantees, and no adversarial testing of the claimed protections. Case Study 2 itself acknowledges that both the model and the dataset were public assets (GPT-2 and CAMEL Biology), so the exercise is a trial run rather than a demonstration that the setup can keep genuinely sensitive model weights or evaluation data private in a high-stakes setting; the conclusion that 'trustworthy privacy-preserving technical solutions ... have succeeded in real-world governance scenarios' therefore overstates what the paper actually shows.
[Sec. 2] The paper asserts that the described technologies 'enable users to verify their trust in a system's relevant privacy and security properties' and that PETs have reached 'a new level of maturity,' but it provides no explicit threat model, adversarial model, or formal specification of which security properties each setup guarantees. Without these, the paper cannot support the normative conclusion that AI model owners 'can no longer use privacy, security, and IP as conclusive excuses' for denying access, because the extent to which these setups actually protect against malicious models or malicious model owners remains unspecified.

minor comments (4)

[Ref. [19]] The author list in reference [19] contains the name 'Kellye Trask' twice; please remove the duplicate.
[Sec. 4] The statement that 'running the evaluation took under two hours' is given without any hardware specification, enclave type, or dataset size, making the runtime claim difficult to interpret or reproduce.
[Sec. 5] The future-work discussion mentions verification mechanisms to prevent model owners from deleting prediction logs or switching out models, but it does not cite any existing implementation, prototype, or research paper for such mechanisms.
[Sec. 6] The paper would benefit from a dedicated limitations paragraph that explicitly addresses adversarial scenarios in which the model itself tries to leak data, or the model owner uses its approval rights to steer or block research; the current text touches on this in Section 6 but does not integrate it into the technical assessment.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is a descriptive case-study survey; its claims rest on cited primary sources and unsupported security assumptions, not on self-referential derivation.

full rationale

This paper does not present a derived mathematical result or a prediction chain; it is an advocacy-oriented survey of OpenMined's privacy-enhancing technology deployments. The two case studies are reported from cited primary sources: the Christchurch Call write-up [1] and OpenMined's own blog posts [19,20]. Although relying on the implementer's own account for the success of the technology is an evidentiary-independence concern, it is not circularity under the stated criteria: the cited authors are not the authors of the present paper, and the paper does not define its conclusions in terms of those citations. The secure-enclave discussion in Section 4 claims that attestation certificates 'proving that it only ran the evaluation code that both Anthropic and UK AISI approved' establish the necessary privacy properties; this is an unsupported technical assumption about attestation semantics, but it is not a circular reduction because attestation is not defined as equivalent to confidentiality. No equation equates an output to an input by construction, no fitted parameter is renamed as a prediction, and no uniqueness theorem is imported from the authors' own prior work. The paper even acknowledges its limitation that model owners can approve or deny all research questions, which undercuts the strength of the 'external scrutiny' claim without making it circular. Consequently, the honest finding is no significant circularity; any criticism belongs to correctness, security, or evidence quality rather than to circular reasoning.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no free parameters or invented entities. Its central claim rests on domain assumptions about enclave trust, owner approval, and the representativeness of two pilots, plus unverified case-study findings.

assumptions (4)

domain assumption Secure enclaves provide trustworthy attestation and execution isolation for both model weights and evaluation data.
Section 4 relies on the enclave's certificates to prove that only approved code ran; this requires trusting the enclave hardware, attestation service, and absence of side channels, none of which is argued in the paper.
domain assumption A model-owner approval step is compatible with meaningful external scrutiny.
The core workflow in Section 2 has the model owner approve all researcher questions, and Section 6 concedes owners may decline; the paper still counts the pilots as external scrutiny.
domain assumption Results from a GPT-2 trial with a public biology dataset generalize to frontier models and secret evaluation data.
Section 4 explicitly calls the exercise a trial run and says scaling requires significant engineering, yet the conclusion treats technical feasibility as established.
domain assumption The Dailymotion audit methodology was adequate to support the finding that the recommender does not amplify suggestive content.
Section 3 states the finding with no description of the statistical analysis or validation, so the claim rests on an unshown methodological assumption.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Enabling External Scrutiny of AI Systems with Privacy-Enhancing Technologies." pith.science (2026). https://pith.science/paper/FBFA32GD

@misc{pith2026250205219,
  author       = {Pith},
  title        = {Pith review of: Enabling External Scrutiny of AI Systems with Privacy-Enhancing Technologies},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FBFA32GD}},
  note         = {Machine review of arXiv:2502.05219}
}

read the original abstract

This article describes how technical infrastructure developed by the nonprofit OpenMined enables external scrutiny of AI systems without compromising sensitive information. Independent external scrutiny of AI systems provides crucial transparency into AI development, so it should be an integral component of any approach to AI governance. In practice, external researchers have struggled to gain access to AI systems because of AI companies' legitimate concerns about security, privacy, and intellectual property. But now, privacy-enhancing technologies (PETs) have reached a new level of maturity: end-to-end technical infrastructure developed by OpenMined combines several PETs into various setups that enable privacy-preserving audits of AI systems. We showcase two case studies where this infrastructure has been deployed in real-world governance scenarios: "Understanding Social Media Recommendation Algorithms with the Christchurch Call" and "Evaluating Frontier Models with the UK AI Safety Institute." We describe types of scrutiny of AI systems that could be facilitated by current setups and OpenMined's proposed future setups. We conclude that these innovative approaches deserve further exploration and support from the AI governance community. Interested policymakers can focus on empowering researchers on a legal level.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 18 canonical work pages

[1]

AI trans- parency in practice, October 2024

Jiahao Chen, Jack Bandy, Dave Buckley, and Ruchi Bhatia. AI trans- parency in practice, October 2024. URL https://www.christchurchcall.org/ safe-secure-private-research-finds-third-parties-can-audit-online-algorithms/

work page 2024
[2]

Christchurch Call initiative on algorithmic outcomes, September 2022

Christchurch Call. Christchurch Call initiative on algorithmic outcomes, September 2022. URL https://www.christchurchcall.org/christchurch-call-initiative-on-algorithmic-outcomes/

work page 2022
[3]

UK pushes for greater access to AI’s inner workings to assess risks

Cristina Criddle. UK pushes for greater access to AI’s inner workings to assess risks. Financial Times, September 2023. URL www.ft.com/content/4427689c-b98e-483e-bde3-2d82808a266f

work page arXiv 2023
[4]

OpenAI GPT2, 2019

Hugging Face. OpenAI GPT2, 2019. URL https://huggingface.co/docs/transformers/model doc/gpt2

work page 2019
[5]

Our plans for researchers on Reddit, May 2024

KeyserSosa. Our plans for researchers on Reddit, May 2024. URL www.reddit.com/r/ reddit4researchers/comments/1co0mqa/our plans for researchers on reddit/

work page 2024
[6]

Camel: communicative agents for “mind” exploration of large scale language model society, May 2023

Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: communicative agents for “mind” exploration of large scale language model society, May 2023. URL https://huggingface.co/datasets/camel-ai/biology

work page 2023
[7]

A safe harbor for AI evaluation and red teaming, March 2024

Shayne Longpre, Sayash Kapoor, Kevin Klyman, Ashwin Ramaswami, Rishi Bommasani, Borhane Blili-Hamelin, Yangsibo Huang, Aviya Skowron, Zheng-Xin Yong, Suhas Kotha, Yi Zeng, Weiyan Shi, Xianjun Yang, Reid Southen, Alexander Robey, Patrick Chao, Diyi Yang, Ruoxi Jia, Daniel Kang, Sandy Pentland, Arvind Narayanan, Percy Liang, and Peter Henderson. A safe harb...

arXiv 2024
[8]

What is federated learning?, May 2020

Antonio Lopardo. What is federated learning?, May 2020. URL https://blog.openmined.org/ what-is-federated-learning/

work page 2020

Show all 21 references

[9]

What is secure multi-party computation?, May 2020

Antonio Lopardo, Ayoub Benaissa, and Th´ eo Ryffel. What is secure multi-party computation?, May 2020. URL https://blog.openmined.org/what-is-secure-multi-party-computation/

2020
[10]

Ho, Dorothy Chou, Helen Toner, Inioluwa Deborah Raji, Irene Solaiman, James W

Aaron Maniam, Alondra Nelson, Ben Garfinkel, Brian Christian, Daniel E. Ho, Dorothy Chou, Helen Toner, Inioluwa Deborah Raji, Irene Solaiman, James W. Phillips, Karine Perset, Marc Aidinoff, Matthew Botvinick, Matthew J. Salganik, Rumman Chowdhury, Samuel R. Bowman, Sebastien ...

2023
[11]

A new model for international, privacy-preserving data science

Curtis Mitchell. A new model for international, privacy-preserving data science. 2024 USENIX Conference on Privacy Engineering Practice and Respect, June 2024. URL www.usenix.net/ conference/pepr24/presentation/mitchell

2024
[12]

Senator Christopher Coons of Delaware

Office of U.S. Senator Christopher Coons of Delaware. Coons, Portman, Klobuchar announce legislation to ensure transparency at social media plat- forms, December 2021. URL www.coons.senate.gov/news/press-releases/ coons-portman-klobuchar-announce-legislation-to-ensure-transpar...

2021
[13]

Introduction to differential privacy, 2020

OpenMined. Introduction to differential privacy, 2020. URL https://openmined.github.io/ PyDP/introduction.html

2020
[14]

PySyft’s documentation, 2021

OpenMined. PySyft’s documentation, 2021. URL https://openmined.github.io/PySyft/

2021
[15]

Inioluwa Deborah Raji, Peggy Xu, Colleen Honigsberg, and Daniel E. Ho. Outsider oversight: designing a third party audit ecosystem for AI governance, June 2022. URL https://arxiv.org/ abs/2206.04737

2022 arXiv
[16]

Srishilesh P. S. and Adam J. Hall. What are zero knowledge proofs?, October 2020. URL https://blog.openmined.org/zero-knowledge-proof/

2020
[17]

Model evaluation for extreme risks, September

Toby Shevlane, Sebastian Farquhar, Ben Garfinkel, Mary Phuong, Jess Whittlestone, Jade Leung, Daniel Kokotajlo, Nahema Marchal, Markus Anderljung, Noam Kolt, Lewis Ho, Divya Siddarth, Shahar Avin, Will Hawkins, Been Kim, Iason Gabriel, Vijay Bolina, Jack Clark, Yoshua Bengio, ...
[18]

Lacey Strahm. Response to the National Telecommunications and Information Administra- tion (NTIA) request for comment (RFC) on dual use foundation artificial intelligence (AI) models with widely available model weights, 2023. URL https://downloads.regulations.gov/ NTIA-2023-00...

2023
[19]

How to audit an AI model owned by someone else (Part 1), June 2023

Andrew Trask, Akshay Sukumar, Antti Kalliokoski, Bennett Farkas, Callis Ezenwaka, Carmen Popa, Curtis Mitchell, Dylan Hrebenach, George-Cristian Muraru, Ionesio Junior, Irina Bejan, Ishan Mishra, Ivoline Ngong, Jack Bandy, Jess Stahl, Julian Cardonnet, Kellye Trask, Kellye Tra...

2023
[20]

Secure enclaves for AI evaluation, November 2024

Andrew Trask, Aziz Berkay Yesilyurt, Bennett Farkas, Callis Ezenwaka, Carmen Popa, Dave Buckley, Eelco van der Wel, Francesco Mosconi, Grace Han, Ionesio Junior, Irina Bejan, Ishan Mishra, Khoa Nguyen, Koen van der Veen, Kyoko Eng, Lacey Strahm, Logan Graham, Madhava Jay, Mate...

2024
[2023]

URL https://arxiv.org/abs/2305.15324

arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

AI trans- parency in practice, October 2024

Jiahao Chen, Jack Bandy, Dave Buckley, and Ruchi Bhatia. AI trans- parency in practice, October 2024. URL https://www.christchurchcall.org/ safe-secure-private-research-finds-third-parties-can-audit-online-algorithms/

work page 2024

[2] [2]

Christchurch Call initiative on algorithmic outcomes, September 2022

Christchurch Call. Christchurch Call initiative on algorithmic outcomes, September 2022. URL https://www.christchurchcall.org/christchurch-call-initiative-on-algorithmic-outcomes/

work page 2022

[3] [3]

UK pushes for greater access to AI’s inner workings to assess risks

Cristina Criddle. UK pushes for greater access to AI’s inner workings to assess risks. Financial Times, September 2023. URL www.ft.com/content/4427689c-b98e-483e-bde3-2d82808a266f

work page arXiv 2023

[4] [4]

OpenAI GPT2, 2019

Hugging Face. OpenAI GPT2, 2019. URL https://huggingface.co/docs/transformers/model doc/gpt2

work page 2019

[5] [5]

Our plans for researchers on Reddit, May 2024

KeyserSosa. Our plans for researchers on Reddit, May 2024. URL www.reddit.com/r/ reddit4researchers/comments/1co0mqa/our plans for researchers on reddit/

work page 2024

[6] [6]

Camel: communicative agents for “mind” exploration of large scale language model society, May 2023

Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: communicative agents for “mind” exploration of large scale language model society, May 2023. URL https://huggingface.co/datasets/camel-ai/biology

work page 2023

[7] [7]

A safe harbor for AI evaluation and red teaming, March 2024

Shayne Longpre, Sayash Kapoor, Kevin Klyman, Ashwin Ramaswami, Rishi Bommasani, Borhane Blili-Hamelin, Yangsibo Huang, Aviya Skowron, Zheng-Xin Yong, Suhas Kotha, Yi Zeng, Weiyan Shi, Xianjun Yang, Reid Southen, Alexander Robey, Patrick Chao, Diyi Yang, Ruoxi Jia, Daniel Kang, Sandy Pentland, Arvind Narayanan, Percy Liang, and Peter Henderson. A safe harb...

arXiv 2024

[8] [8]

What is federated learning?, May 2020

Antonio Lopardo. What is federated learning?, May 2020. URL https://blog.openmined.org/ what-is-federated-learning/

work page 2020