Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Attestable Audits: Verifiable AI Safety Benchmarks Using Trusted Execution Environments

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Using hardware enclave attestations, a new protocol lets users verify that an AI model interaction is bound to a specific model hash and a published audit result, without exposing weights or benchmark data.

desk verdict A sensible TEE audit protocol with a working prototype, but the quantized-model approximation is load-bearing and unresolved. read the letter →

arxiv 2506.23706 v1 pith:L3NBXDUK submitted 2025-06-30 cs.AI cs.CLcs.CR

Christoph Schnabl , Daniel Hugenroth , Bill Marino , Alastair R. Beresford This is my paper
classification cs.AIcs.CLcs.CR
keywords TrustedexecutionenvironmentattestationAIsafetybenchmarksverifiableauditsconfidentialcomputingmodelquantizationgovernancetransparencylog
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Attestable Audits is a protocol that makes AI-safety benchmarks verifiable without exposing what they run on: it executes audit code and data inside a hardware-protected enclave, a kind of trusted execution environment (TEE), and publishes cryptographic attestations that bind the hashed model, benchmark, and result together. The design then ties every user-facing inference back to that published result, so a user can check that the model answering their prompt is the exact model that received the stated audit score, even though the model weights and the audit dataset remain secret. The authors build a working implementation on AWS Nitro Enclaves using a quantized Llama-3.1-8B model and three standard benchmarks, and report that the enclave route costs about 2.2× the price of a comparable CPU run and 21.7× the price of a GPU run. If the approach holds, it gives regulators and users a concrete way to move AI governance from contract-based assurance to evidence that users can verify for themselves, and it lets closed-weight models be audited on untrusted third-party infrastructure.

What carries the argument

The load-bearing mechanism is the TEE attestation, written in the paper as $A_{in\to out}$: a hardware-signed statement that a freshly booted enclave, described by platform configuration registers, hashed an input (model weights, audit code, audit data, prompt) and produced a hashed output, with the signature verifiable against the hardware vendor's public key. Around this core, the design uses a key-encapsulation mechanism and authenticated encryption so that model providers and auditors upload encrypted assets that only the enclave can decrypt, and a transparency log stores the attestations so users can check that the model hash serving them matches the hash that received audit score R. The sandbox layer around model and audit code prevents untrusted code from corrupting the measurements or the attestation logic.

What would settle it

Score the full-precision model and the audited quantized model on the same benchmark with a held-out adversarial prompt set; the claim fails if the quantized model receives acceptance-level scores (e.g., toxicity below threshold, refusal rate above threshold) on prompts where the full-precision model produces toxic output or a successful jailbreak, since the INFERENCE attestation would then certify the full model's compliance on the strength of the quantized model's result.

Watch

Extended reading notes

Core claim

The paper's central claim is that a TEE attestation can act like a zero-knowledge certificate for AI audits: one signed measurement that binds the hashes of the audit code, the audit data, and the model weights to an aggregate result R lets any user verify they are talking to the exact model that received that audit score, while none of those inputs are revealed. The claim is carried by three protocols — PREPARE, in which a full model M is optionally quantized inside the enclave and an attestation bridges the two hashes; the ATTESTABLE AUDIT, in which encrypted audit code and dataset are combined with the model to produce and publish R; and INFERENCE, in which the serving enclave verifies that its loaded weights match the attested hash before answering a user's prompt and returns a fresh attestation linking the prompt, response, and audit result. The paper demonstrates the scheme's feasibility by running MMLU, XSum, and ToxicChat-style benchmarks inside AWS Nitro Enclaves against a 4-bit Llama-3.1-8B model, and it argues the same machinery extends to any AI system where post-hoc proof of which audited model ran is needed.

Load-bearing premise

The audit is executed on a quantized version of the model, and the paper assumes that the quantized model's audit score is a valid approximation of the full-precision model's safety behavior, with only a small ablation in the appendix to support it.

Editorial extensions

If this is right

  • A user can verify, before sending a prompt, that the serving enclave holds the exact model hash linked to a published audit result R, so a model swap or version mix-up becomes detectable by anyone.
  • Regulators can request and inspect attested audit outcomes without the model provider releasing weights or the auditor releasing benchmark data, and can cross-check results against the transparency log.
  • The protocol prevents an operator from deploying a different model than the one that was audited, since the inference enclave aborts if the loaded weights do not match the attested hash.
  • The same design extends to non-LLM AI systems, such as the operator of a self-driving car demonstrating in court that the exact audited model was driving at a given time.
  • Performance data suggest that GPU-capable enclaves could cut the current 21.7× inference cost gap to roughly 5×, making the approach more practical.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the central claim holds, the quantization bridge in PREPARE is the critical weak spot: the audit's score R is computed on Mq, yet the INFERENCE attestation binds the full model M to R, so a natural extension is to make the M-to-Mq equivalence an attested, periodically re-audited invariant rather than a one-time ablation.
  • An extension the paper discusses in passing but does not formalize is multi-vendor attestation: a user who distrusts one hardware vendor could accept an audit only if independent enclaves from different vendors produce identical attested results, trading stronger integrity against the confidentiality risk that a single broken enclave leaks the secrets.
  • A testable extension beyond the paper is running the audit at serving time on the user's own prompt distribution rather than a fixed benchmark set, so the attestation would certify that the model compiled on the inputs a given user actually submits.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Attestable Audits, a protocol that runs AI-safety benchmarks inside Trusted Execution Environments (TEEs) and publishes cryptographic attestations binding model hashes, audit code and data, and benchmark results. The protocol has three phases: PREPARE, in which a provider can quantize a model inside an enclave; ATTESTABLE AUDIT, in which a benchmark is run against the model inside an enclave and the result is attested; and INFERENCE, in which users receive prompts processed inside an enclave together with an attestation linking the model hash, prompt, output, and audit result. The authors implement a prototype on AWS Nitro Enclaves using Llama-3.1-8B-Instruct and report benchmark scores and cost and latency overheads. The central claim is that users can verify they are interacting with a compliant, audited model without exposing model weights or audit data.

Significance. If the protocol achieved its stated goals, it would be a useful contribution to technical AI governance by providing a concrete, deployable mechanism for third-party verifiable audits of confidential models. The paper's strengths include a clean composition of standard cryptographic primitives (KEM, AEAD) with TEE attestation, a working prototype, reproducible benchmark settings, and an honest reporting of performance overhead. The paper does not provide machine-checked proofs or a full formal security analysis; the main correctness concern is the unproven transfer from an audit of a quantized model to a claim about the full model. The evaluation is a feasibility study rather than an adversarial security evaluation, which is appropriate for a prototype but limits the strength of the security claims.

major comments (3)
  1. [§4 (Protocols), Algorithms 1-3; Appendix A.4.2] The protocol's central correctness claim is not supported as stated. The PREPARE protocol (Algorithm 1) only attests that Mq = QUANTIZE(M), a computational fact; it does not attest any behavioral equivalence. The audit (Algorithm 2) runs on Mq and produces R, but the user-facing INFERENCE protocol (Algorithm 3, line 21) issues AM,p→x,R binding HASH(M), p, x, and R, thereby certifying the full model M. The only bridge is the ablation in Appendix A.4.2, which is run outside the enclave and covers three benchmarks at 500 prompts each; it shows aggregate score differences but cannot rule out adversarial or accidental choices of M for which Q4(M) passes the audit while M is unsafe. A user's attestation could therefore certify an unsafe full model. To make the claimed guarantee that users verify interaction with a compliant AI model sound, the protocol must either audit and serve the same model (Mq), provide a formal or per-model validated equivalence argument, or explicitly re-scope the paper's claims to quantized models.
  2. [§4, Algorithm 2 line 17; Algorithm 3 line 21] The audit attestation is not actually bound to the audit result. Algorithm 2 line 17 calls ATTEST({hMq, hAC+AD}) and omits R from the attested data, even though the protocol text and Algorithm 3 line 21 rely on AMq,AC+AD→R certifying R. As written, a published audit attestation authenticates the model and audit material but not the score, which breaks goal G2 and the user-facing guarantee in Algorithm 3. This appears fixable by including R in the ATTEST call, but it is a load-bearing omission, not merely a typo. A related typo is Algorithm 2 line 11, where the second DECAPSULATE uses c1 instead of c2.
  3. [§3.1 threat model vs. §5 security paragraph] The security goals G1-G6 are argued narratively rather than proven, and the threat model is not reconciled with the security claims. Section 3.1 gives adversaries (A2) capabilities including side-channel attacks and VM rollbacks, but Section 5 states that VM-level enclave isolation and full-memory encryption deny physical attackers (A2) access to data-in-use, without explaining how the cited attacks (Li et al., Schlüter et al., De Meulemeester et al.) are excluded. For a protocol whose contribution is verifiability, the authors should give either game-based security definitions with explicit TEE assumptions or an adversarial evaluation; otherwise the confidence in G3 and G6 rests on informal argument.
minor comments (5)
  1. [§4 and Algorithm 2] The attestation notation is inconsistent: the text uses AMq,AC+AD→R while Algorithm 2 line 20 uses AMq,AC,AD→R; please standardize.
  2. [Abstract and §4] The phrase verify interaction with a compliant AI model overstates the current protocol, since the audit is performed on Mq rather than M; consider rephrasing to a compliant (possibly quantized) model until the gap in major comment 1 is resolved.
  3. [Table 2 and §5] Please clarify the relationship between the reported Toxicity rate in Table 2 and the jailbreak attempts and toxic outputs percentages in the text; the current presentation conflates classifier outputs with refusal rates.
  4. [Appendix A.4.2] The sentence We can observer more variance for the 4 bit model contains a typo, and the metric in Figure 3 is described as cosine BERT-embedded similarity while Table 2 calls it BERT Score; please align terminology.
  5. [Appendix A.4.2] The quantization ablation is described as performed outside enclaves; this important scope limitation should appear in the main evaluation section, not only in the appendix.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the protocol is a construction over standard TEE attestation and cryptographic primitives, with no fitted parameter or self-citation chain doing derivational work.

full rationale

The paper makes no claim of deriving benchmark results from inputs by construction. Attestable Audits is a protocol construction on standard primitives: TEE attestation, KEM, and AEAD, and each protocol step binds exactly the hashes and results it computes inside the enclave. There is no equation in which an output equals an input by definition, and no fitted parameter is renamed as a prediction. The PREPARE-to-INFERENCE chain does rely on the empirical assumption that auditing the quantized model Mq approximates the safety behavior of the full model M; however, the paper states this explicitly as an approximation supported by an ablation in Appendix A.4.2, and the gap is an unresolved soundness/validity assumption rather than a circular reduction. Similarly, Algorithm 2 line 17 omitting R from the signed attestation is an implementation or pseudocode defect, not a definitional identity. The evaluation is self-contained against external GPU and CPU baselines, and benchmark hyperparameters are chosen for feasibility, not derived. Citations are to external work; there is no load-bearing self-citation chain. Overall, no circularity is present.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard hardware trust assumptions, cryptographic primitives, and two paper-specific assumptions: the sandbox's isolation guarantees and the quantized-model equivalence. No entirely new entities are introduced.

free parameters (2)
  • Quantization level (Q4_K_M) for Llama-3.1-8B = 4-bit (Q4_K_M)
    Chosen to fit AWS Nitro Enclave memory constraints; the whole feasibility prototype and the quantized-as-approximation assumption depend on this choice. Not fitted to data, but a hand-picked design parameter.
  • Benchmark sampling hyperparameters (temperature, top-p, n_len, context size) = temp 0.1/0.25/0.3, top-p 0.7/0.75, n_len 256/512, context 4096/8192
    Chosen by hand in Appendix A.2 to "reflect typical workloads"; they influence the reported accuracy, BERT score, and toxicity rates used in the feasibility claim.
assumptions (4)
  • domain assumption The TEE hardware vendor (AWS) is trusted: attestation keys are non-extractable, enclave memory encryption works, and the host cannot tamper with enclave execution.
    The protocol's security goals G1-G6 rely on the integrity of remote attestation from AWS Nitro; the paper relies on vendor trust in Sections 2 and 7.
  • standard math Standard cryptographic primitives (collision-resistant hash, IND-CCA KEM, IND-CCA AEAD) are secure.
    Section 3.2 introduces them and uses them for encryption and attestation binding.
  • domain assumption A sandbox inside the TEE isolates malicious model or audit code so it cannot corrupt attestation generation or TEE logic.
    Section 4 states "The sandbox ensures that malicious code that is part of AC or Mq cannot interfere with the integrity of the overall TEE logic", but no formal isolation proof is given.
  • ad hoc to paper The audit of the quantized model Mq is a valid approximation of the safety behavior of the full model M.
    PREPARE protocol allows the provider to attest Mq as a quantized version of M and use Mq's audit result R for M; the attached ablation (A.4.2) shows small score differences but no formal equivalence. Location: Section 4 PREPARE, Section 7 Discussion.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Attestable Audits: Verifiable AI Safety Benchmarks Using Trusted Execution Environments." pith.science (2026). https://pith.science/paper/L3NBXDUK

@misc{pith2026250623706,
  author       = {Pith},
  title        = {Pith review of: Attestable Audits: Verifiable AI Safety Benchmarks Using Trusted Execution Environments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L3NBXDUK}},
  note         = {Machine review of arXiv:2506.23706}
}
read the original abstract

Benchmarks are important measures to evaluate safety and compliance of AI models at scale. However, they typically do not offer verifiable results and lack confidentiality for model IP and benchmark datasets. We propose Attestable Audits, which run inside Trusted Execution Environments and enable users to verify interaction with a compliant AI model. Our work protects sensitive data even when model provider and auditor do not trust each other. This addresses verification challenges raised in recent AI governance frameworks. We build a prototype demonstrating feasibility on typical audit benchmarks against Llama-3.1.

Figures

Figures reproduced from arXiv: 2506.23706 by the authors.

Figure 1
Figure 1. Overview of the Attestable Audit protocol. A The auditor and regulator agree on audit code AC and dataset AD. B Optionally, the provider prepares a (quantized) version Mq of model M, verifiable via attestation AM→Mq . C In the audit, the auditor loads encrypted AC + AD into a fresh TEE and the provider loads encrypted M. D The audit result R and attestation AMq,AC+AD→R are published to a transparency log L. E The us… view at source ↗
Figure 2
Figure 2. Token distribution for each of the three tasks for the two modes (I) enclave (top), (II) compute-constant (bottom) A.4. Benchmark Feasibility The following section provides additional context on the fea￾sibility of running AI safety benchmarks through Attestable Audits. We argue that the results for each of the three tasks are sound and align with expectations, then dive into detailed timing measurements and token-d… view at source ↗
Figure 3
Figure 3. Cosine similarity scores of XSum BERT embeddings for models quantized to 2-, 4-, and 8-bit Summarization [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: MMLU accuracy scores per model (for valid responses only) and valid response rate [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Toxicity rate by quantization level, measured with a DistilBERT-Base multilingual cased classifier Toxicity [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A TEE-Based Architecture for Confidential and Dependable Process Attestation in Authorship Verification

    cs.CR 2026-02 unverdicted novelty 7.0 of 10

    First TEE-based architecture for continuous process attestation with hardware tamper resistance, tiered assurance levels, Markov-chain dependability modeling, and resilient protocol achieving over 99.5% evidence chain...

  2. LLM Harms: A Taxonomy and Discussion

    cs.CY 2025-12 unverdicted novelty 3.0 of 10

    This paper proposes a taxonomy of LLM harms in five categories and suggests mitigation strategies plus a dynamic auditing system for responsible development.

Reference graph

Works this paper leans on

69 extracted references · 34 canonical work pages · cited by 2 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Common Regulatory Capacity for AI

    Aitken, M., Leslie, D., Ostmann, F., Pratt, J., Margetts, H., and Dorobantu, C. Common Regulatory Capacity for AI . Technical report, The Alan Turing Institute, 2022. URL https://doi.org/10.5281/zenodo.6838946

    work page doi:10.5281/zenodo.6838946 2022

  3. [3]

    AMD Secure Encrypted Virtualization (SEV) , 2025

    AMD. AMD Secure Encrypted Virtualization (SEV) , 2025. https://www.amd.com/en/developer/sev.html. Last accessed April 2025

    work page 2025

  4. [4]

    Anthropic AI Risk Report , 2023

    Anthropic. Anthropic AI Risk Report , 2023. URL https://www-cdn.anthropic.com/1adf000c8f675958c2ee23805d91aaade1cd4613/responsible-scaling-policy.pdf

    work page 2023

  5. [5]

    Private Cloud Compute : A new frontier for AI privacy in the cloud, 2025

    Apple. Private Cloud Compute : A new frontier for AI privacy in the cloud, 2025. https://security.apple.com/blog/private-cloud-compute/. Last accessed April 2025

    work page 2025

  6. [6]

    Confidential computing on NVIDIA h100 GPUs for secure and trustworthy AI , 2023

    Apsey, E., Rogers, P., O'Connor, M., and Nertney, R. Confidential computing on NVIDIA h100 GPUs for secure and trustworthy AI , 2023. https://developer.nvidia.com/blog/confidential-computing-on-h100-gpus\\-for-secure-and-trustworthy-ai/. Last accessed April 2025

    work page 2023

  7. [7]

    Situational Awareness: The Decade Ahead , 2024

    Aschenbrenner, L. Situational Awareness: The Decade Ahead , 2024. Available at https://situational-awareness.ai/

    work page 2024

  8. [8]

    AWS Nitro Enclaves , 2024

    AWS. AWS Nitro Enclaves , 2024. https://aws.amazon.com/ec2/nitro/nitro-enclaves/. Last accessed December 2024

    work page 2024

Show all 69 references
  1. [9]

    MS MARCO: A Human Generated MAchine Reading COmprehension Dataset , 2018

    Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., Rosenberg, M., Song, X., Stoica, A., Tiwary, S., and Wang, T. MS MARCO: A Human Generated MAchine Reading COmprehension Dataset , 2018. URL https://arxiv.org/abs...

    2018 arXiv

  2. [10]

    M., Hendrycks, D., Newman, J., and Nonnecke, B

    Barrett, A. M., Hendrycks, D., Newman, J., and Nonnecke, B. Actionable Guidance for High-Consequence AI Risk Management: Towards Standards Addressing AI Catastrophic Risks , 2023. URL https://arxiv.org/abs/2206.08966

    2023 arXiv

  3. [11]

    Brundage, M. et al. Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims , 2020. URL https://arxiv.org/abs/2004.07213

    2020 arXiv

  4. [12]

    D., Steinke, T., Hayase, J., Cooper, A

    Carlini, N., Paleka, D., Dvijotham, K. D., Steinke, T., Hayase, J., Cooper, A. F., Lee, K., Jagielski, M., Nasr, M., Conmy, A., Yona, I., Wallace, E., Rolnick, D., and Tramèr, F. Stealing Part of a Production Language Model , 2024. URL https://arxiv.org/abs/2403.06634

    2024 arXiv

  5. [13]

    Casper, S., Ezell, C., Siegmann, C., Kolt, N., Curtis, T. L., Bucknall, B., Haupt, A., Wei, K., Scheurer, J., Hobbhahn, M., Sharkey, L., Krishna, S., Von Hagen, M., Alberti, S., Chan, A., Sun, Q., Gerovitch, M., Bau, D., Tegmark, M., Krueger, D., and Hadfield-Menell, D. Black-...

    2024

  6. [14]

    Cen, S. H. and Alur, R. From Transparency to Accountability and Back: A Discussion of Access and Evidence in AI Auditing , 2024. URL https://arxiv.org/abs/2410.04772

    2024 arXiv

  7. [15]

    Harms from Increasingly Agentic Algorithmic Systems

    Chan, A., Salganik, R., Markelius, A., Pang, C., Rajkumar, N., Krasheninnikov, D., Langosco, L., He, Z., Duan, Y., Carroll, M., Lin, M., Mayhew, A., Collins, K., Molamohammadi, M., Burden, J., Zhao, W., Rismani, S., Voudouris, K., Bhatt, U., Weller, A., Krueger, D., and Mahara...

    2023

  8. [16]

    D., Zhao, J., and Koushanfar, F

    Chen, H., Fu, C., Rouhani, B. D., Zhao, J., and Koushanfar, F. DeepAttest: an end-to-end attestation framework for deep neural networks . In Proceedings of the 46th International Symposium on Computer Architecture , ISCA '19, pp.\ 487–498, New York, NY, USA, 2019. Association ...

    2019

  9. [17]

    H., Sun, M., Li, K., Chen, Z., and Wang, X

    Chen, H., Chen, H. H., Sun, M., Li, K., Chen, Z., and Wang, X. A verified confidential computing as a service framework for privacy preservation . In 32nd USENIX Security Symposium (USENIX Security 23) , pp.\ 4733--4750, 2023

    2023

  10. [18]

    N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, J

    Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, J. E., and Stoica, I. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference , 2024. URL https://arxiv.org/abs/2403.04132

    2024 arXiv

  11. [19]

    BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions

    Clark, C., Lee, K., Chang, M., Kwiatkowski, T., Collins, M., and Toutanova, K. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions . CoRR, abs/1905.10044, 2019. URL http://arxiv.org/abs/1905.10044

    1905 arXiv

  12. [20]

    Intel SGX explained

    Costan, V. Intel SGX explained. IACR Cryptol, EPrint Arch, 2016

    2016

  13. [21]

    BadRAM : Practical memory aliasing attacks on trusted execution environments

    De Meulemeester, J., Wilke, L., Oswald, D., Eisenbarth, T., Verbauwhede, I., and Van Bulck, J. BadRAM : Practical memory aliasing attacks on trusted execution environments. In 46th IEEE Symposium on Security and Privacy ( S&P ) , May 2025

    2025

  14. [22]

    Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models , 2024

    Dong, Y., Jiang, X., Liu, H., Jin, Z., Gu, B., Yang, M., and Li, G. Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models , 2024. URL https://arxiv.org/abs/2402.15938

    2024 arXiv

  15. [23]

    Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation , 2025

    Eriksson, M., Purificato, E., Noroozian, A., Vinagre, J., Chaslot, G., Gomez, E., and Fernandez-Llorca, D. Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation , 2025. URL https://arxiv.org/abs/2502.06559

    2025 arXiv

  16. [24]

    OpenAI and Anthropic agree to let U .S

    Field, H. OpenAI and Anthropic agree to let U .S. AI Safety Institute test and evaluate new models . CNBC, August 2024. URL https://www.cnbc.com/2024/08/29/openai-and-anthropic-agree-to-let-us\\-ai-safety-institute-test-models.html. Published 3:01 PM EDT, Updated 6:01 PM EDT

    2024

  17. [25]

    Private LoRA Fine-Tuning of Open-Source LLMs with Homomorphic Encryption

    Frery, J., Bredehoft, R., Klemsa, J., Meyre, A., and Stoian, A. Private LoRA Fine-Tuning of Open-Source LLMs with Homomorphic Encryption . https://arxiv.org/abs/2505.07329, 2025. arXiv:2505.07329

    2025 arXiv

  18. [26]

    Gehman, S., Gururangan, S., Sap, M., Choi, Y., and Smith, N. A. RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models . CoRR, abs/2009.11462, 2020. URL https://arxiv.org/abs/2009.11462

    2009 arXiv

  19. [27]

    llama.cpp: Port of LLaMA model in pure C/C++ , 2023

    Gerganov, G. llama.cpp: Port of LLaMA model in pure C/C++ , 2023. URL https://github.com/ggerganov/llama.cpp

    2023

  20. [28]

    Cheating Automatic LLM Benchmarks: Null Models Achieve High Scores via Fairwashing

    Grollier, X., Kazilsky, Y., et al. Cheating Automatic LLM Benchmarks: Null Models Achieve High Scores via Fairwashing . https://arxiv.org/abs/2410.07137, 2024. arXiv:2410.07137

    2024 arXiv

  21. [29]

    COMPL-AI Framework: A Technical Interpretation and LLM Benchmarking Suite for the EU Artificial Intelligence Act

    Guldimann, P., Spiridonov, A., Staab, R., Jovanović, N., Vero, M., Vechev, V., Gueorguieva, A.-M., Balunović, M., Konstantinov, N., Bielik, P., Tsankov, P., and Vechev, M. COMPL-AI Framework: A Technical Interpretation and LLM Benchmarking Suite for the EU Artificial Intellige...

    arXiv

  22. [30]

    Measuring Massive Multitask Language Understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring Massive Multitask Language Understanding . CoRR, abs/2009.03300, 2020. URL https://arxiv.org/abs/2009.03300

    2009 arXiv

  23. [31]

    M., Kocisk \' y , T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., and Blunsom, P

    Hermann, K. M., Kocisk \' y , T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., and Blunsom, P. Teaching Machines to Read and Comprehend . CoRR, abs/1506.03340, 2015. URL http://arxiv.org/abs/1506.03340

    2015 arXiv

  24. [32]

    Governance of Artificial Intelligence (AI)

    House of Commons . Governance of Artificial Intelligence (AI) . Technical report, House of Commons Science, Innovation and Technology Committee , 2024. URL https://committees.parliament.uk/publications/45145/documents/223578/default/

    2024

  25. [33]

    Intel Trust Domain Extensions (Intel TDX) , 2025

    Intel. Intel Trust Domain Extensions (Intel TDX) , 2025. https://www.intel.com/content/www/us/en/developer/tools/trust-domain-extensions/overview.html. Last accessed April 2025

    2025

  26. [34]

    E., Liang, P., and Narayanan, A

    Kapoor, S., Bommasani, R., Klyman, K., Longpre, S., Ramaswami, A., Cihon, P., Hopkins, A., Bankston, K., Biderman, S., Bogen, M., Chowdhury, R., Engler, A., Henderson, P., Jernite, Y., Lazar, S., Maffulli, S., Nelson, A., Pineau, J., Skowron, A., Song, D., Storchan, V., Zhang,...

    2024 arXiv

  27. [35]

    M., Melis, G., and Grefenstette, E

    Kocisk \' y , T., Schwarz, J., Blunsom, P., Dyer, C., Hermann, K. M., Melis, G., and Grefenstette, E. The NarrativeQA Reading Comprehension Challenge . CoRR, abs/1712.07040, 2017. URL http://arxiv.org/abs/1712.07040

    2017 arXiv

  28. [36]

    M., Uszkoreit, J., Le, Q., and Petrov, S

    Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M.-W., Dai, A. M., Uszkoreit, J., Le, Q., and Petrov, S. Natural questions: A benchmark for questio...

    2019 doi

  29. [37]

    AI Fairness in Practice , 2023

    Leslie, D., Rincón, C., Briggs, M., Perini, A., Jayadeva, S., Borda, A., Bennett, S., Burr, C., Aitken, M., Katell, M., Fischer, C., Wong, J., and Kherroubi Garcia, I. AI Fairness in Practice , 2023. URL https://zenodo.org/doi/10.5281/zenodo.10680527

    2023 doi

  30. [38]

    CIPHERLEAKS : Breaking constant-time cryptography on AMD SEV via the ciphertext side channel

    Li, M., Zhang, Y., Wang, H., Li, K., and Cheng, Y. CIPHERLEAKS : Breaking constant-time cryptography on AMD SEV via the ciphertext side channel. In 30th USENIX Security Symposium (USENIX Security 21) , pp.\ 717--732, 2021

    2021

  31. [39]

    D., Ré, C., Acosta-Navas, D., Hudson, D

    Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., Newman, B., Yuan, B., Yan, B., Zhang, C., Cosgrove, C., Manning, C. D., Ré, C., Acosta-Navas, D., Hudson, D. A., Zelikman, E., Durmus, E., Ladhak, F., Rong, F....

    2023 arXiv

  32. [40]

    ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation , 2023

    Lin, Z., Wang, Z., Tong, Y., Wang, Y., Guo, Y., Wang, Y., and Shang, J. ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation , 2023. URL https://arxiv.org/abs/2310.17389

    2023 arXiv

  33. [41]

    A Safe Harbor for AI Evaluation and Red Teaming , 2024 a

    Longpre, S., Kapoor, S., Klyman, K., Ramaswami, A., Bommasani, R., Blili-Hamelin, B., Huang, Y., Skowron, A., Yong, Z.-X., Kotha, S., Zeng, Y., Shi, W., Yang, X., Southen, R., Robey, A., Chao, P., Yang, D., Jia, R., Kang, D., Pentland, S., Narayanan, A., Liang, P., and Henders...

    2024 arXiv

  34. [42]

    Data Authenticity , Consent , and Provenance for AI Are All Broken : What Will It Take to Fix Them ? An MIT Exploration of Generative AI, mar 27 2024 b

    Longpre, S., Mahari, R., Obeng-Marnu, N., Brannon, W., South, T., Kabbara, J., and Pentland, S. Data Authenticity , Consent , and Provenance for AI Are All Broken : What Will It Take to Fix Them ? An MIT Exploration of Generative AI, mar 27 2024 b . https://mit-genai.pubpub.or...

    2024

  35. [43]

    PPFL: Privacy-preserving Federated Learning with Trusted Execution Environments , 2021

    Mo, F., Haddadi, H., Katevas, K., Marin, E., Perino, D., and Kourtellis, N. PPFL: Privacy-preserving Federated Learning with Trusted Execution Environments , 2021. URL https://arxiv.org/abs/2104.14380

    2021 arXiv

  36. [44]

    Machine learning with confidential computing: A systematization of knowledge

    Mo, F., Tarkhani, Z., and Haddadi, H. Machine learning with confidential computing: A systematization of knowledge . ACM computing surveys, 56 0 (11): 0 1--40, 2024

    2024

  37. [45]

    Auditing of AI: Legal, Ethical and Technical Approaches

    M \"o kander, J. Auditing of AI: Legal, Ethical and Technical Approaches . Digital Society, 2, 2023. URL https://api.semanticscholar.org/CorpusID:265045993

    2023

  38. [46]

    R., and Floridi, L

    M \"o kander, J., Schuett, J., Kirk, H. R., and Floridi, L. Auditing Large Language Models: A Three-Layered Approach . AI and Ethics, 4 0 (4): 0 1085--1115, 2023. doi:10.1007/s43681-023-00289-2

    2023 doi

  39. [47]

    B., and Lapata, M

    Narayan, S., Cohen, S. B., and Lapata, M. Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization , 2018

    2018

  40. [48]

    A., and Alstott, J

    Nevo, S., Lahav, D., Karpur, A., Bar-On, Y., Bradley, H. A., and Alstott, J. Securing AI Model Weights: Preventing Theft and Misuse of Frontier Models. RAND Corporation, Santa Monica, CA, 2024. doi:10.7249/RRA2849-1

    2024 doi

  41. [49]

    Office, C. G. Colorado Governor Signs AI Regulation: A New Era for AI Compliance , 2024. URL https://aminiconant.com/colorado-governor-signs-ai-regulation\\-a-new-era-for-artificial-i\ -compliance/

    2024

  42. [50]

    OpenAI on Advanced AI Risks and Safety , 2024

    OpenAI. OpenAI on Advanced AI Risks and Safety , 2024. URL https://openai.com/global-affairs/our-approach-to-frontier-risk/

    2024

  43. [51]

    How to Audit an AI Model Owned by Someone Else (Part 1)

    OpenMined . How to Audit an AI Model Owned by Someone Else (Part 1) . https://openmined.org/blog/ai-audit-part-1/, November 2023. OpenMined Blog; Last accessed May 2025

    2023

  44. [52]

    Parliament, T. E. and Union, T. C. O. T. E. Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence and amending certain Union legislative acts . http://data.europa.eu/eli/reg/2024/1689/oj, 2024

    2024

  45. [53]

    M., and Bowman, S

    Parrish, A., Chen, A., Nangia, N., Padmakumar, V., Phang, J., Thompson, J., Htut, P. M., and Bowman, S. R. BBQ: A Hand-Built Bias Benchmark for Question Answering , 2022. URL https://arxiv.org/abs/2110.08193

    2022 arXiv

  46. [54]

    and Santos, N

    Pinto, S. and Santos, N. Demystifying ARM TrustZone: A comprehensive survey . ACM computing surveys (CSUR), 51 0 (6): 0 1--36, 2019

    2019

  47. [55]

    D., Smart, A., White, R

    Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., and Barnes, P. Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing . In Proceedings of the 2020 Conference on Fairness, ...

    2020

  48. [56]

    S., Rajkumar, N., Moës, N., Ladish, J., Bau, D., Bricman, P., Guha, N., Newman, J., Bengio, Y., South, T., Pentland, A., Koyejo, S., Kochenderfer, M

    Reuel, A., Bucknall, B., Casper, S., Fist, T., Soder, L., Aarne, O., Hammond, L., Ibrahim, L., Chan, A., Wills, P., Anderljung, M., Garfinkel, B., Heim, L., Trask, A., Mukobi, G., Schaeffer, R., Baker, M., Hooker, S., Solaiman, I., Luccioni, A. S., Rajkumar, N., Moës, N., Ladi...

    2025 arXiv

  49. [57]

    OpenAI may leave the EU if regulations bite - CEO

    Reuters . OpenAI may leave the EU if regulations bite - CEO . Reuters, May 2023. URL https://www.reuters.com/technology/openai-may-leave-eu-if-regulations-\ -ceo-2023-05-24/. Published 5:22 PM EDT, updated 2 years ago

    2023

  50. [58]

    Confidential Computing Proofs: An alternative to cryptographic zero-knowledge

    Russinovich, M., Fournet, C., Zaverucha, G., Benaloh, J., Murdoch, B., and Costa, M. Confidential Computing Proofs: An alternative to cryptographic zero-knowledge . Queue, 22 0 (4): 0 73--100, 2024

    2024

  51. [59]

    Heckler: Breaking confidential VMs with malicious interrupts

    Schl \"u ter, B., Sridhara, S., Kuhne, M., Bertschi, A., and Shinde, S. Heckler: Breaking confidential VMs with malicious interrupts. In USENIX Security , 2024

    2024

  52. [60]

    Trusted Machine Learning Models Unlock Private Inference for Problems Currently Infeasible with Cryptography , 2025

    Shumailov, I., Ramage, D., Meiklejohn, S., Kairouz, P., Hartmann, F., Balle, B., and Bagdasarian, E. Trusted Machine Learning Models Unlock Private Inference for Problems Currently Infeasible with Cryptography , 2025. URL https://arxiv.org/abs/2501.08970

    2025 arXiv

  53. [61]

    The gradient of generative AI release: Methods and considerations

    Solaiman, I. The gradient of generative AI release: Methods and considerations. arXiv preprint arXiv:2302.04844, 2023

    2023 arXiv

  54. [62]

    South, T., Camuto, A., Jain, S., Nguyen, S., Mahari, R., Paquin, C., Morton, J., and Pentland, A. S. Verifiable evaluations of machine learning models using zkSNARKs , 2024. URL https://arxiv.org/abs/2402.02675

    2024 arXiv

  55. [63]

    and Kulkarni, Sandeep\, S

    Sperling, L. and Kulkarni, Sandeep\, S. SONNI: Secure Oblivious Neural Network Inference . https://arxiv.org/abs/2504.18974, 2025. To appear in SECRYPT 2025; arXiv:2504.18974

    2025 arXiv

  56. [64]

    Audit Cards: Contextualizing AI Evaluations , 2025

    Staufer, L., Yang, M., Reuel, A., and Casper, S. Audit Cards: Contextualizing AI Evaluations , 2025. URL https://arxiv.org/abs/2504.13839

    2025 arXiv

  57. [65]

    and Zhang, H

    Sun, H. and Zhang, H. Securely Proving Legitimacy of Training Data and Logic for AI Regulation . Preprint, 2023. URL https://blog.genlaw.org/CameraReady/22.pdf

    2023

  58. [66]

    White House Executive Order on Safe, Secure, and Trustworthy Artificial Intelligence , 2023

    The White House . White House Executive Order on Safe, Secure, and Trustworthy Artificial Intelligence , 2023. URL https://www.whitehouse.gov/briefing-room/presidential-actions/2023/10/30/executive-order-on-the-safe-secure-\ -trustworthy-development-and\\-use-of-artificial-int...

    2023

  59. [67]

    LLaMA: Open and Efficient Foundation Language Models , 2023

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. LLaMA: Open and Efficient Foundation Language Models , 2023. URL https://arxiv.org/abs/2302.13971

    2023 arXiv

  60. [68]

    F., and Ward, F

    van der Weij, T., Hofstätter, F., Jaffe, O., Brown, S. F., and Ward, F. R. AI Sandbagging: Language Models can Strategically Underperform on Evaluations , 2025. URL https://arxiv.org/abs/2406.07358

    2025 arXiv

  61. [69]

    HellaSwag: Can a Machine Really Finish Your Sentence? CoRR, abs/1905.07830, 2019

    Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. HellaSwag: Can a Machine Really Finish Your Sentence? CoRR, abs/1905.07830, 2019. URL http://arxiv.org/abs/1905.07830

    1905 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.