Pith. sign in

REVIEW 4 major objections 4 minor 33 references

An automated testing-and-repair loop repaired every defect and certificate rejection it surfaced in the agent-built Axon compiler, updating formal correctness proofs when needed and showing no reward hacking.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-03 16:58 UTC pith:R7WXNMDV

load-bearing objection A genuinely new integration of agent-driven testing, repair, and Lean proof update for a verified compiler, with a real artifact; the central repair claims hold, but the Fcmp repair modified a trusted semantics axiom, so 'verified' stops at the model. the 4 major comments →

arxiv 2607.28928 v1 pith:R7WXNMDV submitted 2026-07-31 cs.SE

Automated Testing and Repair for Verified Compilers Generated by a Coding Agent

classification cs.SE
keywords automated testingverified compilercredible compilationcertificate checkerproof repaircoding agentsreward hackingoperational semantics
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents ACDC, a system that automatically tests and repairs verified compilers. It targets a compiler called Axon, which was itself generated entirely by a coding agent under human supervision, and which mixes verified code, checked optimizations, unverified text interfaces, and a formal specification. ACDC's claim is that the compiler's own structure—certificates, a certificate checker, and a formal operational semantics of the target machine—can be turned into precise testing oracles and repair guides. Applying the system, every surfaced defect or certificate rejection was repaired, several repairs required updating formal correctness proofs, and after a modest number of repairs new random inputs no longer surfaced defects. The paper further claims that neither the compiler nor the repairs show evidence of reward hacking, i.e., of overfitting to the benchmark set that drove development.

Core claim

On its own terms, the discovery is that a verified compiler's credibility machinery can double as a testing and repair infrastructure. Axon's optimizations emit certificates that a verified certificate checker accepts or rejects; ACDC turns rejections into records that name the offending optimization and the failed checker property. The ASM operational semantics, made executable and proved equivalent to the formal one, is used as a reference model against randomly generated assembly programs run on the real machine. The repair agents then use the seeds and diagnostics in these records to reproduce, localize, and fix defects—including, when the operational semantics or certificate checker cha

What carries the argument

Credible-compilation certificates and a verified certificate checker: every optimizing pass emits a certificate intended to prove refinement, and the checker tests 31 properties, producing a rejection record identifying the pass and the failed property. An executable version of the ASM operational semantics, proved equivalent to the non-executable semantics used in the proofs, serves as a reference for comparing model behavior with real machine behavior. Seed-preserving randomized test generators—type-directed random programs, skeletal variants, equivalence-modulo-inputs mutations, random instruction sequences, and print/parse round trips—make defects reproducible for the repair agent, which

Load-bearing premise

The entire correctness and repair chain assumes that the formal model of the target machine's instructions accurately describes what the real hardware does; if that model is wrong, both the verified proofs and the tests can pass while the compiled program misbehaves on an actual machine.

What would settle it

Run the repaired compiler on a fresh random sample outside the development distribution and find either a certificate rejection or a model-versus-machine outcome difference that the repair loop cannot eliminate, or find a machine program whose observable behavior differs from the outcome the repaired compiler's semantics predicts.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Verified compilers that use credible compilation can be tested against their own certificates and semantics, not just by differential compilation.
  • Repairs to verified code need not stop at code: repair agents can update formal correctness proofs when the certificate checker or operational semantics changes, keeping end-to-end guarantees intact.
  • A small number of targeted repairs can clear a coding-agent-generated compiler of every defect that randomized testing surfaces, so such compilers can reach a clean state.
  • Certificate rejections are a rich defect signal: they can indicate unsound transforms, incorrect certificate generation, or an overly tight checker, and each class is repairable automatically.
  • The absence of reward hacking in the compiler and repairs suggests agent-generated verified code can generalize beyond the development benchmark set, though the paper frames this as absence of evidence, not proof.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same recipe—make the formal semantics executable, test it against the real artifact, and feed failing cases back to a repair agent—should transfer to other verified systems with checked components, not only compilers; the key requirement is a formal model that can be turned into an oracle.
  • The 'tight certificate checker' repairs reveal an implicit design policy: when a correct optimization produces a certificate the checker rejects, automation may loosen the checker, fix the certificate generator, or discard the optimization, and the choice is made by cost, not principle. Future systems might want to record and audit such engineering decisions.
  • The no-reward-hacking conclusion is about generalization on newly generated random inputs from the same distribution; testing under a deliberately shifted distribution—different program shapes, larger programs, other architectures—would be a stronger test of the same claim.
  • Because correctness is proved against the ASM model, the ultimate guarantee for real deployments depends on a separate hardware-conformance argument; ACDC's semantics-versus-machine tests are evidence for that correspondence but cannot close the gap for instructions or behaviors not sampled.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents ACDC, an automated testing and repair system for Axon, a verified compiler generated by a coding agent. ACDC exploits Axon's structure — certificates, the certificate checker, the ASM operational semantics, and the unverified text interfaces — to generate targeted defect/rejection records via randomized testing (Csmith-style, SPE, EMI, differential semantics-vs-machine tests, and parser/printer round-trips). It then invokes a coding agent to repair each surfaced defect and, when necessary, update the Lean correctness proofs. The evaluation reports that all surfaced defects and certificate rejections were repaired across three campaigns (ASM semantics/printer, certificate checker, parser/printer), provides per-repair resource consumption, and argues that neither Axon nor the repairs show evidence of reward hacking. A public repository contains code, diffs, defect records, and repair transcripts.

Significance. The paper is timely and useful: it appears to be the first demonstration of automated repair of a verified compiler with mechanical proof updates, and it proposes a concrete methodology for probing reward hacking in agent-generated code. The per-defect case studies, the public repository, and the Lean-checked proof updates are concrete strengths. The main caveats are that the Fcmp repair modifies a trusted axiom (so the proof updates do not establish hardware correctness), the resource tables exclude failed repair attempts, and the reward-hacking evaluation relies on agent-generated tests and agent audits. If these points are addressed, the paper would be a solid contribution to the emerging study of agent-generated verified software.

major comments (4)
  1. [§V-B5 (Fcmp)] The Fcmp repair 'restated the trusted float compare axiom over fcmpFlags' and introduced an opaque fcmpFlags function. Since the original axiom was already wrong, the Lean proof updates only show consistency of the rest of the model with a new axiom; they do not show that the axiom matches the ARM64 machine. The 6–32-instruction differential tests (§IV-A) are bounded and cannot exhaust IEEE edge cases such as NaN payloads, signed zero, or FPCR rounding modes. Please explicitly state that the hardware correctness of the repaired semantics is not machine-checked, and either strengthen validation of fcmpFlags or scope the 'all defects repaired' claim accordingly.
  2. [§V-A, Tables I–IV] The resource columns used for RQ4 exclude 'any earlier failed/escalated attempts' by the repair agent, as the table notes state. Reporting only the successful fix session understates the true cost and gives readers no information about repair reliability. Please report total campaign resources including failed attempts, or at least provide per-defect counts of failed/escalated sessions and their costs, so the RQ4 answer reflects actual consumption.
  3. [§V-B3 vs. Table I] The text for the Shift repair says 'No proof changes — the proofs all go through verbatim with this change,' but Table I lists Proof LOC +2/−2 for Shift. This is an internal contradiction in the reported data. Please clarify whether proof lines changed and, if so, what those changes were.
  4. [§V (RQ5)] The 'no evidence of reward hacking' conclusion rests on unseen random batches, agent-written test scripts, and agent-performed code audits. The paper gives no quantitative summary of the unseen-test evaluation (number of programs, pass rates, distribution choice), and the audits are produced by the same class of agent that wrote the code. Because the repair agent may modify trusted axioms (as in Fcmp), unseen-batch success does not in itself rule out reward hacking of the semantics. Please provide concrete generalization statistics and, if possible, an independent human audit of a sample of repairs.
minor comments (4)
  1. [§I] Typo: 'operational semanatics' should be 'operational semantics'.
  2. [§II] The proof of equivalence between the executable and non-executable ASM operational semantics is part of the trust base; this should be mentioned in the threats-to-validity section.
  3. [§V-A] The 30-minute detect phase, the convergence criteria (one clean round for semantics/parser, eight for certificates), the SPE variant count, and the 20-record duplicate cap are free parameters. The paper does not analyze sensitivity to these choices; please acknowledge this in Section VI.
  4. [Tables I–IV] The definition of Proof LOC as covering 'the propositional checker specification and the correctness proofs' is clear for checker repairs but less so for semantics repairs such as Fcmp, where the proof updates also touch 'semantics mirrors.' Please clarify the scope of Proof LOC for semantics changes.

Circularity Check

0 steps flagged

No significant circularity: ACDC's testing and repair claims are grounded in independent differential testing, unseen-random-batch validation, and machine-checked Lean proofs, with the Fcmp axiom issue being an acknowledged trust anchor rather than a circular reduction.

full rationale

The paper's claimed derivation chain is not circular. The central testing technique for the ASM semantics/printer is differential: it compares the executable operational semantics (proved equivalent to the formal one) against actual ARM64 execution of assembled programs, using independently generated random instruction sequences. Repairs are validated against previously unseen batches with new seeds, and the campaigns only converge after additional clean detect phases (e.g., 4,600 programs with no defects for the semantics/printer campaign; eight consecutive clean rounds for the certificate checker). Proof updates are mechanically checked by Lean, so the certificate-checker repairs are not self-justifying. The reward-hacking evaluation rests primarily on generalization to unseen random inputs, not merely on the agent's own code audits; the audits are supplementary. The Fcmp repair does restate a 'trusted float compare axiom' over an opaque fcmpFlags function, which means the formal correctness theorem is relative to that unverified axiom and does not by itself establish hardware correctness. However, the paper explicitly acknowledges this limitation in Section IV-A ('if the operational semantics does not accurately reflect the semantics of the target machine, the compiled program can generate the wrong outcome') and in the footnote. This is a validity threat, not a circularity: the proof system's conclusion is not defined in terms of the repair's own output, and the differential tests provide independent empirical evidence. Self-citations (e.g., [17] for Axon, credible-compilation references) provide context and public artifacts, and no load-bearing argument reduces to an unverified self-citation. No equation or fitted parameter is renamed as a prediction, and no uniqueness or ansatz is imported solely from the authors' prior work. Therefore the appropriate finding is no significant circularity.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 0 invented entities

The paper introduces no new theoretical entities such as particles or forces; code-level abstractions such as the opaque fcmpFlags function are implementation artifacts with Lean proofs, not postulated entities. The testing and repair results depend on several hand-chosen campaign parameters, the faithfulness of the ASM semantics to hardware, and trust in Lean and the coding agent. The central repair-effectiveness claim is supported by public artifacts and Lean proofs, but the absence-of-reward-hacking claim rests on sampling and informal inspection.

free parameters (7)
  • detect phase duration = 30 minutes
    Chosen by the authors for all campaigns; determines how many programs are tested before each repair phase.
  • certificate-campaign convergence criterion = 8 consecutive clean 30-minute detect rounds
    Hand-picked stopping rule; differs from the one-round criterion used for semantics/parser campaigns, affecting the strength of the 'all rejections eliminated' claim.
  • constant selection probabilities = 1/3 random, 2/3 curated extremal pool
    Test-generation heuristic used by the random ASM/AST generators; influences coverage and the defect mix surfaced.
  • ASM sequence length bounds = 6-32 instructions
    Random test generation parameter chosen to keep simulation and execution tractable.
  • SPE variants per seed = 6
    Number of Skeletal Program Enumeration variants generated per Csmith seed program; drives the T3b test volume.
  • duplicate record cap = 20
    Cap on defect/rejection records with the same error message; affects reported defect counts and repair workload.
  • parser/semantics convergence criterion = 1 clean 30-minute detect round
    Stopping rule for the operational semantics and parser campaigns; shorter than the certificate-checker criterion.
axioms (5)
  • domain assumption The ASM operational semantics accurately models the actual ARM64 machine executed by the generated assembly.
    Paper states in Section II and IV-A that this semantics is unverifiable without a formal hardware model; all correctness proofs and ACDC's semantics-vs-machine testing depend on it.
  • domain assumption The executable ASM operational semantics generated by the coding agent is equivalent to the non-executable semantics in Axon.
    Section II says testing uses this executable version and that equivalence was proved in Lean; the proof is trusted through Lean.
  • domain assumption Lean 4 and its kernel are correct, and the generated proof updates are mechanically sound.
    The credibility of 'verified' claims and the proof-update repairs rests on Lean's correctness (methodology, Section V-A).
  • domain assumption Randomized test generation and the chosen convergence rounds are sufficient evidence that no further defects or reward hacking exist.
    The negative 'no evidence of reward hacking' conclusion is sampling-based; the paper itself lists threats to validity in Section VI.
  • domain assumption Coding-agent repair behavior under Claude Code 4.8 is representative and reproducible across runs.
    Section VI notes that runs with the same agent may produce different results due to agent randomization.

pith-pipeline@v1.3.0-daily-deepseek · 16393 in / 12303 out tokens · 119483 ms · 2026-08-03T16:58:37.626427+00:00 · methodology

0 comments
read the original abstract

We present an agent based automated testing and repair system for verified compilers that contain four kinds of code: verified code, checked code, unverified code, and specification. We present specialized defect detection techniques that exploit the structure present in such compilers. For each surfaced defect the system invokes a coding agent to repair the defect and validate the repair. We evaluate the system on the Axon compiler, a compiler completely generated by a coding agent operating under developer supervision. The compiler was validated during development on a small benchmark set (the Livermore benchmarks), raising the possibility that its coding agent reward hacked the compiler. We also evaluate the possibility that the repair system reward hacked the repairs and find no evidence of reward hacking in either the Axon compiler or the repairs. We present results that characterize the testing and repair effectiveness and discuss repair characteristics.

Figures

Figures reproduced from arXiv: 2607.28928 by Martin Rinard.

Figure 1
Figure 1. Figure 1: Axon compiler structure, annotated with operational semantics and trust-category coloring. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 2 linked inside Pith

  1. [1]

    A survey of compiler testing,

    J. Chen, J. Patra, M. Pradel, Y . Xiong, H. Zhang, D. Hao, and L. Zhang, “A survey of compiler testing,”ACM Computing Surveys, vol. 53, no. 1, 2020

  2. [2]

    Random testing for C and C++ compilers with yarpgen,

    V . Livinskii, D. Babokin, and J. Regehr, “Random testing for C and C++ compilers with yarpgen,”Proc. ACM Program. Lang., vol. 4, no. OOPSLA, 2020

  3. [3]

    Finding and understanding bugs in C compilers,

    X. Yang, Y . Chen, E. Eide, and J. Regehr, “Finding and understanding bugs in C compilers,” inProceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2011, 2011

  4. [4]

    Compiler validation via equivalence mod- ulo inputs,

    V . Le, M. Afshari, and Z. Su, “Compiler validation via equivalence mod- ulo inputs,” inACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’14, 2014

  5. [5]

    Finding deep compiler bugs via guided stochastic program mutation,

    V . Le, C. Sun, and Z. Su, “Finding deep compiler bugs via guided stochastic program mutation,” inProceedings of the 2015 ACM SIG- PLAN International Conference on Object-Oriented Programming, Sys- tems, Languages, and Applications, OOPSLA 2015, 2015

  6. [6]

    Finding compiler bugs via live code mutation,

    C. Sun, V . Le, and Z. Su, “Finding compiler bugs via live code mutation,” inProceedings of the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2016, 2016

  7. [7]

    Crellvm: Verified credible compilation for LLVM,

    J. Kang, Y . Kim, Y . Song, J. Lee, S. Park, M. D. Shin, Y . Kim, S. Cho, J. Choi, C.-K. Hur, and K. Yi, “Crellvm: Verified credible compilation for LLVM,” inProceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), 2018

  8. [8]

    Alive2: bounded translation validation for LLVM,

    N. P. Lopes, J. Lee, C. Hur, Z. Liu, and J. Regehr, “Alive2: bounded translation validation for LLVM,” inPLDI ’21: 42nd ACM SIGPLAN International Conference on Programming Language Design and Im- plementation, 2021

  9. [9]

    A formally verified compiler back-end,

    X. Leroy, “A formally verified compiler back-end,”Journal of Automated Reasoning, vol. 43, no. 4, pp. 363–446, 2009

  10. [10]

    A. W. Appel, R. Dockins, A. Hobor, L. Beringer, J. Dodds, G. Stewart, S. Blazy, and X. Leroy,Program Logics for Certified Compilers. New York, NY , USA: Cambridge University Press, 2014

  11. [11]

    The verified CakeML compiler backend,

    Y . K. Tan, M. O. Myreen, R. Kumar, A. Fox, S. Owens, and M. Nor- rish, “The verified CakeML compiler backend,”Journal of Functional Programming, vol. 29, p. e2, 2019

  12. [12]

    Credible compilation,

    M. C. Rinard, “Credible compilation,” MIT Laboratory for Computer Science, Tech. Rep. MIT-LCS-TR-776, 1999

  13. [13]

    Credible compilation with pointers,

    M. C. Rinard and D. Marinov, “Credible compilation with pointers,” inProceedings of the FLoC Workshop on Run-Time Result Verification (RTRV), 1999

  14. [14]

    Credible compilation,

    D. Marinov, “Credible compilation,” S.M. Thesis, Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, Cambridge, Massachusetts, Sep. 2000. [Online]. Available: [dspace.mit.edu](https://dspace.mit.edu/handle/1721.1/86621)

  15. [15]

    VOC: A translation validator for optimizing compilers,

    L. D. Zuck, A. Pnueli, Y . Fang, and B. Goldberg, “VOC: A translation validator for optimizing compilers,”Electronic Notes in Theoretical Computer Science (ENTCS), vol. 65, no. 2, pp. 2–18, 2002, proceedings of COCV 2002

  16. [16]

    Validating optimizing compilers,

    L. D. Zuck, A. Pnueli, B. Goldberg, C. W. Barrett, Y . Fang, and Y . Hu, “Validating optimizing compilers,”Electronic Notes in Theoretical Com- puter Science (ENTCS), vol. 141, no. 2, pp. 37–53, 2005, proceedings of COCV 2004

  17. [17]

    Testing, credible compilation, and verification in the axon verified compiler in lean and claude code,

    M. Rinard, “Testing, credible compilation, and verification in the axon verified compiler in lean and claude code,” inPrinciples of Agentic Engineering Workshop PAgE ’26, ACM SIGPLAN Conference on Pro- gramming Language Design and Implementation, PLDI ’26, 2026

  18. [18]

    The Livermore Fortran Kernels: A computer test of the numerical performance range,

    F. H. McMahon, “The Livermore Fortran Kernels: A computer test of the numerical performance range,” Lawrence Livermore National Laboratory, Livermore, CA, Tech. Rep. UCRL-53745, Dec. 1986, source distributed via netlib at https://www.netlib.org/benchmark/livermore

  19. [19]

    Is the cure still worse than the disease? test overfitting by llms in automated program repair,

    T. Ahmed, J. Ganhotra, A. Shinnar, and M. Hirzel, “Is the cure still worse than the disease? test overfitting by llms in automated program repair,”CoRR, vol. abs/2511.16858, 2025

  20. [20]

    Impossiblebench: Measuring llms’ propensity of exploiting test cases,

    Z. Zhong, A. Raghunathan, and N. Carlini, “Impossiblebench: Measuring llms’ propensity of exploiting test cases,”CoRR, vol. abs/2510.20270, 2025

  21. [21]

    Reward hacking benchmark: Measuring exploits in LLM agents with tool use,

    K. Thaman, “Reward hacking benchmark: Measuring exploits in LLM agents with tool use,”CoRR, vol. abs/2605.02964, 2026. [Online]. Available: https://doi.org/10.48550/arXiv.2605.02964

  22. [22]

    The trusted computing base of the compcert verified compiler,

    D. Monniaux and S. Boulmé, “The trusted computing base of the compcert verified compiler,” inProgramming Languages and Systems - 31st European Symposium on Programming, ESOP 2022, ser. Lecture Notes in Computer Science, vol. 13240. Springer, 2022

  23. [23]

    Testing a formally verified compiler,

    D. Monniaux, L. Gourdin, S. Boulmé, and O. Lebeltel, “Testing a formally verified compiler,” inTests and Proofs - 17th International Conference, TAP 2023, ser. Lecture Notes in Computer Science, vol. 14066. Springer, 2023

  24. [24]

    Skeletal program enumeration for rigorous compiler testing,

    Q. Zhang, C. Sun, and Z. Su, “Skeletal program enumeration for rigorous compiler testing,” inProceedings of the 38th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2017, 2017

  25. [25]

    Formal verification of a realistic compiler,

    X. Leroy, “Formal verification of a realistic compiler,”Communications of the ACM, vol. 52, no. 7, pp. 107–115, 2009

  26. [26]

    Repairagent: An au- tonomous, llm-based agent for program repair,

    I. Bouzenia, P. T. Devanbu, and M. Pradel, “Repairagent: An au- tonomous, llm-based agent for program repair,” in47th IEEE/ACM International Conference on Software Engineering, ICSE 2025, 2025

  27. [27]

    Autocoderover: Autonomous program improvement,

    Y . Zhang, H. Ruan, Z. Fan, and A. Roychoudhury, “Autocoderover: Autonomous program improvement,” inProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2024, 2024

  28. [28]

    Demystifying llm-based software engineering agents,

    C. S. Xia, Y . Deng, S. Dunn, and L. Zhang, “Demystifying llm-based software engineering agents,”Proc. ACM Softw. Eng., vol. 2, no. FSE, 2025

  29. [29]

    The living review on automated program repair,

    M. Monperrus, “The living review on automated program repair,” HAL/archives-ouvertes.fr, Tech. Rep. hal-01956501, 2018

  30. [30]

    Adapting proof automation to adapt proofs,

    T. Ringer, N. Yazdani, J. Leo, and D. Grossman, “Adapting proof automation to adapt proofs,” inProceedings of the 7th ACM SIGPLAN International Conference on Certified Programs and Proofs, CPP 2018, 2018

  31. [31]

    Baldur: Whole-proof generation and repair with large language models,

    E. First, M. N. Rabe, T. Ringer, and Y . Brun, “Baldur: Whole-proof generation and repair with large language models,” inProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2023, 2023

  32. [32]

    Proof repair across quotient type equivalences,

    C. Viola, M. Fan, and T. Ringer, “Proof repair across quotient type equivalences,”Proc. ACM Program. Lang., vol. 9, 2025

  33. [33]

    Mostly automated proof repair for verified libraries,

    K. Gopinathan, M. Keoliya, and I. Sergey, “Mostly automated proof repair for verified libraries,”Proc. ACM Program. Lang., vol. 7, 2023