Pith. sign in

REVIEW 2 major objections 22 references

ACES: Who Tests the Tests? Leave-One-Out AUC Consistency for Code Generation

T0 review · 2 major / 0 minor · reviewed 2026-07-14 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Test votes should rank, not merely count: leave-one-out AUC identifies which generated tests distinguish correct code from incorrect code without knowing either label.

desk verdict Clean LOO-AUC identity that turns the binary pass matrix into principled test weights; theory and Pass@k gains both look solid. read the letter →

arxiv 2604.03922 v2 pith:QD35AOPB submitted 2026-04-05 cs.LG

Hui Sun , Yun-Ji Zhang , Zheng Xie , Ren-Biao Liu , Yali Du , Xin-Ye Li , Ming Li This is my paper
classification cs.LG
keywords codegenerationLLMtestselectionleave-one-outAUCPass@kweightedvotingdiscriminativepowerbinarypassmatrix
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Choosing among many LLM-generated programs by running them on LLM-generated tests is circular: you need good tests to judge code and good code to judge tests. This paper claims you never need to decide which tests are correct. A test is valuable for selection only if it separates correct programs from incorrect ones. By holding out each test, ranking candidates from the remaining tests, and measuring agreement with the held-out pass/fail column (leave-one-out AUC), the authors prove that this observable agreement is proportional to the test’s latent discriminative power. From that identity they build two lightweight weighting schemes that use only the binary pass matrix and raise Pass@k on standard code-generation benchmarks.

What carries the argument

The LOO-AUC Identity (Theorem 3): expected leave-one-out AUC excess equals a positive coefficient times the test’s latent discriminative power. The coefficient depends only on pass-rate variance and leave-one-out ranking quality, so the identity converts an unobservable quality into a measurable consistency score that drives both closed-form and optimized weights.

What would settle it

On problems with known correct codes, compute each test’s true discriminative power; if the sign of LOO-AUC excess systematically disagrees with that sign whenever average discriminative power is positive, the central identity is false.

Watch

Extended reading notes

Core claim

The expected leave-one-out AUC of a test, measured against the ranking induced by all other tests, is proportional to that test’s discriminative power—the difference in pass rates between correct and incorrect codes. Therefore tests can be weighted by an observable consistency score that recovers the sign of their true ranking value in expectation, breaking the circular dependency without external labels.

Load-bearing premise

The closed-form guarantees need the average test to be slightly better than random at separating correct from incorrect code; if the whole pool is worse than chance on average, the simple correction no longer works.

Editorial extensions

If this is right

  • Closed-form ACES-C weights recover the sign of each test’s discriminative power in expectation and approach oracle signal-to-noise under the average-quality condition.
  • Iterative ACES-O can still improve rankings when average test quality is low by co-evolving weights and the leave-one-out ranking.
  • Uniform majority voting is improved by reweighting tests with LOO-AUC consistency using nothing beyond the binary pass matrix.
  • Combining the same scores with complementary static-analysis signals further raises Pass@k beyond either signal alone.
  • The authors argue the same internal-consistency principle extends to other noisy-evaluator settings such as LLM-as-judge ensembles and process reward models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same leave-one-out consistency score could reweight noisy generated verifiers in math or multi-step reasoning, not only code.
  • When LLM-generated tests are highly correlated, the independence assumptions behind the expectation identity loosen, suggesting a direct experiment on de-correlated test suites.
  • Pre-filtering candidates before optimization acts as a curriculum for the ranking objective; similar staged weighting may help other multi-annotator ranking problems.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 0 minor

Summary. The paper addresses the circular dependency in selecting LLM-generated code with LLM-generated tests: reliable tests are needed to judge code, and reliable code to judge tests. The key claim is that test value for ranking is discriminative power δ_j = α_j − β_j, not correctness per se. They define leave-one-out AUC (LOO-AUC) from the binary pass matrix alone, prove E[LOO-AUC_j(w)] − 1/2 = c_j(w)·δ_j (Theorem 3), and derive two methods: ACES-C (closed-form pass-rate-corrected weights under Assumption 4, with oracle-approximation guarantees) and ACES-O (iterative logistic-surrogate optimization of a LOO-AUC objective without that assumption). Both use only the pass matrix. Empirically they report state-of-the-art Pass@k among execution-only methods on HumanEval, HumanEval+, and MBPP, with further gains when combined with DS3 static analysis.

Significance. If the results hold, this is a genuine advance: a first provable, pass-matrix-only criterion for separating informative from misleading tests, with closed-form and optimization variants that are cheap and complementary. The LOO-AUC identity is clean and useful beyond code generation (noisy evaluators, LLM-as-judge ensembles). Strengths include complete proofs (Appendix A), quantitative oracle approximation (Corollary 8), explicit isolation of Assumption 4 and its failure regime, thorough ablations (selection vs weighting, sensitivity to m′/n′/K, convergence, hyperparameters), and reproducible comparison on shared GPT-3.5-Turbo candidates/tests. The work is both theoretically grounded and practically deployable.

major comments (2)
  1. Section 2.2 and Theorem 3 rest on conditional independence of test outcomes given correctness labels. That is standard IRT-style modeling and is stated clearly, but LLM-generated tests are often template-correlated and share failure modes. The identity remains valid under the model, yet the practical reliability of LOO-AUC as a proxy for δ_j depends on how much residual dependence remains in real pass matrices. Please quantify pairwise column correlations (or conditional mutual information) on the experimental pass matrices and, if non-negligible, discuss how dependence would bias LOO-AUC or the recovered weights. The conclusion flags this as future work; a short empirical check in §4.3 or Appendix C would make the central claim more robust for practitioners.
  2. Assumption 4 (δ̄ > 2√(ln 2/m)) is the only extra fragility for ACES-C (Proposition 5, Theorem 6). Table 7 and Figures 3/6–8 show it holds on 71–83% of non-trivial tasks and that the Hard region (low δ̄) is essentially unsolved by any pass-matrix method. That isolation is honest, but the main text still presents ACES as generally SOTA among execution-only methods. Please state more explicitly in §4.2–4.3 (and perhaps the abstract) the practical decision rule: when estimated average quality is near or below the threshold, prefer ACES-O or complementary static signals (DS3) rather than ACES-C alone. This is load-bearing for correct use of the closed-form guarantee.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: LOO-AUC is an observable computed only from the pass matrix; the identity links it to latent discriminative power without assuming that power.

full rationale

The paper’s central move is to break the code–test circular dependency, and the derivation does so cleanly. LOO-AUC_j(w) is defined solely from the binary pass matrix B (leave-one-out scores S^(−j) and column B:,j); the latent labels y and the latent δ_j never enter the algorithm. Theorem 3 is a standard expectation identity under an explicit generative model (i.i.d. codes, class-conditional pass rates, conditional independence of tests given labels): E[LOO-AUC_j]−1/2 = c_j(w)·δ_j. ACES-C then multiplies the observable LOO-AUC excess by the empirical pass-rate factor p_j(1−p_j) so that E[q_j]∝δ_j under Assumption 4; that is estimation of a latent parameter, not a quantity forced by construction from a fit of the same quantity. ACES-O maximizes a differentiable surrogate of the same observable objective. There is no self-definitional loop, no fitted constant later re-labeled as a prediction, no load-bearing uniqueness theorem imported from the authors, and no renaming of a known empirical pattern as a first-principles result. Classical citations (Hoeffding, RankBoost, AUC consistency, item-response theory) supply standard tools, not circular premises. Assumption 4 is an explicit, falsifiable average-quality condition whose empirical failure regime is isolated and handled by ACES-O; it is not a circular definition. Score 0 is therefore appropriate.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The central claim rests on a standard conditional-independence model of binary outcomes, one mild average-quality inequality (Assumption 4), and ordinary AUC/ROC definitions. No free parameters are fitted to obtain the identity itself; ACES-O hyperparameters affect only the practical optimiser. No new physical or mathematical entities are postulated.

free parameters (3)
  • ACES-O surrogate sharpness γ
    Controls tightness of the logistic approximation to the step function; set by hand (default 10) and shown to be stable for γ ≥ 5.
  • ACES-O learning rate η and step count T
    Optimiser schedule; defaults given per benchmark and shown stable over wide ranges.
  • Pre-filtering cutoff K
    Number of majority-vote top candidates retained before ACES-O optimisation; chosen for speed/quality trade-off.
assumptions (4)
  • domain assumption Conditioned on correctness label y_i, a code’s outcomes on distinct tests are i.i.d. with class-conditional rates α_j, β_j (classical IRT-style conditional independence).
    Stated in §2.2; used for every expectation identity and concentration bound.
  • ad hoc to paper Assumption 4: average discriminative power δ̄ > 2√(ln 2 / m).
    Introduced in §3.1 solely to guarantee that every leave-one-out ranking under uniform weights already beats chance; empirically verified but not universally true.
  • domain assumption Codes and tests are sampled independently from the LLM.
    Stated in §2.2; justifies treating the pass matrix columns as exchangeable given labels.
  • standard math Standard probabilistic definition of AUC and Hoeffding’s inequality.
    Used throughout the proofs in Appendix A.
invented entities (1)
  • Leave-one-out AUC (LOO-AUC_j) independent evidence
    purpose: Observable surrogate for the latent discriminative power δ_j of each test.
    Defined from ordinary AUC applied to leave-one-out scores; not a new physical object, but a new derived statistic whose expectation identity is the paper’s core contribution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ACES: Who Tests the Tests? Leave-One-Out AUC Consistency for Code Generation." pith.science (2026). https://pith.science/paper/QD35AOPB

@misc{pith2026260403922,
  author       = {Pith},
  title        = {Pith review of: ACES: Who Tests the Tests? Leave-One-Out AUC Consistency for Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QD35AOPB}},
  note         = {Machine review of arXiv:2604.03922}
}
abstract

Selecting LLM-generated code candidates using LLM-generated tests is challenging because the tests themselves may be incorrect. Existing methods either treat all tests equally or rely on ad-hoc heuristics to filter unreliable tests. Yet determining test correctness requires knowing which codes are correct, creating a \emph{circular dependency}. Our key insight is that we need not determine test correctness at all: \emph{test votes should rank, not merely count}. What matters is not how many codes pass a test, but whether the test can \emph{distinguish} correct from incorrect code. We break the circular dependency via leave-one-out evaluation: hold out one test, rank codes by their aggregate scores on all remaining tests, and measure whether the held-out test's pass/fail pattern agrees with this ranking. We formalize this agreement as the leave-one-out AUC~(LOO-AUC) and prove that the expected LOO-AUC is proportional to each test's ability to separate correct code from incorrect code. Building on this, we propose \textbf{ACES}~(\textbf{A}UC \textbf{C}onsist\textbf{E}ncy \textbf{S}coring) with two complementary variants: ACES-C provides closed-form weights that provably approximate the oracle in expectation under a mild assumption on average test quality; ACES-O drops this assumption and iteratively optimizes a differentiable LOO-AUC objective. Both operate solely on the binary pass matrix with negligible overhead, and achieve state-of-the-art Pass@$k$ on multiple code generation benchmarks.

Review history (2 revisions) →

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

22 extracted references · 12 linked inside Pith

  1. [1]

    Evaluating large language models trained on code.Arxiv Preprint Arxiv:2107.03374,

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code.Arxiv Preprint Arxiv:2107.03374,

    arXiv

  2. [2]

    DeepSeek-Coder: When the large language model meets programming–the rise of code intelligence.Arxiv Preprint Arxiv:2401.14196,

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yifan Wu, YK Li, et al. DeepSeek-Coder: When the large language model meets programming–the rise of code intelligence.Arxiv Preprint Arxiv:2401.14196,

    arXiv

  3. [3]

    Large language monkeys: Scaling inference compute with repeated sampling.Arxiv Preprint Arxiv:2407.21787,

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling.Arxiv Preprint Arxiv:2407.21787,

    arXiv

  4. [4]

    10 ACES: LOO-AUC Consistency for Code GenerationA PREPRINT Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida I. Wang. Natural language to code translation with execution. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3533–3546. Association for Computational Linguistics,

    2022

  5. [5]

    Hung Quoc To, Minh Huynh Nguyen, and Nghi D. Q. Bui. Functional overlap reranking for neural code genera- tion. InFindings of the Association for Computational Linguistics: Acl 2024, pages 3686–3704. Association for Computational Linguistics,

    2024

  6. [6]

    Program synthesis with large language models.Arxiv Preprint Arxiv:2108.07732,

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models.Arxiv Preprint Arxiv:2108.07732,

    arXiv

  7. [7]

    GPT-4 technical report.Arxiv Preprint Arxiv:2303.08774,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report.Arxiv Preprint Arxiv:2303.08774,

    arXiv

  8. [8]

    Code llama: Open foundation models for code.Arxiv Preprint Arxiv:2308.12950,

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code.Arxiv Preprint Arxiv:2308.12950,

    arXiv

Show all 22 references
  1. [9]

    Starcoder 2 and the stack v2: The next generation.Arxiv Preprint Arxiv:2402.19173,

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder 2 and the stack v2: The next generation.Arxiv Preprint Arxiv:2402.19173,

    arXiv

  2. [10]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-Coder technical report.Arxiv Preprint Arxiv:2409.12186,

    arXiv

  3. [11]

    Sampling-based approximations to minimum Bayes risk decoding for neural machine translation

    Bryan Eikema and Wilker Aziz. Sampling-based approximations to minimum Bayes risk decoding for neural machine translation. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10978–10993. Association for Computational Linguistics,

    2022

  4. [12]

    ALGO: Synthesizing algorithmic programs with generated oracle verifiers

    Kexun Zhang, Danqing Wang, Jingtao Xia, William Yang Wang, and Lei Li. ALGO: Synthesizing algorithmic programs with generated oracle verifiers. InThirty-Seventh Conference on Neural Information Processing Systems, 2023a. Dacheng Li, Shiyi Cao, Chengkun Cao, Xiuyu Li, Shangyin ...

    2025

  5. [13]

    Consistency meets verification: Enhancing test generation quality in large language models without ground-truth solutions.Arxiv Preprint Arxiv:2602.10522,

    Hamed Taherkhani, Alireza DaghighFarsoodeh, Mohammad Chowdhury, Hung Viet Pham, and Hadi Hemmati. Consistency meets verification: Enhancing test generation quality in large language models without ground-truth solutions.Arxiv Preprint Arxiv:2602.10522,

  6. [14]

    DOCE: Finding the sweet spot for execution- based code generation.Arxiv Preprint Arxiv:2408.13745,

    Haau-Sing Li, Patrick Fernandes, Iryna Gurevych, and André FT Martins. DOCE: Finding the sweet spot for execution- based code generation.Arxiv Preprint Arxiv:2408.13745,

    arXiv

  7. [15]

    Noise tolerance of learning to rank under class-conditional label noise.Arxiv Preprint Arxiv:2208.02126,

    Dany Haddad. Noise tolerance of learning to rank under class-conditional label noise.Arxiv Preprint Arxiv:2208.02126,

    arXiv

  8. [16]

    Training verifiers to solve math word problems, 2021.Arxiv Preprint Arxiv:2110.14168,

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems, 2021.Arxiv Preprint Arxiv:2110.14168,

    2021 arXiv

  9. [17]

    Leet- CodeDataset: A temporal dataset for robust evaluation and efficient training of code LLMs.arXiv preprint arXiv:2504.14655,

    Yunhui Xia, Wei Shen, Yan Wang, Jason Klein Liu, Huifeng Sun, Siyue Wu, Jian Hu, and Xiaolong Xu. Leet- CodeDataset: A temporal dataset for robust evaluation and efficient training of code LLMs.arXiv preprint arXiv:2504.14655,

    arXiv

  10. [18]

    It follows that the true AUC A(w) is the same for every pair (c+ ∈C +, c − ∈C −) and that two codes from the same class have identically distributed scores

    depend only on the test and the label, not on the specific code. It follows that the true AUC A(w) is the same for every pair (c+ ∈C +, c − ∈C −) and that two codes from the same class have identically distributed scores. Hoeffding’s inequality.Several proofs below rely on the...

    1963

  11. [19]

    Under uniform initial weights, the leave-one-out scores reduce to unweighted row sums with one column removed; since the AUC is scale-invariant, normalization is unnecessary

    B.1 ACES-C Algorithm 1 implements ACES-C (Section 3.1). Under uniform initial weights, the leave-one-out scores reduce to unweighted row sums with one column removed; since the AUC is scale-invariant, normalization is unnecessary. The algorithm has no tunable parameters: it co...

    2015

  12. [20]

    7:end for 8:ℓ←ℓ+η∇ ℓ P j wj \LOO-AUCj(w){gradient ascent (Adam in practice)} 9:end for 10:w←softmax(ℓ) 11:returns i =P j wjBij for alli 20 ACES: LOO-AUC Consistency for Code GenerationA PREPRINT The surrogate objective is non-convex because \LOO-AUCj(w) depends on w through th...

    2026

  13. [21]

    correct(α j =1)incorrect(α j <1) perf

    MV AUC:0.60; ACES-C AUC:0.77; ACES-O AUC:1.00. correct(α j =1)incorrect(α j <1) perf. perm. constructive misleading(δ j <0) t1 t2 t3 t4 t5 t6 t8 t9 t7 t10 MV -C -O ✓c 1 1 1 1 1 1 0 1 0 0 0 .60 1.00 1.00 ✓c 2 1 1 1 0 0 1 0 0 0 0 .40 .00 .99 ✓c 3 1 1 0 1 1 0 0 1 0 0 .50 1.00 .70...

    2024

  14. [22]

    assert {entry point}(...) ==

    C.13 Generation Prompts HumanEval, HumanEval+, and MBPP.We use the prompts from the MPSC protocol [Huang et al., 2024], reproduced below: Prompt for generating candidate solutions (HumanEval / HumanEval+ / MBPP) I want you to act like a Python programmer. I will give you the d...

    2024

Pith tools

Reviewed July 14, 2026 · model on record in the stance chip above.