Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Rethinking Verification for LLM Code Generation: From Generation to Testing

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SAGA, a human-LLM test generator, catches 90.6% of hidden code bugs and produces benchmarks that re-rank top models.

desk verdict Real bottleneck, plausible method, but the headline verifier-accuracy numbers are circular because the generator and the evaluation share the same human bug set. read the letter →

arxiv 2507.06920 v2 pith:PL2N6E2Y submitted 2025-07-09 cs.CL

Zihan Ma , Taolin Zhang , Maosong Cao , Junnan Liu , Wenwei Zhang , Minnan Luo , Songyang Zhang , Kai Chen This is my paper
classification cs.CL
keywords test-casegenerationcodeevaluationverifieraccuracydetectionrateLLMbenchmarkshuman-LLMcollaborationRLVRtest-suitediversity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Standard code benchmarks use small test suites that miss many bugs, so LLM scores are inflated and reinforcement learning from verifiable rewards gets noisy feedback. The paper argues this happens because LLM-generated test cases cluster around the same systematic failure patterns, while human bugs are diverse and dispersed. It formalizes two metrics, detection rate and verifier accuracy, builds TCGBench from recent competitive-programming problems with tens of thousands of real incorrect submissions, and proposes SAGA, a pipeline that combines correct human solutions and incorrect human submissions to generate targeted tests. On its 270-problem curated subset, SAGA reaches a 90.62% detection rate and 32.58% verifier accuracy at 50 tests, and the benchmark it synthesizes, CodeCompass, scores 10.78 percentage points higher on verifier accuracy than LiveCodeBench-v6 suites on a shared subset. If the method holds up, it offers a way to build stricter, less gameable code benchmarks and improve the reward signals used to train code models.

What carries the argument

SAGA is the central mechanism: a two-pronged, prompt-driven pipeline. Multidimensional Analysis uses roughly ten correct human solutions per problem to deconstruct their defensive logic into mathematical constraints such as equivalence classes and boundary values, then generates targeted adversarial inputs; Differential Analysis pairs each incorrect submission with its author's later corrected solution and asks the LLM where the two diverge, directly exposing constraint-handling and edge-case errors. Generated Python case scripts carry math explanations and self-validation code, and a ground-truth interpreter turns inputs into outputs. The paper also supplies a theoretical backstop: if test-detection events are positively correlated with effective average correlation $\bar{\rho}_{\mathrm{eff}}$ and average per-test detection probability $\bar{p}$, detection rate saturates at $1-(1-\bar{p})^{1/\bar{\rho}_{\mathrm{eff}}}<1$ as the number of tests grows, which motivates breaking correlation rather than merely adding tests.

What would settle it

Hold out a set of wrong solutions that SAGA never saw, such as submissions collected from contests released after its training cutoff, and run SAGA-built verifiers on them. If verifier accuracy stays near the random-interpreter baseline on that held-out set, the claim that SAGA excels at detecting unseen human errors is refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that the verifier, not the generator, is the weak link in LLM code evaluation, and that verification quality can be engineered by feeding an LLM structured human priors from both sides of the error boundary. SAGA (Strategic Adversarial & Constraint-differential GenerAtive workflow) analyzes correct human solutions to extract constraints and boundary strategies, and differentially analyzes incorrect human submissions against their corrected versions to find inputs where the wrong program diverges; Python case scripts then generate test inputs, a ground-truth interpreter produces outputs, and self-validation code checks the inputs before use. On TCGBench-Lite, SAGA reaches a detection rate of 90.62% and a verifier accuracy of 32.58% at 50 tests, compared with 81.07% and 16.72% for the random input-interpreter baseline, and the CodeCompass suites it synthesizes raise verifier accuracy by 10.78 percentage points over LiveCodeBench-v6 on the shared subset. The paper interprets these gains as evidence that test-case generation should be judged not by the number of tests but by per-test potency and inter-test correlation, and that human error diversity is an input to generation, not a byproduct of it.

Load-bearing premise

The headline detection and verifier-accuracy numbers are computed against the same human wrong submissions that SAGA was shown, so the proof does not yet cover wrong programs SAGA has never seen.

Editorial extensions

If this is right

  • Code generation leaderboards built this way should show lower, and more separable, pass@1 numbers; in the paper's shared subset the relative drop averaged 9.56% and two model pairs swapped ranks.
  • RLVR pipelines that use SAGA-style verifiers should receive sharper reward signals and face less reward hacking, because more wrong programs are actually rejected.
  • More tests alone cannot close the gap: the saturation corollary and measurements show random input sampling plateaus far below complete detection.
  • The SAGA-distilled 7B specialist reaches a detection rate of 89.44% and a verifier accuracy of 29.11% at 50 tests, beating larger-model baselines, so high-quality test-case generation can be made affordable.
  • The same recipe can scale to larger problem sets, providing training data with more accurate rewards.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The most informative follow-up experiment would be time-sliced: train SAGA on submissions before a cutoff, then score its verifiers on submissions collected after the cutoff; a large drop would bound how much of the reported gap carries over to genuinely unseen bugs.
  • The differential-analysis step is a general recipe, pair any incorrect artifact with its corrected counterpart and ask an LLM where they diverge, which could transfer to bug-finding in reasoning chains or proof checking.
  • The saturation bound is a general statement about correlated tests, so the same reasoning could be used to audit any test-suite generator, including non-LLM fuzzers, by estimating its effective correlation from an accuracy-versus-size curve.
  • Distilling SAGA into a 7B model suggests test-case synthesis could eventually run inside the training loop at scale, not just at benchmark construction time.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper addresses the quality of test suites used to evaluate LLM-generated code. It defines Detection Rate (DR) and Verifier Accuracy (VAcc) metrics, constructs TCGBench (1,840 competitive programming problems with an average of 36.66 incorrect human submissions per problem), and proposes SAGA, a human-LLM collaborative test-case generation framework that leverages both correct solutions (S_human) and incorrect submissions (S_wrong) via Multidimensional and Differential Analysis. The paper reports DR@50=90.62% and VAcc@50=32.58% for SAGA on a 270-problem subset, claims a 10.78 percentage-point higher VAcc than LiveCodeBench-v6 on a shared AtCoder subset, and derives an asymptotic saturation bound for DR. It also introduces TCGCoder-7B, a distilled 7B model for test generation.

Significance. The problem is important and the paper contributes a substantial benchmark and a plausible method. The idea of mining both correct and incorrect human solutions is valuable, and the paper includes extensive experiments and releases code/data. However, the headline results are not interpretable as generalized verifier quality because the evaluation is circular: the same incorrect submissions that seed SAGA's Differential Analysis are used to compute DR and VAcc. Consequently, the significance of the proposed method for detecting unseen bugs is currently untested. The theoretical saturation result also rests on an unproven approximating model.

major comments (4)
  1. [§3.1 and §3.2.1, Table 2] The evaluation of SAGA is circular: Differential Analysis consumes the TCGBench-Lite wrong submissions S_wrong to produce inputs I_diff, and Table 2 then reports DR@50=90.62% and VAcc@50=32.58% computed on the same S_wrong set (average 41.41 per problem). These numbers measure how well the generated suite covers the exact bugs shown to the generator, not its ability to detect unseen incorrect programs. The paper's own ablation in Appendix H.2 shows that removing Differential Analysis lowers VAcc@50 by 8.35 points on Codeforces, confirming that the reported gains flow through this leakage channel. A held-out set of human bugs that is never shown to the generator must be used to support the claimed generalized verifier quality.
  2. [§2.1, Definition 2] Definition 2 defines VAcc over the infinite set S_wrong(P) of all incorrect solutions, but the experiments replace it with the finite collection of human submissions used to build the test suites. The paper provides no argument that this finite sample is representative of the infinite set. The reported VAcc values are therefore not estimates of the quantity in Definition 2. The authors should either redefine VAcc for finite evaluation sets or supply a representativeness analysis (e.g., convergence of VAcc as the number of bugs grows).
  3. [§2.2 and Appendix C, Corollary 1] Corollary 1 and the supporting Theorem 1 rest on an approximating effective-sample-size model whose parameters p_bar and rho_bar_eff are not estimated from data. The proof is a sketch, and the empirical validation in Figure 4 shows only qualitative saturation without a quantitative fit. This is insufficient to support the stated asymptotic bound or the claim that the experiments 'validate the impact of correlation rho'. I recommend presenting this as a heuristic or providing a rigorous derivation and model fit.
  4. [Tables 2–3, Figures 7–8] The paper reports point estimates without uncertainty quantification. The comparison with LiveCodeBench-v6 in Table 3 is based on only 101 shared AtCoder problems, and the 10.78 percentage-point VAcc gap and the pass@1 re-ranking in Figures 7–8 could easily arise from sampling noise. Bootstrap confidence intervals or per-problem standard errors should be reported for all headline metrics.
minor comments (5)
  1. [§3.2 vs Table 2] Section 3.2 states SAGA achieves DR 93.81% and VAcc 41.33% on the full TCGBench, whereas Table 2 reports DR@50=90.62% and VAcc@50=32.58% on TCGBench-Lite; the discrepancy is not explained.
  2. [Abstract and Contributions] The abstract mentions a 12.14% Verifier Accuracy improvement, while the contributions list says 15.86% and Table 2 shows a 15.86 percentage-point gain over Input-Interpreter; clarify which comparison is meant.
  3. [§3.1] The term 'Shuman' is used without definition; it appears to denote the set of correct human solutions (S_correct). Please define it at first use.
  4. [Table 4] In Table 4, 'Avg.Swrong/Problem' is a typo; it should be 'Avg. S_wrong/Problem'.
  5. [§3.2.1] The claim that TCGCoder-7B 'outperforms all these established baselines' is ambiguous because SAGA itself beats TCGCoder-7B on most metrics; rephrase to say it outperforms the non-SAGA baselines.

Circularity Check

2 steps flagged · score 7.0 of 10

TCGBench DR/VAcc and CodeCompass verifier-accuracy gains are scored on the same human-bug pool SAGA's Differential Analysis consumes, so the headline detection claims are largely by construction.

  1. fitted input called prediction [Sec. 3.1 (Differential Analysis) feeding Sec. 3.2.1 / Table 2 metrics; Definition 2 in Sec. 2.1]
    "It compares failed submissions (Swrong) with their corrected versions (S′correct) to find inputs Idiff where Swrong(Idiff) != S′correct(Idiff)... The incorporation of these differentially identified inputs Idiff into the test suite T creates a more rigorous and challenging evaluation framework, as it specifically targets known failure modes... TCGBench-Lite includes an average of 41.41 incorrect submissions (Swrong) per problem."

    Definition 2 defines Verifier Accuracy over the set of all incorrect solutions Swrong(P), and Definition 1 defines Detection Rate over incorrect candidate solutions; the experiments in Table 2 replace this domain with the TCGBench-Lite human-bug pool (avg. 41.41 submissions/problem). That same pool is the input to SAGA's Differential Analysis, which constructs inputs Idiff on which each analyzed Swrong disagrees with the corrected version and adds those inputs to suite T. By construction, each analyzed submission is then killed by a test, so DR@50=90.62% and VAcc@50=32.58% measure fit to generator-seen bugs rather than detection of unseen incorrect programs. Appendix H.2 confirms the channel: removing Differential Analysis costs 8.35 VAcc@50 points on Codeforces.

  2. fitted input called prediction [Sec. 4 and Table 3 (CodeCompass vs. LiveCodeBench-v6, shared 101 AtCoder subset); abstract claim of Verifier Acc '10.78% higher']
    "The verifiers within CodeCompass are synthesized using SAGA for the 270 contemporary problems that also constitute TCGBench-Lite... CodeCompass demonstrates markedly higher efficacy in identifying faulty solutions... V Acc@40 19.61% 30.39%"

    The Verifier Accuracy gap that drives the abstract's 10.78% claim (30.39% vs 19.61% at @40) is computed on the same TCGBench-Lite human-bug pool from which CodeCompass's SAGA-synthesized tests were differentially constructed, whereas LiveCodeBench-v6's tests were not built from that pool. Both suites are therefore scored on bugs one of them was explicitly shown, so the measured verifier-quality superiority is inflated by asymmetric exposure to the evaluation set. The pass@1 comparisons (Figures 7-8) are partially independent because LLM-generated model solutions were not used in test construction, which is why this is partial rather than total circularity.

full rationale

The paper's central numeric claims reduce, at least partly, to its own inputs. SAGA's Differential Analysis (Sec. 3.1) is given the incorrect human submissions Swrong (41.41 per problem in TCGBench-Lite), constructs inputs on which those submissions disagree with corrected versions, and adds the resulting tests to the suite. The evaluation metrics (Definitions 1-2) are then measured on that same Swrong pool, and Table 2 reports DR@50=90.62% and VAcc@50=32.58% as headline evidence of verifier quality. Because the differentially-added tests are constructed to fail the very submissions counted as 'detected,' the headline detection numbers are substantially forced by construction; the paper's own H.2 ablation (8.35 VAcc points lost without Differential Analysis) shows the gain flows through this evaluation-set channel. The CodeCompass verifier-quality comparison in Table 3 inherits the same contamination, since CodeCompass's tests were built from the same TCGBench-Lite bugs on which both suites are scored. I found no load-bearing self-citation, no imported uniqueness theorem, and no ansatz-smuggled-via-citation pattern; the CodeCompass pass@1 re-ranking results (Figures 7-8) retain partially independent signal because those evaluations use fresh LLM solutions. The circularity is therefore substantial but not total: the per-bug detection for differentially-analyzed submissions is by construction, while the aggregate claims still also reflect the Multidimensional-Analysis component and unpaired submissions. Score 7 reflects that one or more headline predictions reduce by construction while independent content remains in the benchmark-strictness pass@1 measurements.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central experimental claim rests on the assumption that the finite set of human wrong submissions used to generate tests also defines ground truth for evaluation; this is the main circularity burden. The theoretical saturation bound adds modeling assumptions, p_bar and rho_bar_eff, that are not measured independently.

free parameters (2)
  • p_bar (average per-test detection probability)
    Latent parameter in Corollary 1; assumed stable but never estimated or reported, so the saturation bound is not checked numerically.
  • rho_bar_eff (effective average correlation between detection events)
    Introduced ad hoc in Appendix C to approximate the variance of correlated Bernoulli trials; no independent estimate is given.
assumptions (3)
  • domain assumption All functions in Swrong are truly incorrect and representative of all possible incorrect programs.
    Definition 2 idealizes Swrong as the set of all incorrect solutions; experiments use a finite sample of human WA and TLE submissions, and SAGA's Differential Analysis consumes this same sample.
  • ad hoc to paper Detection events across test cases can be approximated by Bernoulli trials with common p and correlation rho (effective-sample-size model).
    Appendix C Eq. 1 assumes average variance approximates p(1-p) and pairwise covariance is rho*p(1-p); the resulting Corollary 1 is a proof sketch.
  • domain assumption Ground-truth solutions fP used as interpreters are correct.
    TCG output computation relies on online-judge reference solutions; a buggy interpreter would invalidate the test outputs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking Verification for LLM Code Generation: From Generation to Testing." pith.science (2026). https://pith.science/paper/PL2N6E2Y

@misc{pith2026250706920,
  author       = {Pith},
  title        = {Pith review of: Rethinking Verification for LLM Code Generation: From Generation to Testing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PL2N6E2Y}},
  note         = {Machine review of arXiv:2507.06920}
}
read the original abstract

Large language models (LLMs) have recently achieved notable success in code-generation benchmarks such as HumanEval and LiveCodeBench. However, a detailed examination reveals that these evaluation suites often comprise only a limited number of homogeneous test cases, resulting in subtle faults going undetected. This not only artificially inflates measured performance but also compromises accurate reward estimation in reinforcement learning frameworks utilizing verifiable rewards (RLVR). To address these critical shortcomings, we systematically investigate the test-case generation (TCG) task by proposing multi-dimensional metrics designed to rigorously quantify test-suite thoroughness. Furthermore, we introduce a human-LLM collaborative method (SAGA), leveraging human programming expertise with LLM reasoning capability, aimed at significantly enhancing both the coverage and the quality of generated test cases. In addition, we develop a TCGBench to facilitate the study of the TCG task. Experiments show that SAGA achieves a detection rate of 90.62% and a verifier accuracy of 32.58% on TCGBench. The Verifier Accuracy (Verifier Acc) of the code generation evaluation benchmark synthesized by SAGA is 10.78% higher than that of LiveCodeBench-v6. These results demonstrate the effectiveness of our proposed method. We hope this work contributes to building a scalable foundation for reliable LLM code evaluation, further advancing RLVR in code generation, and paving the way for automated adversarial test synthesis and adaptive benchmark integration.

Figures

Figures reproduced from arXiv: 2507.06920 by the authors.

Figure 1
Figure 1. (a) Verifiers synthesized primarily from [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The code evaluation pipeline and different [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Direct generation issues: (a) Low quality of LLM-generated tests. (b) High self-pass rates [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Experimental validation of Input-Interpreter (random sampling) limitations on TCGBench. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Overview of the SAGA framework. SAGA leverages both GroundTruth (correct human [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: SAGA outperforms the Baseline (Random Input-Interpreter) and its individual analytical [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: Model Ranking Changes. 4 Towards Advanced Applications of SAGA SAGA’s proven capability in generating high-quality, diverse, and discriminative test suites (Section 3) enables advancements in LLM code evaluation and training. This section primarily explores the develop…
Figure 9
Figure 9. Figure 9: SAGA performance on Codeforces (CF) and Nowcoder (NC) problems from the full [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: SAGA performance with different LLM backbones (Qwen-Coder, Qwen-72B, DeepSeek [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Performance comparison (VAcc@50 and DR@50) of SAGA, its components, and [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: Heatmap illustrating AUC@50 performance of mixed random test suites. Diagonal [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. When the Reward Suite Is Leaky: A Preregistered Causal Contrast of Natural Verifier False Positives in RLVR

    cs.LG 2026-07 accept novelty 7.0 of 10

    Natural false positives in MBPP rewards inflate train reward and pay for real bugs, yet a preregistered leaky-vs-hardened GRPO contrast finds held-out non-inferiority and selection of pre-existing errors, not learned ...

  2. Judging Is Not Enumerating: Silent Omissions in LLM-Authored Acceptable Sets

    cs.AI 2026-08 conditional novelty 6.0 of 10

    LLMs under one-shot greedy decoding enumerate or materialize acceptable sets much worse than they judge membership, a gap that persists across scale, family, and generation and is dominated by omissions.

Reference graph

Works this paper leans on

52 extracted references · 16 canonical work pages · cited by 2 Pith papers

  1. [1]

    Burke, Tsong Yueh Chen, John A

    Saswat Anand, Edmund K. Burke, Tsong Yueh Chen, John A. Clark, Myra B. Cohen, Wolfgang Grieskamp, Mark Harman, Mary Jean Harrold, and Phil McMinn. An orchestrated survey of methodologies for automated software test case generation. J. Syst. Softw., 86(8):1978–2001,

    work page 1978

  2. [2]

    Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P

    Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Is...

    arXiv doi:10.48550/arxiv.2312.11805 2023

  3. [3]

    Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J

    Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V . Le, and Charles Sutton. Program synthesis with large language models. CoRR, abs/2108.07732, 2021. URL https://arxiv. org/abs/2108.07732

    arXiv 2021

  4. [4]

    Anisotropy-induced Extrinsic Chirality and Chiral Discrimination of Surface Plasmon Polaritons

    S. Beyleda and V . Gruhn. Bintest - search-based test case generation. In Proceedings 27th Annual International Computer Software and Applications Conference. COMPAC 2003, pages 28–33, 2003. doi: 10.1109/CMPSAC.2003.1245318

    work page Pith review arXiv 2003

  5. [5]

    Symbolic execution for software testing: three decades later

    Cristian Cadar and Koushik Sen. Symbolic execution for software testing: three decades later. Commun. ACM, 56(2):82–90, 2013. doi: 10.1145/2408776.2408795. URL https: //doi.org/10.1145/2408776.2408795

    arXiv 2013

  6. [7]

    Teaching large language models to self-debug

    Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128, 2023

    arXiv 2023

  7. [8]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https://arxiv.org/abs/2501.12948. 10

    arXiv 2025

  8. [9]

    A multi-layer fault triggering framework based on evolutionary strategy guided symbolic execution for automated test case generation

    Zhiyu Duan, Yujia Li, Pubo Ma, Xiaodong Gou, and Shunkun Yang. A multi-layer fault triggering framework based on evolutionary strategy guided symbolic execution for automated test case generation. In 22nd IEEE International Conference on Software Quality, Reliability, and Security, QRS 2022 - Companion, Guangzhou, China, December 5-9, 2022, pages 255–262....

    arXiv 2022

Show all 52 references
  1. [10]

    Competitive programming with large reasoning models

    Ahmed El-Kishky, Alexander Wei, Andre Saraiva, Borys Minaiev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera Gilaberte, Jakub Pachocki, Jerry Tworek, Lorenz Kuhn, Lukasz Kaiser, Mark Chen, Max Schwarzer, Mostafa Rohaninejad, Nat McAleese, o3 contribu...

    arXiv

  2. [11]

    Dissecting american fuzzy lop: A fuzzbench evaluation

    Andrea Fioraldi, Alessandro Mantovani, Dominik Christian Maier, and Davide Balzarotti. Dissecting american fuzzy lop: A fuzzbench evaluation. ACM Trans. Softw. Eng. Methodol., 32 (2):52:1–52:26, 2023. doi: 10.1145/3580596. URL https://doi.org/10.1145/3580596

    2023 doi

  3. [12]

    Applied longitudinal analysis

    Garrett M Fitzmaurice, Nan M Laird, and James H Ware. Applied longitudinal analysis. John Wiley & Sons, 2012

    2012

  4. [13]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y . Wu, Y . K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming - the rise of code intelligence. CoRR, abs/2401.14196, 2...

    2024 doi arXiv

  5. [14]

    Afshin Mansouri, and Yuanyuan Zhang

    Mark Harman, S. Afshin Mansouri, and Yuanyuan Zhang. Search-based software engineering: Trends, techniques and applications. ACM Comput. Surv., 45(1), December 2012. ISSN 0360-0300. doi: 10.1145/2379776.2379787. URL https://doi.org/10.1145/2379776. 2379787

    2012

  6. [15]

    Codecot: Tackling code syntax errors in cot reasoning for code generation, 2024

    Dong Huang, Qingwen Bu, Yuhao Qing, and Heming Cui. Codecot: Tackling code syntax errors in cot reasoning for code generation, 2024. URL https://arxiv.org/abs/2308.08784

    2024 arXiv

  7. [16]

    Zhang, Michael Luck, Qingwen Bu, Yuhao Qing, and Heming Cui

    Dong Huang, Jie M. Zhang, Michael Luck, Qingwen Bu, Yuhao Qing, and Heming Cui. Agentcoder: Multi-agent-based code generation with iterative testing and optimisation, 2024. URL https://arxiv.org/abs/2312.13010

    2024 arXiv

  8. [17]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jia- jun Zhang, Bowen Yu, Kai Dang, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186, 2024

    2024 arXiv

  9. [18]

    Testgeneval: A real world unit test generation and test completion benchmark, 2024

    Kush Jain, Gabriel Synnaeve, and Baptiste Rozière. Testgeneval: A real world unit test generation and test completion benchmark, 2024

    2024

  10. [19]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  11. [20]

    Survey sampling

    Leslie Kish. Survey sampling. new york: John wesley & sons. Am Polit Sci Rev, 59(4):1025, 1965

    1965

  12. [21]

    Coderl: Mastering code generation through pretrained models and deep reinforcement learning

    Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu-Hong Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances i...

    2022

  13. [22]

    Large language models as test case generators: Performance evaluation and enhancement

    Kefan Li and Yuan Yuan. Large language models as test case generators: Performance evaluation and enhancement. CoRR, abs/2404.13340, 2024. doi: 10.48550/ARXIV .2404.13340. URL https://doi.org/10.48550/arXiv.2404.13340

    2024 doi arXiv

  14. [23]

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, ...

    2023

  15. [24]

    Taco: Topics in algorithmic code generation dataset

    Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. Taco: Topics in algorithmic code generation dataset. arXiv preprint arXiv:2312.14852, 2023

    2023 arXiv

  16. [25]

    Yujia Li, David H. Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, ...

    2022 doi arXiv

  17. [26]

    Sample size calculations for studies with correlated observations

    Guanghan Liu and Kung-Yee Liang. Sample size calculations for studies with correlated observations. Biometrics, pages 937–947, 1997

    1997

  18. [27]

    Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: //openrev...

    2023

  19. [28]

    Evaluating language models for efficient code generation

    Jiawei Liu, Songrun Xie, Junhao Wang, Yuxiang Wei, Yifeng Ding, and Lingming Zhang. Evaluating language models for efficient code generation. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=IBCBMeAhmC

    2024

  20. [29]

    Zhang, Zhenpeng Chen, Federica Sarro, Gang Huang, and Yun Ma

    Kaibo Liu, Yudong Han, Yiyang Liu, Jie M. Zhang, Zhenpeng Chen, Federica Sarro, Gang Huang, and Yun Ma. Trickybugs: A dataset of corner-case bugs in plausible programs. In 2024 IEEE/ACM 21st International Conference on Mining Software Repositories (MSR), pages 113–117, 2024

    2024

  21. [30]

    Sampling: design and analysis

    Sharon L Lohr. Sampling: design and analysis. Chapman and Hall/CRC, 2021

    2021

  22. [31]

    Dynamic scaling of unit tests for code reward modeling, 2025

    Zeyao Ma, Xiaokang Zhang, Jing Zhang, Jifan Yu, Sijia Luo, and Jie Tang. Dynamic scaling of unit tests for code reward modeling, 2025. URL https://arxiv.org/abs/2501.01054

    2025 arXiv

  23. [32]

    Manès, HyungSeok Han, Choongwoo Han, Sang Kil Cha, Manuel Egele, Edward J

    Valentin J.M. Manès, HyungSeok Han, Choongwoo Han, Sang Kil Cha, Manuel Egele, Edward J. Schwartz, and Maverick Woo. The art, science, and engineering of fuzzing: A survey. IEEE Transactions on Software Engineering , 47(11):2312–2331, 2021. doi: 10.1109/TSE.2019. 2946563

    2021 doi

  24. [33]

    Search-based software test data generation: a survey

    Phil McMinn. Search-based software test data generation: a survey. Softw. Test. Verification Reliab., 14(2):105–156, 2004. doi: 10.1002/STVR.294. URL https://doi.org/10.1002/ stvr.294. 12

    2004 doi

  25. [34]

    Wang, and Xi Victoria Lin

    Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-Tau Yih, Sida I. Wang, and Xi Victoria Lin. LEVER: learning to verify language-to-code generation with execution. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarl...

    2023

  26. [35]

    GPT-4 technical report

    OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/ARXIV .2303. 08774. URL https://doi.org/10.48550/arXiv.2303.08774

    2023 doi arXiv

  27. [36]

    Carlos Pacheco and Michael D. Ernst. Randoop: feedback-directed random testing for java. In Companion to the 22nd ACM SIGPLAN Conference on Object-Oriented Programming Systems and Applications Companion , OOPSLA ’07, page 815–816, New York, NY , USA, 2007. Association for Comp...

    2007

  28. [37]

    Codeforces cots

    Guilherme Penedo, Anton Lozhkov, Hynek Kydlí ˇcek, Loubna Ben Allal, Edward Beech- ing, Agustín Piqueres Lajarín, Quentin Gallouédec, Nathan Habib, Lewis Tunstall, and Leandro von Werra. Codeforces cots. https://huggingface.co/datasets/open-r1/ codeforces-cots, 2025

    2025

  29. [38]

    Code llama: Open foundation models for code

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défoss...

    2023 doi arXiv

  30. [39]

    libFuzzer – a library for coverage-guided fuzz testing

    Kostya Serebryany and the LLVM Team. libFuzzer – a library for coverage-guided fuzz testing. https://llvm.org/docs/LibFuzzer.html, 2016. Accessed: 2024-04-15

    2016

  31. [40]

    Execution-based code generation using deep reinforcement learning

    Parshin Shojaee, Aneesh Jain, Sindhu Tipirneni, and Chandan K Reddy. Execution-based code generation using deep reinforcement learning. arXiv preprint arXiv:2301.13816, 2023

    2023 arXiv

  32. [41]

    TESTEV AL: benchmarking large language models for test case generation

    Wenhan Wang, Chenyuan Yang, Zhijie Wang, Yuheng Huang, Zhaoyang Chu, Da Song, Ling- ming Zhang, An Ran Chen, and Lei Ma. TESTEV AL: benchmarking large language models for test case generation. CoRR, abs/2406.04531, 2024. doi: 10.48550/ARXIV .2406.04531. URL https://doi.org/10....

    2024 doi arXiv

  33. [42]

    Joty, and Steven C

    Yue Wang, Weishi Wang, Shafiq R. Joty, and Steven C. H. Hoi. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 202...

    2021

  34. [43]

    Yue Wang, Hung Le, Akhilesh Gotmare, Nghi D. Q. Bui, Junnan Li, and Steven C. H. Hoi. Codet5+: Open code large language models for code understanding and generation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Proceedings of the 2023 Conference on Empirical Methods i...

    2023 doi

  35. [44]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng...

    2024 arXiv

  36. [45]

    Feedback- controlled random test generation

    Kohsuke Yatoh, Kazunori Sakamoto, Fuyuki Ishikawa, and Shinichi Honiden. Feedback- controlled random test generation. In Michal Young and Tao Xie, editors, Proceedings of the 2015 International Symposium on Software Testing and Analysis, ISSTA 2015, Baltimore, MD, USA, July 12...

    2015

  37. [46]

    Acecoder: Acing coder rl via automated test-case synthesis

    Huaye Zeng, Dongfu Jiang, Haozhe Wang, Ping Nie, Xiaotong Chen, and Wenhu Chen. Acecoder: Acing coder rl via automated test-case synthesis. ArXiv, 2502.01718, 2025

    2025 arXiv

  38. [47]

    Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x

    Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Lei Shen, Zihan Wang, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x. In Proceedings of the 29th ACM SIGK...

    2023

  39. [48]

    Bigcodebench: Bench- marking code generation with diverse function calls and complex instructions

    Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Bench- marking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15...

    2024 arXiv

  40. [52]

    Direct Generation: LLMs produce complete test cases (inputs and outputs). This includes assertion-focused methods (e.g., CodeCoT [15], AgentCoder [16]) and direct input-output syn- thesis (e.g., TestChain [ 22], CodeRM [ 31], AceCoder [ 46]), aiming for logical coverage and bo...

  41. [53]

    powerful

    Input-Interpreter: LLMs generate test inputs, which are then executed by a ground-truth solution to derive outputs. This is seen in LiveCodeBench [19],CodeForce-Cot [37] and related to LLM- 15 guided fuzzing. EvalPlus [27, 28] also aligns by mutating seed inputs for execution....

    2024

  42. [2013]

    URL https://doi.org/10.1016/j.jss.2013.02

    doi: 10.1016/J.JSS.2013.02.061. URL https://doi.org/10.1016/j.jss.2013.02. 061

    2013 doi

  43. [2021]

    URL https://arxiv.org/abs/2107.03374

    arXiv

  44. [2025]

    URL https://doi.org/10.48550/arXiv.2502

    doi: 10.48550/ARXIV .2502.06807. URL https://doi.org/10.48550/arXiv.2502. 06807

    doi arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.