pith. machine review for the scientific record. sign in

arxiv: 2207.10397 · v2 · submitted 2022-07-21 · 💻 cs.CL · cs.AI· cs.PL· cs.SE

Recognition: no theorem link

CodeT: Code Generation with Generated Tests

Bei Chen , Fengji Zhang , Anh Nguyen , Daoguang Zan , Zeqi Lin , Jian-Guang Lou , Weizhu Chen
Authors on Pith no claims yet

Pith reviewed 2026-05-15 23:51 UTC · model grok-4.3

classification 💻 cs.CL cs.AIcs.PLcs.SE
keywords code generationtest case generationlanguage modelcode selectionexecution agreementHumanEvalfunctional correctnesspre-trained models
0
0 comments X

The pith

CodeT generates test cases with the same model to select correct code samples via dual execution agreement.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes CodeT, a method that uses pre-trained language models both to produce multiple code solutions for a programming problem and to generate corresponding test cases automatically. It then executes the samples on those tests and selects the solution that shows consistent outputs across tests and agreement with other samples. This approach avoids the cost of manual test creation while improving selection quality. Experiments on HumanEval, MBPP, APPS, and CodeContests with five different models demonstrate large gains in pass@1 rates, including raising HumanEval performance to 65.8 percent. The core benefit is higher functional correctness without extra human effort on test writing.

Core claim

CodeT generates test cases for code samples using the same pre-trained language model, then applies dual execution agreement that checks both output consistency against the generated tests and agreement among the code samples themselves to pick the best solution.

What carries the argument

Dual execution agreement on model-generated tests and code samples, which selects the sample with highest consistency on test outputs and cross-sample agreement.

If this is right

  • CodeT raises pass@1 on HumanEval to 65.8 percent, an 18.8 percent absolute gain over code-davinci-002.
  • The method delivers consistent improvements across four benchmarks and five models of varying sizes.
  • Automatic test generation reduces the need for costly manual test creation while increasing test scenario coverage.
  • Selection quality improves for both small and large pre-trained language models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The approach could be extended to iterative refinement where selected code is used to generate more targeted tests.
  • It may apply to other structured generation tasks such as theorem proving or query synthesis where agreement on outputs can serve as a proxy for correctness.
  • Generated tests might reveal systematic weaknesses in the base model by highlighting failure modes that multiple samples share.

Load-bearing premise

Agreement between independently generated code samples on independently generated tests reliably indicates functional correctness rather than shared bugs or test weaknesses.

What would settle it

Finding a case where the code selected by CodeT passes all generated tests and agrees with peer samples yet fails on a comprehensive hidden test suite that covers edge cases missed by the generated tests.

read the original abstract

The task of generating code solutions for a given programming problem can benefit from the use of pre-trained language models such as Codex, which can produce multiple diverse samples. However, a major challenge for this task is to select the most appropriate solution from the multiple samples generated by the pre-trained language models. A natural way to evaluate the quality and correctness of a code solution is to run it against a set of test cases, but the manual creation of such test cases is often costly and time-consuming. In this paper, we propose a novel method, CodeT, that leverages the same pre-trained language models to automatically generate test cases for the code samples, thus reducing the human effort and increasing the coverage of the test scenarios. CodeT then executes the code samples using the generated test cases, and performs a dual execution agreement, which considers both the consistency of the outputs against the generated test cases and the agreement of the outputs with other code samples. We conduct comprehensive experiments on four benchmarks, HumanEval, MBPP, APPS and CodeContests, using five different pre-trained language models with varying sizes and capabilities. Our results show that CodeT can significantly improve the performance of code solution selection over previous methods, achieving remarkable and consistent gains across different models and benchmarks. For instance, CodeT improves the pass@1 metric on HumanEval to 65.8%, which represents an absolute improvement of 18.8% over the code-davinci-002 model, and an absolute improvement of more than 20% over the previous state-of-the-art results.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper proposes CodeT, a method that uses the same pre-trained language models to generate test cases for code samples produced by those models, then selects the best sample via dual execution agreement (output consistency on the generated tests plus agreement across samples). Comprehensive experiments on HumanEval, MBPP, APPS, and CodeContests with five models report large gains, including raising HumanEval pass@1 to 65.8% (absolute +18.8% over code-davinci-002 and >20% over prior SOTA).

Significance. If the gains are attributable to genuinely better functional selection rather than correlated blind spots in the generated tests, the work would meaningfully reduce dependence on human-written tests for code-generation pipelines and improve practical reliability of LM-based code assistants. The multi-benchmark, multi-model evaluation protocol is a clear strength.

major comments (2)
  1. [Section 3.2] Section 3.2 (dual execution agreement): the selection criterion assumes that agreement between code samples on LM-generated tests reliably signals functional correctness; because the identical model family produces both the candidate programs and the tests, the paper must supply evidence that shared distributional blind spots (e.g., missing edge cases) do not cause the method to prefer mutually buggy programs. Without such analysis the 18.8-point HumanEval gain cannot be fully attributed to improved correctness.
  2. [Section 4.1] Section 4.1 and Table 1: the reported pass@1 numbers depend on the precise number of generated samples per problem, the test-filtering threshold, and the exact definition of “agreement”; these parameters are not fully enumerated, making it impossible to reproduce the exact 65.8% figure or to isolate the contribution of the agreement step versus simple majority voting.
minor comments (2)
  1. [Figures 2-3] Figure 2 and Figure 3: axis labels and legend entries for the five models are too small and overlap; enlarging and clarifying them would improve readability.
  2. [Abstract] Abstract and Section 1: the claim of “more than 20% over previous state-of-the-art” should cite the exact prior work and metric value for each benchmark.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and insightful comments on our paper. We address each of the major comments below and have updated the manuscript accordingly to enhance clarity, reproducibility, and the analysis of our method's robustness.

read point-by-point responses

  1. Referee: [Section 3.2] Section 3.2 (dual execution agreement): the selection criterion assumes that agreement between code samples on LM-generated tests reliably signals functional correctness; because the identical model family produces both the candidate programs and the tests, the paper must supply evidence that shared distributional blind spots (e.g., missing edge cases) do not cause the method to prefer mutually buggy programs. Without such analysis the 18.8-point HumanEval gain cannot be fully attributed to improved correctness.

    Authors: We agree that demonstrating the absence of shared blind spots is crucial for attributing the performance gains to improved functional selection. Our current experiments show substantial improvements over baselines across diverse benchmarks, suggesting that the method captures correctness beyond mere agreement on common cases. However, to directly address this concern, we will add a new subsection in the revised manuscript analyzing the diversity and coverage of the generated tests, including comparisons with human-written tests on edge cases and failure modes. This will help quantify whether the agreement is driven by correct behavior rather than correlated errors. revision: yes

  2. Referee: [Section 4.1] Section 4.1 and Table 1: the reported pass@1 numbers depend on the precise number of generated samples per problem, the test-filtering threshold, and the exact definition of “agreement”; these parameters are not fully enumerated, making it impossible to reproduce the exact 65.8% figure or to isolate the contribution of the agreement step versus simple majority voting.

    Authors: We apologize for the lack of complete hyperparameter details in the original submission. In the revised manuscript, we will include a dedicated section that fully specifies all experimental parameters used in our experiments, including the number of samples generated per problem, the test generation process and filtering threshold, and the precise definition of dual execution agreement. Additionally, we will provide ablation studies to isolate the contribution of the dual agreement mechanism from simple majority voting. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical method evaluated on external benchmarks

full rationale

The paper presents CodeT as an empirical selection procedure: LM-generated code samples are executed against LM-generated tests, and selection uses observed output agreement. Reported gains (e.g., HumanEval pass@1 = 65.8%) are measured against fixed external test suites (HumanEval, MBPP, APPS, CodeContests) that are independent of the paper's own fitted quantities or definitions. No equations define a target metric in terms of the selection rule itself, no parameters are fitted to a subset and then called a prediction, and no load-bearing uniqueness theorem or ansatz is imported via self-citation. The central claim therefore remains falsifiable by the external benchmarks and does not reduce to its inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The approach rests on the domain assumption that language models can produce useful tests and that output agreement correlates with correctness. No free parameters or invented entities are introduced in the abstract description.

axioms (1)
  • domain assumption Pre-trained language models can generate both code solutions and test cases of sufficient quality to enable agreement-based selection.
    Invoked in the description of CodeT's core mechanism.

pith-pipeline@v0.9.0 · 5601 in / 1234 out tokens · 39609 ms · 2026-05-15T23:51:28.385332+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 19 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. RubricRefine: Improving Tool-Use Agent Reliability with Training-Free Pre-Execution Refinement

    cs.LG 2026-05 unverdicted novelty 7.0

    RubricRefine improves tool-use agent reliability to 0.86 on M3ToolEval by generating rubrics for pre-execution contract checking and iterative repair, outperforming baselines at 2.6X lower latency while showing no gai...

  2. POSTCONDBENCH: Benchmarking Correctness and Completeness in Formal Postcondition Inference

    cs.SE 2026-05 unverdicted novelty 7.0

    POSTCONDBENCH is a new multilingual benchmark that evaluates LLM postcondition generation on real code using defect discrimination to assess completeness beyond surface matching.

  3. Self-Consistency from Only Two Samples: CoT-PoT Ensembling for Efficient LLM Reasoning

    cs.CL 2026-04 unverdicted novelty 7.0

    CoT-PoT ensembling achieves self-consistency accuracy in LLMs with only two samples for 78.6% of tasks, reducing computation by 9.3x compared to standard methods.

  4. AdverMCTS: Combating Pseudo-Correctness in Code Generation via Adversarial Monte Carlo Tree Search

    cs.SE 2026-04 unverdicted novelty 7.0

    AdverMCTS frames code generation as a minimax game where an attacker evolves tests to expose flaws in solver-generated code, yielding more robust outputs than static-test baselines.

  5. BACE: LLM-based Code Generation through Bayesian Anchored Co-Evolution of Code and Test Populations

    cs.NE 2026-03 unverdicted novelty 7.0

    BACE reformulates LLM code synthesis as Bayesian co-evolution of code and test populations anchored on minimal public examples, achieving superior performance on LiveCodeBench v6.

  6. Reflexion: Language Agents with Verbal Reinforcement Learning

    cs.AI 2023-03 conditional novelty 7.0

    Reflexion lets LLM agents improve via stored verbal reflections on task feedback, reaching 91% pass@1 on HumanEval and outperforming prior GPT-4 results.

  7. Uncertainty Quantification for LLM-based Code Generation

    cs.SE 2026-05 unverdicted novelty 6.0

    RisCoSet applies multiple hypothesis testing to construct risk-controlling partial-program prediction sets for LLM code generation, achieving up to 24.5% less code removal than prior methods at equivalent risk levels.

  8. RubricRefine: Improving Tool-Use Agent Reliability with Training-Free Pre-Execution Refinement

    cs.LG 2026-05 unverdicted novelty 6.0

    RubricRefine raises average tool-use reliability to 0.86 on M3ToolEval across seven models by scoring candidate code against generated contract rubrics before execution, beating prior inference-time methods at 2.6X lo...

  9. PaT: Planning-after-Trial for Efficient Test-Time Code Generation

    cs.CL 2026-05 unverdicted novelty 6.0

    PaT defers planning until after failed trials in LLM code generation, enabling heterogeneous cheap-plus-powerful model setups that match large-model performance at roughly 69% lower cost.

  10. EGREFINE: An Execution-Grounded Optimization Framework for Text-to-SQL Schema Refinement

    cs.DB 2026-05 unverdicted novelty 6.0

    EGRefine optimizes column renamings via execution-grounded verification and view materialization to recover Text-to-SQL accuracy lost to schema naming issues while guaranteeing query equivalence.

  11. MEMCoder: Multi-dimensional Evolving Memory for Private-Library-Oriented Code Generation

    cs.SE 2026-04 unverdicted novelty 6.0

    MEMCoder boosts LLM code generation for private libraries by 16.31% pass@1 via a multi-dimensional evolving memory that distills usage guidelines from execution feedback and combines them with static docs.

  12. You Don't Need Public Tests to Generate Correct Code

    cs.SE 2026-04 unverdicted novelty 6.0

    DryRUN lets LLMs create their own test inputs and run internal simulations for self-correcting code generation, matching the performance of test-dependent methods like CodeSIM on LiveCodeBench without public tests or ...

  13. Co-Located Tests, Better AI Code: How Test Syntax Structure Affects Foundation Model Code Generation

    cs.SE 2026-04 unverdicted novelty 6.0

    Co-locating tests with implementation code yields substantially higher preservation and correctness in foundation-model-generated programs than separated test syntax.

  14. Ensemble-Based Uncertainty Estimation for Code Correctness Estimation

    cs.SE 2026-03 unverdicted novelty 6.0

    Ensemble Semantic Entropy improves correlation with code correctness over single-model methods and powers a cascading scaling system that cuts FLOPs by 64.9% while preserving performance on LiveCodeBench.

  15. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code

    cs.SE 2024-03 unverdicted novelty 6.0

    LiveCodeBench collects 400 recent contest problems to create a contamination-free benchmark evaluating LLMs on code generation and related capabilities like self-repair and execution.

  16. Bridging the Gap between User Intent and LLM: A Requirement Alignment Approach for Code Generation

    cs.SE 2026-04 unverdicted novelty 5.0

    REA-Coder improves LLM code generation by iteratively aligning requirements with model understanding and verifying outputs against the aligned spec.

  17. WebMAC: A Multi-Agent Collaborative Framework for Scenario Testing of Web Systems

    cs.SE 2026-04 unverdicted novelty 5.0

    WebMAC uses three specialized multi-agent modules to clarify test scenarios, partition them for adequacy, and generate executable scripts, yielding 30-60% higher success rates and 29% better efficiency than SOTA on fo...

  18. FLeX: Fourier-based Low-rank EXpansion for multilingual transfer

    cs.LG 2026-04 unverdicted novelty 4.0

    LoRA fine-tuning of Code Llama with Fourier regularization raises Java pass@1 from 34.2% to 42.1% while using a small high-quality dataset.

  19. A Survey on Large Language Models for Code Generation

    cs.CL 2024-06 unverdicted novelty 3.0

    A systematic literature review that organizes recent work on LLMs for code generation into a taxonomy covering data curation, model advances, evaluations, ethics, environmental impact, and applications, with benchmark...

Reference graph

Works this paper leans on

17 extracted references · 17 canonical work pages · cited by 18 Pith papers · 6 internal anchors

  1. [1]

    Program Synthesis with Large Language Models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732,

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901,

    work page 1901

  3. [3]

    Evaluating Large Language Models Trained on Code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374,

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    Training Verifiers to Solve Math Word Problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168,

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    InCoder: A Generative Model for Code Infilling and Synthesis

    Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. arXiv preprint, 2022a. Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Le...

    work page internal anchor Pith review arXiv

  6. [6]

    Fault-aware neural code rankers

    Jeevana Priya Inala, Chenglong Wang, Mei Yang, Andres Codas, Mark Encarnaci ´on, Shuvendu K Lahiri, Madanlal Musuvathi, and Jianfeng Gao. Fault-aware neural code rankers. arXiv preprint arXiv:2206.03865,

    work page arXiv

  7. [7]

    Coderl: Mastering code generation through pretrained models and deep reinforcement learning

    10 Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven CH Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. arXiv preprint arXiv:2207.01780,

    work page arXiv

  8. [8]

    BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

    Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre- training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461,

    work page internal anchor Pith review Pith/arXiv arXiv 1910

  9. [9]

    On the advance of making language models better reasoners

    Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. On the advance of making language models better reasoners. arXiv preprint arXiv:2206.02336, 2022a. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, R ´emi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-le...

    work page arXiv

  10. [10]

    Leveraging automated unit tests for unsupervised code translation

    Baptiste Roziere, Jie M Zhang, Francois Charton, Mark Harman, Gabriel Synnaeve, and Guil- laume Lample. Leveraging automated unit tests for unsupervised code translation. arXiv preprint arXiv:2110.06773,

    work page arXiv

  11. [11]

    Generate & rank: A multi-task framework for math word problems

    Jianhao Shen, Yichun Yin, Lin Li, Lifeng Shang, Xin Jiang, Ming Zhang, and Qun Liu. Generate & rank: A multi-task framework for math word problems. arXiv preprint arXiv:2109.03034,

    work page arXiv

  12. [12]

    Natural lan- guage to code translation with execution

    Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida I Wang. Natural lan- guage to code translation with execution. arXiv preprint arXiv:2204.11454,

    work page arXiv

  13. [13]

    Unit test case generation with transformers and focal context

    Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. Unit test case generation with transformers and focal context. arXiv preprint arXiv:2009.05617,

    work page arXiv 2009

  14. [14]

    Self-Consistency Improves Chain of Thought Reasoning in Language Models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171,

    work page internal anchor Pith review Pith/arXiv arXiv

  15. [15]

    \nclass

    The number of sampling test cases for each problem is set to 100 for the HumanEval and MBPP benchmarks, and 50 for the APPS and CodeContests benchmarks. When scoring consensus sets in CODE T, we use the square root of|Sx| to reduce the impact caused by code solutions. A supporting experiment can be found in Appendix C. For code solution post-processing, w...

    work page 2021

  16. [16]

    Baseline Filter

    generated solutions per problem for the APPS (CodeContests) benchmark. How- 14 k 1 10 50 100 1000 1 2 10 100 Baseline C ODET APPS INTRODUCTORY 29.3 48.5 60 .9 - - 47.3 18.0 52.7 58.4 9.9 - INTERVIEW 6.4 14.6 25 .4 - - 14.3 7.9 18.2 23.3 8.7 - COMPETITION 2.5 6.3 14 .5 - - 6.2 3.7 9.8 13.6 7.3 - CodeContests 1.0 4.1 7 .1 8.8 15 .2 3.2 2.2 5.6 9.3 5.2 12.3 ...

    work page 2021

  17. [17]

    "" remove_vowels is a function that takes string and returns string without vowels

    Moreover, using only 10 test cases per problem for C ODE T can still improve the baseline pass@ 1 performance of code-cushman-001 by absolute 4.3% and code-davinci-002 by absolute 9.5%. It demonstrates that C ODE T has high test case efficiency and we can use a smaller Sampling Number in real-world application to balance the performance and computation cos...

    work page 2022