Pith. sign in

REVIEW 5 major objections 7 minor 1 cited by

Can LLMs Generate High-Quality Test Cases for Algorithm Problems? TestCase-Eval: A Systematic Evaluation of Fault Coverage and Exposure

T0 review · 5 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TestCase-Eval, a 500-problem benchmark of 2024 Codeforces contests and 100,000 human incorrect solutions, reports that even the best LLM tested exposes only 43.8% of targeted faults, versus 93.3% for human experts.

desk verdict A useful new benchmark with a genuinely novel fault-exposure task, but the headline numbers need a validation and reporting pass before they should be quoted. read the letter →

arxiv 2506.12278 v1 pith:DR2FWU6A submitted 2025-06-13 cs.SE cs.CL

Zheyuan Yang , Zexi Kuang , Xue Xia , Yilun Zhao This is my paper
classification cs.SEcs.CL
keywords test-casegenerationLLMevaluationfaultcoverageexposureCodeforcesalgorithmproblemsbenchmarkcompetitiveprogramming
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TestCase-Eval asks whether LLMs can produce test cases that actually catch faulty algorithm code, not just execute lines or branches. The paper builds a benchmark from 500 Codeforces problems from 2024 and 100,000 real human incorrect submissions, and defines two measurable tasks: Fault Coverage, the fraction of known faulty submissions caught by a set of generated inputs, and Fault Exposure, whether a single input crafted for one faulty submission makes it fail. Across 19 open and proprietary LLMs, the best model reaches 43.8% on Fault Exposure while two human experts reach 93.3%; coverage of diverse faults also trails human performance, though by less. If the benchmark measures what it claims, current LLMs are a long way from expert-level adversarial test generation for algorithm problems.

What carries the argument

The carrying mechanism is an oracle-comparison scoring pipeline: a generated test input is run against a set of known-correct human reference solutions and against a sampled set of known-faulty human submissions, and the input 'detects' a fault when the faulty program's output differs from the oracle's. Fault Coverage aggregates these hits across N generated inputs (Cov@N), while Fault Exposure evaluates a single generated input against one faulty submission. The dataset is constructed to make this meaningful: problems from 2024 to avoid pretraining contamination, no special-judge problems, ten correct solutions verified to agree on outputs, and 200 incorrect submissions per problem stratified by error type, language, and the index of the first failing test.

What would settle it

Take a random sample of LLM-generated test inputs from both tasks, validate each one against the problem's stated input format and constraints, drop or repair the invalid ones, and recompute the coverage and exposure rates. If a large share turns out to be malformed, the current gap would shrink and the central comparison would be an artifact; if nearly all inputs are valid, the gap stands.

Watch

Extended reading notes

Core claim

TestCase-Eval's central claim is that LLM test-case generation should be judged by whether the generated inputs expose real incorrect implementations, and that on that standard all 19 evaluated models fall short of human experts. In the harder Fault Exposure task, the strongest model scores 43.8% against a 93.3% human-expert rate; even the best Fault Coverage numbers (Qwen3-32B at 50.8% cov@1 and 95.7% cov@20) sit just below human experts (56.2% and 97.2%). The paper also finds that reasoning-oriented LLMs outperform general-purpose LLMs, chain-of-thought prompting helps, Wrong Answer and Runtime Error faults are easier to expose than Time/Memory Limit Exceeded faults, and open-source models compete with proprietary ones.

Load-bearing premise

The load-bearing premise is that every LLM-generated test input is syntactically valid and satisfies the problem's constraints, so a differing output from a faulty submission really reflects that submission's bug rather than a malformed input; the paper does not describe any check of this.

Editorial extensions

If this is right

  • If the measured gap is real, LLM-written test sets cannot yet substitute for human adversarial testing on hard algorithm problems, particularly for time- and memory-limit faults.
  • Because reasoning-oriented models and CoT prompting consistently raise scores, the bottleneck is partly deliberate analysis of possible failure modes rather than code-generation fluency.
  • The benchmark's error-type breakdown suggests that progress on asymptotic-complexity reasoning would transfer directly to better detection of TLE and MLE submissions.
  • The competitive open-source results imply that future gains may come from task-specific reasoning training rather than from larger proprietary API models alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same oracle-comparison setup is a ready-made training reward: any generated input that splits a known-correct solution from a known-buggy one gives a cheap scalar signal, and the current 43.8% ceiling suggests large headroom.
  • Editorial inference: the difficulty labels, based on the index of the first failed Codeforces test, are only a proxy; grouping failures by error type and input-size sensitivity would give a cleaner picture of which faults are genuinely hard.
  • Editorial inference: a constraint and format validator over generated inputs would test the benchmark's own validity, since malformed inputs can make correct and incorrect programs fail alike.
  • Editorial inference: the benchmark's 2024-only design makes it a natural instrument for tracking year-over-year progress in adversarial test generation, in the same way coding benchmarks track code generation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper introduces TestCase-Eval, a benchmark for evaluating LLM-generated test cases on algorithm problems. It consists of 500 Codeforces problems from 2024 and a large collection of human-written correct and incorrect solutions. The benchmark defines two tasks: (1) Fault Coverage, where a model generates N test inputs and the union of incorrect submissions detected by these inputs is measured as Cov@N; and (2) Fault Exposure, where a model must generate a single test input that triggers a bug in a given faulty solution, measured as the Fault Exposure Rate. Nineteen LLMs are evaluated under direct-output and CoT prompting. The central empirical claim is that the benchmark is very challenging: the best model (Qwen3-32B) scores 43.8% on Fault Exposure, far below the human-expert rate of 93.3%, and even top models achieve only moderate Fault Coverage.

Significance. If the benchmark is valid, it fills a real gap in LLM test-generation evaluation by moving beyond line/branch coverage to fault-oriented metrics that matter for algorithmic problems. The dataset construction has notable strengths: problems are from 2024 (outside typical pretraining cutoffs), large scale (500 problems, tens of thousands of human submissions), and the two-task design separates broad coverage from targeted fault triggering. The authors also provide experimental evidence on 19 models, including reasoning models, and compare CoT vs direct prompting. However, the quantitative conclusions currently rest on an evaluation pipeline that does not validate the syntactic well-formedness or constraint satisfaction of LLM-generated test inputs. This makes the oracle comparison potentially unreliable and the headline gap (43.8% vs 93.3%) not yet interpretable. The benchmark construction and task definitions are a useful contribution, but the scoring pipeline must be made explicit and validated before the numerical results can be taken as evidence about LLM capability.

major comments (5)
  1. [Section 3.4 and Appendix C.2] The evaluation treats the raw LLM output string directly as a test input without any specified extraction, parsing, or validation step. The CoT prompts in Figures 5 and 7 ask the model to 'Think step by step' and then output inside a code block, so the raw response may include reasoning text, markdown fences, or the code block delimiters themselves; no procedure is described for extracting the actual input. Furthermore, there is no check that the extracted string is syntactically valid or respects the problem's input bounds. Running an invalid or out-of-range input against the oracle and the faulty solution can cause crashes, timeouts, or undefined behavior in both programs, and any difference in exit status or crash signature is counted as 'fault exposure.' This inflates Cov@N and Fault Exposure Rate in a model- and problem-dependent way. Because every reported number, including the headline Qwen3-32B vs human expert comparison, passes through this unvalidated oracle, the quantitative conclusions are not secure. Please specify the exact extraction and validation procedure, report the rate of parse failures and constraint violations per model, and confirm that all scored inputs are well-formed and within stated bounds.
  2. [Section 4.1 and Table 1] The human-expert performance is measured on a randomly sampled subset of 20 problems, while the model scores in Table 1 are apparently aggregated over all 500 problems. If the model scores are not computed on the same 20-problem subset, the headline comparison '93.3% vs 43.8%' confounds model capability with problem difficulty. For example, the 20-problem subset may be easier or harder than the full benchmark. Please report model performance on the same 20-problem subset used for the human evaluation, and clearly state whether the model numbers in Table 1 include the human-evaluated problems. Without this, the central claim of a large human-model gap is not rigorously established.
  3. [Section 3.1 vs Appendix B.2] The paper gives inconsistent counts for the incorrect-submission pool. Section 3.1 states 'we collect 200 incorrect submissions' per problem and a total of '100,000 submissions,' while Appendix B.2 reports a final set of 118,611 incorrect solutions across 500 problems, with an average of 237 per problem. These numbers affect the denominator of Cov@N (which uses |F_total|) and the sampling pool for Task 2. Please reconcile the counts and clarify whether the 100,000 figure in the abstract refers to a sample before filtering or a different stage of the pipeline.
  4. [Section B.2 and Table 2] The filtering criteria in Appendix B.2 state that each sampled solution 'must be semantically valid and executable, passing compilation and basic test cases without syntax or runtime errors,' yet Table 2 and the accompanying analysis include Runtime Error (RE) as a major fault category. This is a direct contradiction. If RE submissions are included in the evaluation, the stated criterion is false; if RE is only a target error type that is not part of the filtered pool, the table's RE column needs a different explanation. Please clarify the inclusion or exclusion of RE submissions and adjust the filtering description accordingly.
  5. [Section 3.1 (oracle definition)] The oracle is described as 'ten correct human-written solutions' that 'consistently produce identical outputs for the same test inputs.' The paper does not specify how the oracle output is determined during evaluation: is it the output of a single reference solution, a majority vote of the ten, or an agreement check that discards inputs where the ten disagree? For a valid generated input, if the ten correct solutions do not all produce the same output, the notion of a 'correct output' is undefined and the fault-exposure decision is ambiguous. Please state the oracle aggregation rule and what happens on disagreement.
minor comments (7)
  1. [Abstract and Section 3.1] The abstract says '100,000 human-crafted solutions,' while Appendix B.2 reports 118,611; please use consistent numbers throughout the paper and in the abstract.
  2. [Figure 1] The evaluation pipeline diagram shows 'Correct code' and 'Incorrect code' feeding into a sandbox but has no input-validation step; add an explicit 'input parsing and constraint check' box to reflect the required procedure.
  3. [Section 4.2] The claim that 'Task 2 yields more stable and reproducible results across multiple evaluation runs, whereas Task 1 scores exhibit higher variance' is presented without evidence; either include variance or confidence intervals, or soften the claim.
  4. [Appendix C.2 (prompts)] The prompts in Figures 5–8 ask for 'a challenging test input' (singular), but Task 1 reports Cov@N for N up to 20; clarify whether N independent generations are sampled per problem, and whether each generation may itself contain multiple test cases (as suggested in Appendix B.4).
  5. [Table 1] The 'Human Expert' row is based on a 20-problem subset; add a table footnote or marker indicating this, so readers are not misled into comparing it directly with model rows over 500 problems.
  6. [References] The citation for the sandbox environment is given as 'ExecEval (Khan et al., 2023)', but the reference list entry is titled 'xcodeeval'; please correct the citation to match the actual work used.
  7. [Limitations section] The Limitations section acknowledges lack of detailed error analysis and performance-bottleneck assessment, but does not acknowledge the input-validation gap or the human-baseline subset mismatch; please extend the limitations to cover these methodological issues.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TestCase-Eval is an execution-defined empirical benchmark; the headline model-vs-human gap is measured, not derived from fitted inputs or load-bearing self-citation.

full rationale

The paper makes no first-principles derivation whose conclusion is equivalent to its inputs. The central metrics, Cov@N and Fault Exposure Rate, are defined in Sections 3.2 and 3.3 directly in terms of execution outcomes: Cov@N counts the union of incorrectly submitted solutions detected by the generated inputs, and Fault Exposure Rate counts whether a generated input produces a differing outcome between correct and faulty code. These definitions are operationalizations of 'test-case quality' as detection of the collected Codeforces incorrect submissions; that is a construct-definition choice for a benchmark, not a circular reduction, because the reported scores are produced by actually running the LLM outputs through the sandbox rather than being algebraically forced by the definition. The human-expert baseline (93.3% in Task 2) is independently elicited from two rated competitive programmers, not derived from model behavior. No parameter is fitted and then renamed as a prediction, and no uniqueness theorem or ansatz is imported from the authors' prior work. Self-citations (e.g., Yu et al. 2024a,b) appear only in related-work positioning and do not carry the argument. The ExecEval sandbox is cited as an external execution tool, not as evidence for the benchmark's conclusions. The paper's own Limitations section explicitly scopes the benchmark to correctness-based faults and notes that performance bottlenecks are not systematically assessed; this is an honest scope limitation, not evidence of circularity. The reader's concern that LLM-generated inputs are not validated for syntax and constraints is a real evaluation-validity threat that could affect the interpretation of the absolute scores, but it is not a circularity: the metric does not presuppose the conclusion that LLMs underperform humans. Therefore no specific circular step can be quoted, and the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No parameter was fitted to data; the entries above are design choices and background assumptions that the evaluation results depend on. The most fragile assumption is that generated inputs are valid, which is currently implicit rather than checked.

free parameters (4)
  • Incorrect submissions per problem (Task 1 pool) = 200 (stated); 237 average (Appendix B.2)
    The dataset size is given as 100,000 total in Section 3.1 but 118,611 in Appendix B.2, so the per-problem count is ambiguous; the score depends on the pool of faults used to compute coverage.
  • Correct solutions per problem for oracle = 10
    Used to determine expected outputs; assumes all ten agree on every generated input, which is not re-verified for each generated test.
  • Faulty solutions sampled per problem for Task 2 = 20
    Stratified sample used for fault exposure; the reported exposure rate depends on this subset.
  • Difficulty quantile split = 40% easy / 30% medium / 30% hard
    Based on the index of the first failing official test case; an arbitrary but explicit choice.
assumptions (4)
  • domain assumption The test-case index where a Codeforces submission first fails is a meaningful proxy for fault difficulty.
    Used to split faulty solutions into Easy/Medium/Hard; the paper acknowledges this is only a reasonable estimate.
  • domain assumption The 10 correct human solutions produce an exact oracle for every valid input.
    Assumed in Section 3.1; if outputs diverge on some generated input, any faulty-solution comparison is unreliable.
  • ad hoc to paper LLM-generated test inputs are well-formed and within problem constraints.
    No validation step is described before running generated inputs; prompts only instruct the model to output a valid test input.
  • domain assumption Codeforces problems dated January to December 2024 are outside the pretraining corpora of the evaluated models.
    Stated in Section 3.1 as a contamination control; not verifiable for all 19 models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can LLMs Generate High-Quality Test Cases for Algorithm Problems? TestCase-Eval: A Systematic Evaluation of Fault Coverage and Exposure." pith.science (2026). https://pith.science/paper/DR2FWU6A

@misc{pith2026250612278,
  author       = {Pith},
  title        = {Pith review of: Can LLMs Generate High-Quality Test Cases for Algorithm Problems? TestCase-Eval: A Systematic Evaluation of Fault Coverage and Exposure},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DR2FWU6A}},
  note         = {Machine review of arXiv:2506.12278}
}
read the original abstract

We introduce TestCase-Eval, a new benchmark for systematic evaluation of LLMs in test-case generation. TestCase-Eval includes 500 algorithm problems and 100,000 human-crafted solutions from the Codeforces platform. It focuses on two pivotal tasks: (1) Fault Coverage, which measures how well LLM-generated test sets probe diverse input scenarios and cover a wide range of potential failure modes. (2) Fault Exposure, which evaluates whether LLMs can craft a tailored test input that reveals a specific incorrect code implementation. We provide a comprehensive assessment of 19 state-of-the-art open-source and proprietary LLMs on TestCase-Eval, offering insights into their strengths and limitations in generating effective test cases for algorithm problems.

Figures

Figures reproduced from arXiv: 2506.12278 by the authors.

Figure 1
Figure 1. An overview of TestCase-Eval and the research pipeline in this study. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. (Top) Performance comparison between CoT prompting and direct-output prompting for Task 2. (Bottom) Overall model performance us￾ing CoT prompting across C++, Java, and Python in Task 2. Open-source models compete with or surpass proprietary counterparts. Our results indicate that leading open-source models are highly com￾petitive. In Task 1 (Fault Coverage), several open-source models, including Qwen3-32B (50.8 cov… view at source ↗
Figure 3
Figure 3. Distribution of Problem Difficulty Levels. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: An example of a problem that needs a special judge. [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: The Chain-of-Thought prompt used in Task1. [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: The Direct Output prompt used in Task1 [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: The Chain-of-Thought prompt used in Task2. [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: The Direct Output prompt used in Task2 [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Coding Agents as Test-Suite Auditors: Finding What Official Suites Miss While Approaching What They Catch

    cs.SE 2026-08 conditional novelty 6.0 of 10

    A target-blind agent-generated test suite with a consensus-and-validator certification chain found 906 verified accepted-but-buggy submissions on AtCoder and led all baselines on post-cutoff Codeforces problems.

Reference graph

Works this paper leans on

32 extracted references · 9 canonical work pages · cited by 1 Pith paper

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Cai, Michael Terry, Quoc V

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021. https://api.semanticscholar.org/CorpusID:237142385 Program synthesis with large language models . ArXiv, abs/2108.07732

    arXiv 2021

  4. [4]

    Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen

    Bei Chen, Fengji Zhang, A. Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2022. https://api.semanticscholar.org/CorpusID:250920542 Codet: Code generation with generated tests . ArXiv, abs/2207.10397

    arXiv 2022

  5. [5]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Jun-Mei Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiaoling Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bing-Li Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Dama...

    work page 2025

  6. [6]

    Mingzhe Du, Anh Tuan Luu, Bin Ji, Qian Liu, and See-Kiong Ng. 2024. https://api.semanticscholar.org/CorpusID:269758190 Mercury: A code efficiency benchmark for code large language models . In Neural Information Processing Systems

    work page 2024

  7. [7]

    OpenAI Ahmed El-Kishky, Alexander Wei, Andre Saraiva, Borys Minaev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera, Jakub W. Pachocki, Jerry Tworek, Lorenz Kuhn, Lukasz Kaiser, Mark Chen, Max Schwarzer, Mostafa Rohaninejad, Nat McAleese, o3 contributors, Oleg Murk, Rhythm Garg, Rui Shu, Szymon Sidor, Vineet Kosaraju, and Wenda Z...

    work page 2025

  8. [8]

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Xiaodong Song, and Jacob Steinhardt. 2021. https://api.semanticscholar.org/CorpusID:234790100 Measuring coding challenge competence with apps . ArXiv, abs/2105.09938

    arXiv 2021

Show all 32 references
  1. [9]

    Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. 2023. https://api.semanticscholar.org/CorpusID:263671541 Mlagentbench: Evaluating language agents on machine learning experimentation . In International Conference on Machine Learning

    2023

  2. [10]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, An Yang, Rui Men, Fei Huang, Shanghaoran Quan, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. 2024. https://api.semanticscholar.org/CorpusID:27270...

    2024 arXiv

  3. [11]

    Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida I. Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. https://api.semanticscholar.org/CorpusID:268379413 Livecodebench: Holistic and contamination free evaluation of large language models f...

    2024 arXiv

  4. [12]

    Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L'elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, T...

    2023 arXiv

  5. [13]

    Chen, Shafiq Joty, and Furu Wei

    Fangkai Jiao, Geyang Guo, Xingxing Zhang, Nancy F. Chen, Shafiq Joty, and Furu Wei. 2024. https://api.semanticscholar.org/CorpusID:274233865 Preference optimization for reasoning with pseudo feedback . ArXiv, abs/2411.16345

    2024 arXiv

  6. [14]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. https://api.semanticscholar.org/CorpusID:263829697 Swe-bench: Can language models resolve real-world github issues? ArXiv, abs/2310.06770

    2023 arXiv

  7. [15]

    Rizwan Parvez, and Shafiq R

    Mohammad Abdullah Matin Khan, M Saiful Bari, Do Xuan Long, Weishi Wang, Md. Rizwan Parvez, and Shafiq R. Joty. 2023. https://api.semanticscholar.org/CorpusID:257365592 xcodeeval: A large scale multilingual multitask benchmark for code understanding, generation, translation and...

    2023 arXiv

  8. [16]

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, R \'e mi Leblond, Tom, Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de, Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowa...

    2022

  9. [17]

    Jiawei Liu, Chun Xia, Yuyao Wang, and Lingming Zhang. 2023. https://api.semanticscholar.org/CorpusID:258437095 Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation . ArXiv, abs/2305.01210

    2023 arXiv

  10. [18]

    Meta. 2024. https://api.semanticscholar.org/CorpusID:271571434 The llama 3 herd of models . ArXiv, abs/2407.21783

    2024 arXiv

  11. [19]

    OpenAI. 2024. https://openai.com/index/hello-gpt-4o/ Hello gpt-4o

    2024

  12. [20]

    OpenAI. 2025. https://openai.com/index/gpt-4-1/ Introducing gpt-4.1 in the api

    2025

  13. [21]

    ByteDance Seed. 2025. https://github.com/ByteDance-Seed/Seed-Coder/blob/master/Seed-Coder.pdf Seed-coder: Let the code model curate data for itself

    2025

  14. [22]

    Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. https://api.semanticscholar.org/CorpusID:258833055 Reflexion: language agents with verbal reinforcement learning . In Neural Information Processing Systems

    2023

  15. [23]

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, ...

    2025 arXiv

  16. [24]

    Mistral AI Team. 2024. https://mistral.ai/news/codestral/ Codestral: Hello, world!

    2024

  17. [25]

    Qwen Team. 2025. https://qwenlm.github.io/blog/qwq-32b/ Qwq-32b: Embracing the power of reinforcement learning

    2025

  18. [26]

    Wenhan Wang, Chenyuan Yang, Zhijie Wang, Yuheng Huang, Zhaoyang Chu, Da Song, Lingming Zhang, An Ran Chen, and Lei Ma. 2024. https://api.semanticscholar.org/CorpusID:270357873 Testeval: Benchmarking large language models for test case generation . ArXiv, abs/2406.04531

    2024 arXiv

  19. [27]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

    2025 arXiv

  20. [28]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng...

    2024 arXiv

  21. [29]

    Qwen An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxin Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, ...

    2024 arXiv

  22. [31]

    Zhaojian Yu, Yilun Zhao, Arman Cohan, and Xiao-Ping Zhang. 2024 b . https://api.semanticscholar.org/CorpusID:275134345 Humaneval pro and mbpp pro: Evaluating large language models on self-invoking code generation . ArXiv, abs/2412.21199

    2024 arXiv

  23. [32]

    Huaye Zeng, Dongfu Jiang, Haozhe Wang, Ping Nie, Xiaotong Chen, and Wenhu Chen. 2025. https://api.semanticscholar.org/CorpusID:276107488 Acecoder: Acing coder rl via automated test-case synthesis

    2025

  24. [33]

    Kexun Zhang, Danqing Wang, Jingtao Xia, William Yang Wang, and Lei Li. 2023. https://api.semanticscholar.org/CorpusID:258865731 Algo: Synthesizing algorithmic programs with generated oracle verifiers . ArXiv, abs/2305.14591

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.