Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that LLMs can build valid, hard, multilingual code benchmarks without manual annotation by generating test inputs, executing them against an LLM-written reference solution in a sandbox, and writing each problem statement l

desk verdict Reverse-order generation is a real contribution, but the leaderboard's reliability rests on the authors' own 87.6% audit of six languages and zero evidence on the other 14. read the letter →

arxiv 2508.09101 v1 pith:MLWZXDFU submitted 2025-08-12 cs.CL cs.SE

Jason Chou , Ao Liu , Yuchi Deng , Zhiying Zeng , Tao Zhang , Haotian Zhu , Jianwei Cai , Yue Mao
show 8 more authors
Chenchen Zhang Lingyun Tan Ziyan Xu Bohui Zhai Hengyi Liu Speed Zhu Wiggin Zhou Fengzong Lian
This is my paper
classification cs.CLcs.SE
keywords codegenerationbenchmarkmultilingualevaluationLLM-generatedbenchmarksLLM-sandboxinteractionreverse-orderproblemtestcaseAutoBenchAI
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AutoCodeGen is a fully automated pipeline that turns raw code snippets into finished programming problems with executable test cases. Instead of asking an LLM to write tests from a problem statement, it reverses the order: an LLM first evolves a snippet into a self-contained reference solution, writes test inputs on that basis, executes solution plus inputs in a multilingual sandbox to obtain the expected outputs, and composes the problem statement last to describe the solution. Test outputs come from execution rather than generation, so the tests are internally consistent by construction; the problem is authored to match the solution, so the benchmark rewards reproducing that solution's behavior. The result, AutoCodeBench, holds 3,920 problems across 20 languages; the best of more than 30 evaluated models scores 52.4% Pass@1, and the combined upper bound across all models is 74.8%. The paper argues this makes large-scale, high-difficulty multilingual evaluation feasible without human curation, and it ships a simplified variant, a base-model completion variant, and a multilingual sandbox.

What carries the argument

The load-bearing mechanism is the reverse-order, execution-verified test generation loop: the reference solution is written first, test inputs are written second, expected outputs are obtained by running the solution on those inputs in the multilingual sandbox, and the problem statement is synthesized last to fit the solution and its tests. What does the work is that the sandbox execution converts test-output correctness from an LLM judgment call into a runtime fact, which is what lets the pipeline claim completeness and correctness of test cases without human review; the reverse order is what ensures the problem, solution, and tests describe the same artifact.

What would settle it

In a random sample of AutoCodeBench problems, have human programmers (or a different model family) write their own test cases directly from the problem statements, then run all models that pass the original generated tests on these independent tests. If pass rates drop sharply, the generated tests encode the reference solution's specific output choices and edge behaviors rather than the requirements stated in the problem.

Watch

Extended reading notes

Core claim

The central claim is that reverse-order generation makes benchmark construction reliable enough to be fully automated, without manual annotation. The pipeline starts from real-world code snippets, evolves each into a self-contained reference solution with a strong code LLM, generates public and private test-input functions, executes the solution together with those inputs in a multilingual sandbox so the test outputs are computed rather than guessed, and then prompts the model to write a problem statement that matches the solution's function names, signatures, and input/output format, embedding the public tests as examples. Three filtering stages follow: a weak model removes problems it can

Load-bearing premise

The load-bearing assumption is that each LLM-generated reference solution is correct ground truth: test outputs are computed by executing that solution and the problem is written to describe it, so any bug or under-specified edge behavior in the solution is encoded as expected behavior—and the paper's own six-language audit found only 87.6% of problem–test pairs valid, meaning more than one in ten violate this premise.

Editorial extensions

If this is right

  • If the pipeline is sound, code benchmarks can be created and refreshed automatically for new languages or domains, removing the human annotation bottleneck that limits current benchmark scale.
  • The gap between the strongest model (52.4%) and the all-models union upper bound (74.8%) implies that combining strengths across models, or distilling cross-model solutions, has substantial headroom on these tasks.
  • Problems that require implementing multiple coordinated functions or classes consistently lower every model's score, identifying multi-logic synthesis as a specific weakness current models share.
  • Execution feedback from the sandbox raises scores for all tested models across three refinement turns, making sandbox-in-the-loop refinement a practical, model-scale-independent lever for improving code generation.
  • The completion-based subset extends high-difficulty multilingual evaluation to base models via few-shot inputs, a regime the paper argues previous base-model benchmarks do not cover.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same reverse-order loop can serve as a training-data generator, which the paper itself flags as possible; a further consequence the paper does not explore is that benchmarks and training sets could co-evolve from one pipeline, letting a model family be evaluated on problems its own lineage generated.
  • Editorial inference: because expected outputs derive from a single reference solution, Pass@1 on this benchmark likely measures fidelity to that solution's behavior—naming, output formatting, edge-case choices—as much as algorithmic correctness; scoring the same problems against independent reference solutions would separate the two.
  • Editorial inference: the manual audit covered only six of the twenty languages, and the other fourteen pass through an extra translation step, so the 87.6% validity figure may not hold for low-resource languages; re-auditing a sample of translated problems would show where incomplete descriptions concentrate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes AutoCodeGen, a fully automated pipeline that generates multilingual code-generation benchmarks from Stack-Edu code snippets. The pipeline works in reverse order: an LLM evolves a self-contained solution, test inputs are generated and executed in a multilingual sandbox to obtain outputs, a problem statement is then written to match the solution and tests, and finally three filtering stages remove too-easy, low-quality, or redundant items. The resulting AutoCodeBench contains 3,920 problems across 20 languages, plus AutoCodeBench-Lite and AutoCodeBench-Complete. The authors evaluate over 30 models and report that the best model reaches only 52.4% Pass@1, that reasoning models outperform non-reasoning models, and that multi-logic problems cause a consistent performance drop. The paper also contributes a multilingual sandbox and a manual audit of six languages reporting 87.6% valid problem-test pairs.

Significance. If the central validity claim holds, the paper makes a useful contribution: it demonstrates a scalable, human-free pipeline for constructing multilingual code benchmarks and provides a large, balanced evaluation suite with unusually broad language coverage. The strengths are real: the execution-based test-output generation is machine-checked, the evaluation spans more than 30 models, the authors are transparent about their manual audit and about possible model bias, and they release a sandbox that could benefit the community. The significance is conditional, however, on the validity evidence. The manual audit reports 12.4% invalid instances in the six audited languages and no audit for the other 14 languages, and the difficulty filter is defined by a DeepSeek model's pass rate while DeepSeek-family models are also evaluated. These issues bear directly on the headline rankings and on the claim that AutoCodeBench is a high-difficulty benchmark, so the contribution is defensible but requires additional robustness evidence before the central claims can be accepted as stated.

major comments (4)
  1. [§2.2.2, §2.2.3, §4.1, Table 8] The test outputs are produced by executing an LLM-generated reference solution, and the problem statement is then written to describe that same solution. This guarantees consistency with one implementation, not correctness with respect to an independent specification. The paper's own manual audit finds only 87.6% valid problem-test pairs in the six audited languages, with incomplete problem descriptions as the most frequent issue (Appendix B). Despite this, Pass@1 and the union 'Current Upper Bound' in Table 4 are computed over all 3,920 problems with no exclusion of the flagged invalid instances. A 12.4% known contamination rate is large enough to shift model scores and possibly rankings, especially because the failure mode is not random noise but a systematic mismatch between the stated problem and the hidden reference behavior. Please provide a sensitivity analysis (e.g., recompute sc
  2. [§2.2.5, §4.1] Fourteen of the twenty languages are constructed by approximate translation from unused data, and the manual audit covers only Python, C++, Java, JavaScript, Go, and Shell. The 87.6% validity figure therefore applies to a minority of the benchmark languages, and the validity of the majority is unmeasured. Translation is not a validity-preserving operation for this pipeline: the translation prompt in Figure 15 rewrites both the reference solution and the test functions, which can introduce new naming mismatches, precision issues, or altered edge-case behavior. The paper should either extend the manual audit to a sample of translated languages or explicitly restrict the validity claim to the six directly generated languages until such evidence is provided.
  3. [§2.2.4, Table 2, §4.2, Table 7] Difficulty labels and the 'easy-problem filter' are both defined by the pass rate of DeepSeek-Coder-V2-Lite, and DeepSeek-family models are among the evaluated models in Table 4. The paper acknowledges a possible favorable bias toward DeepSeek and attempts to measure it in Table 7, but that measurement covers only five models and 3,600 sampled instances in six languages, with no confidence intervals or statistical test. Because the filter removes problems that DeepSeek-Coder-V2-Lite solves, it can differentially affect the measured Pass@1 of DeepSeek-family models relative to other families. Please report the effect of each filtering stage on all evaluated model families, or otherwise quantify the bias more rigorously; the current evidence is too limited to support the claim in §4.2 that the impact is 'minimal.'
  4. [§2.2.4, Appendix B] The LLM-as-Critic prompt in Figure 10 instructs the critic to 'assum[e] the problem is completely correct' and to evaluate only whether the test function is consistent with the problem. Appendix B then reports that the most frequent defect is incomplete problem descriptions. The automated critic is therefore structurally unable to catch the dominant failure mode, and the manual audit is the only validity evidence for the pipeline. This reinforces the need for the robustness analysis requested above; without it, the paper's claim that AutoCodeGen 'ensures the correctness and completeness of test cases' is stronger than the evidence supports.
minor comments (5)
  1. [Table 1 and §4.1] AutoCodeBench is described as 'Human-Free' in Table 1, but §4.1 reports a manual audit by six professional annotators. Please clarify that the claim refers to the generation pipeline, not to post-hoc verification.
  2. [§2.2.6, §3.7, Conclusion] The completion-based benchmark is called AutoCodeBench-Complete in §3.7 and AutoCodeBench-Comp in the Conclusion. Please use one consistent name.
  3. [Table 4] The label 'Current Upper Bound' is potentially misleading: it is the union of problems solved by at least one evaluated model, not an upper bound in the mathematical sense. Consider renaming it 'Union Pass@1' or 'Ensemble Upper Bound.'
  4. [§3.6] Heading contains a typo: 'Multi-T urn' should be 'Multi-Turn.'
  5. [§2.2.4] The sentence 'We sample answers for each problem ten times using the model and validate the correctness via sandbox execution' should specify whether 'correctness' refers to the generated solution or to the model's sampled answer. This affects the interpretation of the 25.1% Python filtering figure.

Circularity Check

3 steps flagged · score 6.0 of 10

Benchmark validity is defined by agreement with the LLM-generated reference solution, and DeepSeek-family models both shape the benchmark (as generators, critics, and difficulty filter) and are ranked on it.

  1. self definitional [§2.2.2 Test Output Generation and §2.2.3 Programming Problem Generation]
    "We concatenate the code solution with test input functions and execute them in the sandbox to obtain the corresponding test outputs. ... Using these guidelines, we prompt DeepSeek-V3-0324 to generate high-quality programming problems based on the code solution (with appropriate comments) and the corresponding test function, while embedding the public test function as example usage."

    By construction, the test outputs are the outputs of the LLM-generated solution, and the problem statement is written afterward to describe that same solution. Correctness and completeness of tests are therefore defined as consistency with one arbitrary implementation, not with an independent specification. A model that reproduces the generated solution's implementation quirks passes; a different but correct solution can fail. The paper's own audit (Appendix B) reports only 87.6% validity in six languages and identifies incomplete problem descriptions as the most frequent issue, confirming that the assumed equivalence between problem and solution does not hold unconditionally.

  2. fitted input called prediction [Table 2 caption / §2.2.4 Difficulty Control; §4.2]
    "The difficulty level is determined by the number of passes in ten samplings of DeepSeek-Coder-V2-Lite. Problems with zero correct solutions are classified as hard, 1-5 correct solutions as medium, and those with more than five as easy."

    The benchmark's difficulty labels and easy-problem filter are defined by the pass rate of DeepSeek-Coder-V2-Lite, and then models from the DeepSeek family are evaluated and ranked on the resulting benchmark. DeepSeek models are also the generators (DeepSeek-V3-0324) and critics (DeepSeek-R1-0528) in the pipeline. The evaluation of DeepSeek-family models is therefore not independent of the selection criteria that shaped the benchmark; the paper concedes this in §4.2: 'our automated process may introduce a favorable bias toward the DeepSeek family of models, but the impact is minimal.'

1 more flagged steps
  1. other [Appendix B Manual Verification]
    "During the annotation process, we assume that the programming problems are completely correct. The primary task of the annotators is to assess the correctness of the test functions and their alignment with the programming problem."

    The human audit, like the LLM-as-Critic stage (§2.2.4: 'assuming the problem is entirely correct'), verifies only consistency between the test functions and the problem description; it never checks the problem description against an independent specification. Since the problem description was generated from the reference solution, the audit measures internal self-consistency of the generation loop, not correctness against an external ground truth. The known failure mode (incomplete problem descriptions) is exactly what this 'assume correct' step prevents the audit from detecting, so the 87.6% figure is a ceiling on self-consistency, not independent validation.

full rationale

The circularity here is not a self-citation or imported uniqueness theorem; it is in the construction of the benchmark oracle. The paper's central validity claim is that AutoCodeGen 'ensures the correctness and completeness of test cases' by generating test inputs and 'obtaining test outputs through a multilingual sandbox,' but those outputs are obtained by executing the same LLM-generated reference solution that later defines the problem statement. Thus the problem, solution, and tests form one self-consistent loop rather than an independent specification-then-verification chain. The quality-control and manual-audit stages explicitly assume the problem is correct, so they do not break this loop. The manual audit's 87.6% validity in six languages, with incomplete descriptions as the most common defect, and the complete absence of audit for 14 translated languages, mean the validity claim is measured to be violated in over 12% of audited cases and unknown elsewhere. Additionally, difficulty labels and the easy-problem filter are defined by DeepSeek-Coder-V2-Lite pass rates, while DeepSeek-family models are among the evaluated models, and the paper acknowledges favorable bias toward DeepSeek. These are genuine partial circularities in the benchmark's validity and ranking claims, though not total: sandbox execution does verify executability, some filtering is external to the final model set, and a manual audit was performed. A score of 6 reflects that the central claims reduce in part to self-consistency with an LLM-generated solution, while still leaving meaningful empirical content.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

No new physical or theoretical entities are postulated. The benchmarks and sandbox are engineering artifacts, not entities requiring independent falsifiable evidence. The main epistemic load is carried by the axioms above: LLM-generated solutions as ground truth, LLM-as-Critic reliability, and sandbox correctness.

free parameters (4)
  • Difficulty thresholds = 0 passes = hard, 1-5 passes = medium, >5 passes = easy
    Used in Table 2 to label Easy/Med/Hard and to describe ACB difficulty; thresholds are arbitrary and computed from DeepSeek-Coder-V2-Lite pass rates.
  • Sampling count for easy-problem filter = 10
    Section 2.2.4: problems solved in all 10 DeepSeek-Coder-V2-Lite samples are discarded. The choice of 10 determines the aggressiveness of the difficulty filter.
  • Test case counts = public <= 3, private >= 7
    Section 2.2.2: design choice for demo vs full tests. The 'completeness' claim rests on this hand-chosen count, not on coverage metrics.
  • Lite subset size = 1,586 (target ~1,500)
    Section 2.2.6: selected by ascending pass count after discarding problems with fewer than 2 passes; the size is an arbitrary practical choice.
assumptions (6)
  • domain assumption LLM-generated code solutions, evolved from Stack-Edu snippets, are correct ground truth for the derived problems.
    Sections 2.2.1-2.2.3: test outputs are obtained by executing these solutions; if a solution is semantically wrong or contains hidden assumptions, the oracle is wrong. Manual verification found 12.4% of problem-test pairs misaligned.
  • domain assumption DeepSeek-R1-0528 LLM-as-Critic reliably identifies low-quality or misaligned problem-test pairs.
    Section 2.2.4 Quality Control uses it to filter data; no quantitative agreement between critic and human annotators is reported.
  • domain assumption The Stack-Edu and Stack v2 code snippets represent practical, diverse real-world programming tasks.
    Section 2.2.1 seeds; no analysis of seed distribution, license effects, or potential duplication with LLM training data is provided.
  • ad hoc to paper DeepSeek-Coder-V2-Lite pass rate is a valid difficulty signal for filtering.
    Section 2.2.4: problems solved 10/10 are discarded. This couples the difficulty definition to one model family's capability.
  • domain assumption Sandbox execution environment is correct across 20+ languages.
    Section 2.2.2 and Appendix C: outputs are generated by running code in the sandbox; no external validation of runtime behavior beyond the six-language manual audit is provided.
  • domain assumption The six-language manual audit generalizes to all 20 languages.
    Section 4.1: accuracy of 87.6% was computed on Python, C++, Java, JS, Go, and Shell; low-resource languages such as Racket and Elixir remain unverified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators." pith.science (2026). https://pith.science/paper/MLWZXDFU

@misc{pith2026250809101,
  author       = {Pith},
  title        = {Pith review of: AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MLWZXDFU}},
  note         = {Machine review of arXiv:2508.09101}
}
read the original abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities across various domains, with code generation emerging as a key area of focus. While numerous benchmarks have been proposed to evaluate their code generation abilities, these benchmarks face several critical limitations. First, they often rely on manual annotations, which are time-consuming and difficult to scale across different programming languages and problem complexities. Second, most existing benchmarks focus primarily on Python, while the few multilingual benchmarks suffer from limited difficulty and uneven language distribution. To address these challenges, we propose AutoCodeGen, an automated method for generating high-difficulty multilingual code generation datasets without manual annotations. AutoCodeGen ensures the correctness and completeness of test cases by generating test inputs with LLMs and obtaining test outputs through a multilingual sandbox, while achieving high data quality through reverse-order problem generation and multiple filtering steps. Using this novel method, we introduce AutoCodeBench, a large-scale code generation benchmark comprising 3,920 problems evenly distributed across 20 programming languages. It is specifically designed to evaluate LLMs on challenging, diverse, and practical multilingual tasks. We evaluate over 30 leading open-source and proprietary LLMs on AutoCodeBench and its simplified version AutoCodeBench-Lite. The results show that even the most advanced LLMs struggle with the complexity, diversity, and multilingual nature of these tasks. Besides, we introduce AutoCodeBench-Complete, specifically designed for base models to assess their few-shot code generation capabilities. We hope the AutoCodeBench series will serve as a valuable resource and inspire the community to focus on more challenging and practical multilingual code generation scenarios.

Figures

Figures reproduced from arXiv: 2508.09101 by the authors.

Figure 1
Figure 1. Tag and Language Distribution across our AutoCodeBench. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The overview of AutoCodeGen. It first generates code solution and the corresponding public/private test input functions based on multilingual code snippets (⃝1 ). They are concatenated and executed in a sandbox to obtain test outputs, which are then combined by the LLM into complete test functions (⃝2 ,⃝3 ,⃝4 ). Based on the code solution and test function, the LLM is prompted to generate accurate programming proble… view at source ↗
Figure 3
Figure 3. The performance comparison of different models across two language sets. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Performance drop of models on multi-logic problems (1,622) compared to full dataset. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Scaling laws for different models. We use DeepSeek-V3-0324 to identify all multi-logical problems in AutoCodeBench and evaluate model performance on them. The results, shown in [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Performance improvement across multi-turn refinement with sandbox feedback. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: AutoCodeBench leaderboard showing Pass@1 performance of various LLMs. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Category Distribution of Different Benchmarks. [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 10
Figure 10. Figure 10: The English prompt of annotation and critic. [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: AutoCodeBench-Lite leaderboard showing Pass@1 performance of various LLMs. [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]
Figure 12
Figure 12. Figure 12: The prompt of generating code solution. Due to the excessive length of the prompt, we have [PITH_FULL_IMAGE:figures/full_fig_p023_12.png]
Figure 13
Figure 13. Figure 13: The prompt of generating test function. Due to the excessive length of the prompt, we have [PITH_FULL_IMAGE:figures/full_fig_p024_13.png]
Figure 14
Figure 14. Figure 14: The prompt of generating programming problem. [PITH_FULL_IMAGE:figures/full_fig_p025_14.png]
Figure 15
Figure 15. Figure 15: The prompt of translating languages. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_15.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PERFOPT-Bench: Evaluating Coding Agents on Software Performance Optimization

    cs.SE 2026-07 conditional novelty 6.0 of 10

    Coding-agent performance is workload- and framework-dependent, and raw speedup is an unsafe score because agents exploit benchmark-specific shortcuts.

  2. From Prompt Optimization to Multi-Dimensional Credibility Evaluation: Enhancing Trustworthiness of Chinese LLM-Generated Liver MRI Reports -- with Preliminary Extension to Lung Cancer

    cs.AI 2025-10 conditional novelty 5.0 of 10

    A three-axis evaluation framework plus prompt-optimization study finds that instruction-rich prompts with 10–15 examples improve LLM-generated Chinese liver MRI report credibility, with Kimi-K2 and DeepSeek-V3 leading.

  3. Cross-Domain Hybrid OPD for Generalizable Search Agents

    cs.CL 2026-08 conditional novelty 4.0 of 10

    A search agent trained with search RL, then jointly trained with distillation from domain experts, recovers most general reasoning ability while keeping search performance, according to Tencent's internal evaluations.

Reference graph

Works this paper leans on

30 extracted references · 3 canonical work pages · cited by 3 Pith papers

  1. [2]

    Anthropic

    URL https: //arxiv.org/abs/2502.02737. Anthropic. Introducing claude 4,

    arXiv

  2. [5]

    Wei-Lin Chen, Zhepei Wei, Xinyu Zhu, Shi Feng, and Yu Meng

    URL https://arxiv.org/ abs/2107.03374. Wei-Lin Chen, Zhepei Wei, Xinyu Zhu, Shi Feng, and Yu Meng. Do llm evaluators prefer themselves for a reason?, 2025a. URL https://arxiv.org/abs/2504.03846. Xiancai Chen, Zhengwei Tao, Kechi Zhang, Changzhi Zhou, Xinyu Zhang, Wanli Gu, Yuanpeng He, Mengdi Zhang, Xunliang Cai, Haiyan Zhao, and Zhi Jin. Revisit self-deb...

    arXiv 2025

  3. [6]

    DeepSeek-AI

    URL https://cursor.com/en. DeepSeek-AI. Deepseek-v3 technical report, 2025a. URL https://arxiv.org/abs/2412.19437. DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025b. URL https://arxiv.org/abs/2501.12948. DeepSeek-AI, Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y. Wu, Yukun Li,...

    arXiv

  4. [8]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y

    URL https://arxiv.org/abs/2507.06261. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming – the rise of code intelligence,

    arXiv

  5. [9]

    URL https://arxiv.org/abs/2401.14196. Jiawei Guo, Ziming Li, Xueling Liu, Kaijing Ma, Tianyu Zheng, Zhouliang Yu, Ding Pan, Yizhi LI, Ruibo Liu, Yue Wang, Shuyue Guo, Xingwei Qu, Xiang Yue, Ge Zhang, Wenhu Chen, and Jie Fu. Codeeditorbench: Evaluating code editing capability of large language models,

    arXiv

  6. [10]

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al

    URL https: //arxiv.org/abs/2404.03543. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938,

    arXiv

  7. [11]

    URL https://arxiv.org/abs/2411.04905. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. Qwen2.5-coder technical...

    arXiv

  8. [12]

    Carlos E

    URL https://arxiv.org/abs/2406.00515. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?,

    arXiv

Show all 30 references
  1. [13]

    Kimi-Team

    URL https://arxiv.org/abs/2310.06770. Kimi-Team. Kimi k2: Open agentic intelligence,

    arXiv

  2. [14]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E

    URL https://arxiv.org/abs/2507.20534. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGO...

    arXiv

  3. [15]

    doi: 10.1126/science.abq1158

    ISSN 1095-9203. doi: 10.1126/science.abq1158. URL http://dx.doi.org/10.1126/science.abq1158. 15 Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Koce...

    doi

  4. [16]

    URL https://arxiv.org/abs/2402.16694. Shanghaoran Quan, Jiaxi Yang, Bowen Yu, Bo Zheng, Dayiheng Liu, An Yang, Xuancheng Ren, Bofei Gao, Yibo Miao, Yunlong Feng, Zekun Wang, Jian Yang, Zeyu Cui, Yang Fan, Yichang Zhang, Binyuan Hui, and Junyang Lin. Codeelo: Benchmarking compe...

    arXiv

  5. [17]

    URL https://arxiv.org/abs/2501.01257. Qwen. Qwen3-coder: Agentic coding in the world,

    arXiv

  6. [18]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al

    URL https://arxiv.org/abs/2412.15115. Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950,

    arXiv

  7. [19]

    URL https://arxiv.org/abs/2506.03524. Tencent. Hunyuan-turbos: Advancing large language models through mamba-transformer synergy and adaptive chain-of-thought,

    arXiv

  8. [20]

    Kaixin Wang, Tianlin Li, Xiaoyu Zhang, Chong Wang, Weisong Sun, Yang Liu, and Bin Shi

    URL https://arxiv.org/abs/2505.15431. Kaixin Wang, Tianlin Li, Xiaoyu Zhang, Chong Wang, Weisong Sun, Yang Liu, and Bin Shi. Software development life cycle perspective: A survey of benchmarks for code large language models and agents, 2025a. URL https://arxiv.org/abs/2505.052...

    2024

  9. [22]

    URL https://arxiv.org/abs/2312.14187. Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, Siyao Liu, Yongsheng Xiao, Liangqiang Chen, Yuyu Zhang, Jing Su, Tianyu Liu, Rui Long, Kai Shen, and Liang Xiang. Multi-...

    arXiv

  10. [23]

    URL https://arxiv.org/abs/2504.02605. Alexander Zhang, Marcus Dong, Jiaheng Liu, Wei Zhang, Yejie Wang, Jian Yang, Ge Zhang, Tianyu Liu, Zhongyuan Peng, Yingshui Tan, Yuanxing Zhang, Zhexu Wang, Weixun Wang, Yancheng He, Ken Deng, Wangchunshu Zhou, Wenhao Huang, and Zhaoxiang ...

    arXiv

  11. [24]

    Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue

    URL https://arxiv.org/abs/2405.04520. Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. OpenCodeInterpreter: Integrating code generation with execution and refinement. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.),...

    2024 arXiv

  12. [25]

    doi: 10.18653/v1/2024.findings-acl.762

    Association for Computational Linguis- tics. doi: 10.18653/v1/2024.findings-acl.762. URL https://aclanthology.org/2024.findings-acl.7 62/. Zihan Zheng, Zerui Cheng, Zeyu Shen, Shang Zhou, Kaiyuan Liu, Hansen He, Dongruixuan Li, Stanley Wei, Hangyi Hao, Jianzhu Yao, Peiyao Shen...

    2024 doi

  13. [26]

    URL https://arxiv.org/ abs/2506.11928. Zhipu. Glm-4.5: Reasoning, coding, and agentic abililties,

    arXiv

  14. [27]

    Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al

    URL https://arxiv.org/abs/2502.09183. Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406.11931,

    arXiv

  15. [28]

    URL https://arxiv.org/abs/2506.10481. Terry Yue Zhuo, Vu Minh Chien, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen GONG, James Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kadd...

    arXiv

  16. [29]

    __main__

    The results indicate that, despite the presence of some noisy data, our benchmark model still demonstrates high accuracy (87.6%). Further- more, even after removing the noise, the current SOTA model shows significant room for improvement (∆43.0), further validating the high di...

    2025

  17. [30]

    22 # Code Benchmark Construction TaskIn order to build the code benchmark, I need you to help me create a Python function, as well as two test functions

    21 20 30 40 50 60 Average Pass@1 Claude Opus 4 (Think) o3 (high) Grok-4 Claude Sonnet 4 (Think) Claude Opus 4 DeepSeek-R1-0528 o4-mini Claude Sonnet 4 Gemini2.5 Pro Qwen3-235B-A22B-Thinking-2507 GPT4.1 (2025-04-14) DeepSeek-V3-0324 Kimi-K2-0711-Preview GLM-4.5-enable Qwen3-235...

    2025

  18. [51]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388,

    arXiv

  19. [2021]

    URL https://arxiv.org/abs/2108.07732. Baidu. Ernie-x1-turbo-32k,

    arXiv

  20. [2022]

    URL https://arxiv.org/abs/2208.08227. Linzheng Chai, Shukai Liu, Jian Yang, Yuwei Yin, JinKe, Jiaheng Liu, Tao Sun, Ge Zhang, Changyu Ren, Hongcheng Guo, Noah Wang, Boyang Wang, Xianjie Wu, Bing Wang, Tongliang Li, Liqun Yang, Sufeng Duan, Zhaoxiang Zhang, and Zhoujun Li. Mcev...

    arXiv

  21. [2024]

    URL https://arxiv.org/abs/2406.11931. Gemini. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities,

    arXiv

  22. [2025]

    URL https://arxiv.org/abs/2504.04030. Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Mart´ın Bl´azquez, Guilherme Penedo, Lewis Tunstall, Andr´es Marafioti, Hynek Kydl ´ıˇcek, Agust´ın Piqueres Lajar ´ın, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguy...

    arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.