Pith. sign in

REVIEW 5 major objections 7 minor 1 cited by

Afterburner: Reinforcement Learning Facilitates Self-Improving Code Efficiency Optimization

T0 review · 5 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Reinforcement learning makes LLMs keep improving their own code's efficiency.

desk verdict The GRPO-vs-SFT/DPO efficiency result is real and worth taking seriously, but the benchmark's GPT-4o test-case quality and missing releases keep it from being a slam dunk. read the letter →

arxiv 2505.23387 v3 pith:Y73VMJFI submitted 2025-05-29 cs.SE cs.AI

Mingzhe Du , Luu Anh Tuan , Yue Liu , Yuhao Qing , Dong Huang , Xinyi He , Qian Liu , Zejun Ma
show 1 more author
See-kiong Ng
This is my paper · ORCID
classification cs.SEcs.AI
keywords codeefficiencyreinforcementlearningGRPOiterativeoptimizationexecutionfeedbacktest-timegenerationbenchmarkconstruction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models write correct code but often inefficient code. This paper claims that closing a generation-evaluation loop with reinforcement learning, rather than supervised fine-tuning or preference optimization, teaches a model to keep improving its own code's efficiency at inference time. On the authors' Venus benchmark and on APPS, their GRPO-trained optimizer raises pass@1 from 47% to 62% and raises the share of human submissions it beats on the combined efficiency metric from 31% to 45%, while SFT and DPO plateau. The claim matters because it identifies online execution feedback as the ingredient that turns test-time iteration into sustained self-improvement.

What carries the argument

The engine is the closed Iterative Optimization Framework: Afterburner generates candidate rewrites, Monolith executes them in isolated containers with CPU affinity and reports pass, time, peak memory, and the time-memory integral, and a greedy rule keeps the best code. GRPO training combines a format reward, a relative correctness reward, and a tanh-clipped efficiency-gain reward, with group-relative advantages letting the policy explore multiple rewrites per prompt and learn from their measured outcomes.

What would settle it

Run the same Afterburner-GRPO rollouts on a fresh set of validator-generated test cases from the same Venus problems, or on hidden test suites, and repeat each execution several times; if the efficiency rankings reverse or the pass@1 and BEYOND improvements disappear under remeasurement, the reported gains are an artifact of the particular test cases or of measurement noise.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the learning objective determines whether iterative optimization saturates or continues. Supervised fine-tuning learns surface rewrite patterns, DPO learns static preferences from offline pairs, and both exhaust their value within a few iterations. GRPO, trained online with rewards from an execution sandbox (format compliance, functional correctness relative to the previous solution, and a bounded relative efficiency gain), develops an adaptive improvement policy that keeps yielding gains across ten iterations. The headline evidence is pass@1 rising from 47% to 62% and the likelihood of outperforming human submissions in efficiency rising from 31% to 45%, with the GRPO model also producing the largest fraction of solutions that beat all human solutions on time, memory, and integral scores.

Load-bearing premise

Everything rests on the assumption that Monolith's runtime measurements on the synthesized test cases faithfully and stably represent real-world efficiency gains.

Editorial extensions

If this is right

  • A correctness-preserving efficiency optimizer can be deployed as a test-time loop: feed a model's first attempt back through the same policy repeatedly and keep the best version.
  • SFT and DPO are insufficient for this task; the paper predicts that offline-only alignment methods plateau on efficiency once their learned patterns or preferences are exhausted.
  • RL with execution feedback can improve correctness and efficiency together, since pass@1 rises alongside the BEYOND efficiency scores.
  • The reward design, based on relative rather than absolute efficiency gains, allows a small base model to surpass the efficiency of much larger proprietary models after enough iterations.
  • The sandbox plus validated test-case generators can supply training signal without ground-truth responses, so the RL loop can scale to new problems without human-written rewrites.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The correctness gains suggest that efficiency rewards act partly as a general self-correction signal; this could transfer to other code-quality dimensions such as readability or security whenever a measurable reward exists.
  • If the synthesized test cases are not representative of deployment inputs, GRPO would overfit to the validator; an immediate extension is to adversarially diversify or augment the test-case generators during training.
  • The framework's cost is many sandbox executions per problem; training a cheap proxy critic to predict Monolith's reward and prune unpromising rollouts before execution could make the loop practical for production codebases.
  • Because KL regularization is disabled in the reported GRPO setup, the exploration diversity that drives iteration may come at the cost of stability; adding a small KL or entropy schedule could make the gains more reproducible.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper introduces Afterburner, an iterative optimization framework that refines LLM-generated code through closed-loop generation and execution feedback from a Monolith sandbox. It compares three training strategies—SFT, DPO, and GRPO—on a new Venus dataset and on APPS, reporting that GRPO with execution feedback yields sustained improvements (pass@1 from 47% to 62% and BEYOND-I from 31% to 45%) while SFT and DPO plateau. The central claim is that online RL with empirical performance feedback enables test-time self-improvement of code efficiency.

Significance. If the empirical findings hold, the paper makes a useful contribution by demonstrating that online RL with execution feedback can drive iterative code-efficiency optimization, and by introducing a relatively large benchmark (Venus) and a sandbox (Monolith). The study includes ablations (Table 2) that support the contributions of feedback and original code, and it provides bootstrapped confidence intervals for several cross-sectional tables. However, the main threat to validity is that both the GRPO reward and the headline evaluation metrics rely on the same GPT-4o-generated test-case distribution, with no independent hold-out test suite; additionally, the APPS evaluation uses the official training split. These issues bear directly on whether the reported gains reflect genuine efficiency improvement or proxy optimization.

major comments (5)
  1. [§5 (Eq. 13) and §4 (Eqs. 7–8)] The headline PASS@1 and BEYOND improvements are measured on test cases generated by GPT-4o and validated only for output consistency across canonical human solutions (Appendix C.1). The same style of generated test cases provides the GRPO reward through Monolith. Because no independent hidden test suite is used (e.g., original platform tests or a separately hand-validated set), the observed gains may reflect specialization to the generated test distribution rather than genuine efficiency improvement. This is a load-bearing issue for the central claim that GRPO enables 'truly self-improve' code efficiency.
  2. [Appendix C.2 and Table 9/Figure 9] The APPS subset used for evaluation is drawn from the official APPS train split, not the test split. Evaluating on the train split is non-standard: these problems are public and likely seen during pre-training, and the additional filter for sufficiently many reference solutions may select an easier subset. The APPS results should be recomputed on the test split or clearly repositioned as a development-set analysis; as presented, they do not robustly support the claim that GRPO transfers to an external benchmark.
  3. [§6.2, Figure 5] The central iterative curves in Figure 5 (PASS@1 and BEYOND-{T,M,I}) are point estimates without confidence intervals or error bars, in contrast to Tables 1 and 9, which report 95% CIs via bootstrapping. Given runtime and memory measurement noise, the asserted differences between SFT/DPO saturation and GRPO continued improvement need uncertainty quantification before the comparison is statistically convincing.
  4. [§4 Eq. (8) and Appendix H] The reward function depends on Monolith time/memory/integral measurements, but no run-to-run variance analysis of those measurements is provided. Appendix G acknowledges 'inherent system noise' and Appendix H uses Docker and CPU affinity, yet the noise floor and its propagation through the tanh reward and the group advantage in Eq. (11) are never quantified. If the reward signal is noisy, the GRPO advantage estimates may be unreliable, which would directly affect whether the training procedure is the cause of the observed gains.
  5. [Reproducibility] The Venus dataset, the Monolith sandbox, and the trained Afterburner checkpoints are not released or linked in the manuscript. This prevents independent verification of the central claims, including the pass@1 and BEYOND improvements and the human-comparison results in Table 3. Releasing the test-case generators, evaluation harness, and model weights, or providing a public demo, would be necessary to confirm the findings.
minor comments (7)
  1. [§3.2, Eq. (3)] The word 'paassed' in the equation is a typo and should read 'passed'.
  2. [§4, DPO dataset] The text says the DPO dataset has 'approximately 13.3K instances per efficiency instruction type' while the total is given as 90,864; this implies about 30.3K per type. Please reconcile the numbers.
  3. [§5 and Appendix C.1/Table 7] The Python task counts are inconsistent: Section 5 states the Venus Python subset contains 2,181 problems, while the curation pipeline (Appendix C.1) reports 1,284 curated problems (984 train + 300 test) and Table 7 lists 2,181 Python train tasks. The source of this discrepancy should be clarified.
  4. [Algorithm 1] The output description says 'Improved code C_out^0', but the algorithm actually returns C_in^{Niter}; the text should match the pseudocode.
  5. [Symbol list] The symbol table entry for PR(x, D) mentions an 'implicit' adjustment that is never defined; Eq. (12) should be stated in a self-contained way.
  6. [Appendix H] The units for E_upper (90, 1048576, 94371840) are not stated; please specify whether the memory value is in kilobytes, bytes, or another unit.
  7. [References / naming] The benchmark is referred to inconsistently as 'PIE', 'PIE4Effi', and 'PIE4PERF' across Sections 1 and 2 and reference [50]; please standardize the name.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the GRPO-vs-SFT/DPO comparison is an empirical result on disjoint train/test splits; sharing Monolith between training and evaluation is a standard RL loop, not a derivation from the claim.

full rationale

The paper makes no formal derivation from first principles; its central claims are empirical. The GRPO reward (Eqs. 6-9) uses Monolith measurements on the training tasks (984 Venus train tasks), while the headline PASS@1 and BEYOND metrics (Eqs. 12-13) are computed on the held-out 300-task Venus test set, so the training objective and the evaluation are not evaluated on the same test cases by construction. Moreover, the reward is a tanh-scaled relative improvement in absolute time/memory/integral, whereas BEYOND is a percentile rank against human solutions; no equation in the paper identifies these two quantities. The self-citations to Mercury [14] and EffiBench [21] motivate the design of Venus and the percentile-rank metric, but the central RL comparison does not reduce to those citations, and the paper additionally evaluates on the external APPS benchmark. The weaker assumption about GPT-4o-generated test cases being unrepresentative or noisy is a legitimate correctness/validity concern, not a circular-reasoning concern, because nothing in the paper defines the predicted outcome in terms of the training signal.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

No new physical or theoretical entities are postulated. Afterburner, Monolith, and Venus are system components, a sandbox, and a dataset, not invented scientific entities.

free parameters (2)
  • Reward weights beta_f, beta_e, beta_c = 0.2, 0.3, 0.5
    Eq. (9) and Appendix E.5; hand-chosen balance among format, correctness, and efficiency rewards in GRPO.
  • Efficiency clipping upper bounds E_upper = 90 s, 1048576 KB, 94371840 integral
    Appendix E.5; hand-set to sandbox timeout and memory limits, directly shape Refficiency.
assumptions (5)
  • domain assumption Monolith runtime measurements are stable enough to serve as RL rewards
    Section 3.2 and Appendix H assume Docker isolation, CPU affinity, and repeated evaluation control noise; no validation of reward-signal stability is presented.
  • domain assumption GPT-4o-generated test cases are representative of problem difficulty
    Appendix C.1 validates consistency of outputs across human solutions, but not input-size distribution or discriminative power.
  • domain assumption Human LeetCode solutions form a fair reference distribution for efficiency
    Section 5 defines BEYOND metrics as percentile ranks against Venus human solutions; if this pool is biased, the metrics are biased.
  • domain assumption Training on Venus train generalizes to Venus test and APPS subset
    Section 6.2 evaluates held-out tasks; possible leakage through pretraining corpora is not analyzed.
  • standard math GRPO objective (Eq. 10) and tanh reward shaping behave as described
    Taken from DeepSeekMath GRPO and standard RL; no formal guarantees invoked.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Afterburner: Reinforcement Learning Facilitates Self-Improving Code Efficiency Optimization." pith.science (2026). https://pith.science/paper/Y73VMJFI

@misc{pith2026250523387,
  author       = {Pith},
  title        = {Pith review of: Afterburner: Reinforcement Learning Facilitates Self-Improving Code Efficiency Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y73VMJFI}},
  note         = {Machine review of arXiv:2505.23387}
}
read the original abstract

Large Language Models (LLMs) generate functionally correct solutions but often fall short in code efficiency, a critical bottleneck for real-world deployment. In this paper, we introduce a novel test-time iterative optimization framework to address this, employing a closed-loop system where LLMs iteratively refine code based on empirical performance feedback from an execution sandbox. We explore three training strategies: Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Group Relative Policy Optimization (GRPO). Experiments on our Venus dataset and the APPS benchmark show that SFT and DPO rapidly saturate in efficiency gains. In contrast, GRPO, using reinforcement learning (RL) with execution feedback, continuously optimizes code performance, significantly boosting both pass@1 (from 47% to 62%) and the likelihood of outperforming human submissions in efficiency (from 31% to 45%). Our work demonstrates effective test-time code efficiency improvement and critically reveals the power of RL in teaching LLMs to truly self-improve code efficiency.

Figures

Figures reproduced from arXiv: 2505.23387 by the authors.

Figure 1
Figure 1. Comparison of iterative optimization performance between a [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Inference Workflow of the Iterative Optimization Framework ( [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Illustration of task-level efficiency metrics. Functional Correctness Ensuring functional correctness is a prerequisite for code genera￾tion models. Following the evaluation paradigm in Codex [9], we employ the PASS@1 = Npassed/Ntotal score to assess the global func￾tional correctness, where Npassed is the number of passed generations and Ntotal is the total num￾ber of test tasks. PR(x, D) = 1 |D| P d∈D 1[d ≥ x]. (1… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Illustration of the training pipeline of [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Iterative Optimization with an Efficient Instruction [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Beginning with 3,535 problems from LeetCode [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 6
Figure 6. Figure 6: Pipeline for constructing the [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: An Example in Venus Python Subset. programming language, interface with the test harness exclusively via standard input (stdin) and standard output (stdout). Test inputs are provided as text streams via stdin, and the solution’s output is captured from stdout. This tex…
Figure 8
Figure 8. Figure 8: Selection procedure for the APPS subset used in our benchmark. Beginning with the official APPS training split (5,000 problems), we discard problems that lack a sufficient number of accepted reference solutions, yielding 2,803 problems in the final dataset. targeting t…
Figure 9
Figure 9. Figure 9: Iterative Optimization Performance on APPS. [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ParEVO: Synthesizing Code for Irregular Data: High-Performance Parallelism through Agentic Evolution

    cs.LG 2026-03 conditional novelty 6.0 of 10

    A pipeline of fine-tuned LLMs plus evolutionary repair produces parallel code for irregular workloads that the authors measure as ~13.6x faster on graph kernels and up to 4.1x faster than a human expert on Maximal Ind...

Reference graph

Works this paper leans on

67 extracted references · 22 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Santacoder: don’t reach for the stars!arXiv preprint arXiv:2301.03988, 2023

    Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Car- los Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. Santacoder: don’t reach for the stars!arXiv preprint arXiv:2301.03988, 2023

    arXiv 2023

  3. [3]

    An orchestrated survey of methodologies for automated software test case generation.Journal of systems and software, 86(8):1978–2001, 2013

    Saswat Anand, Edmund K Burke, Tsong Yueh Chen, John Clark, Myra B Cohen, Wolfgang Grieskamp, Mark Harman, Mary Jean Harrold, Phil McMinn, Antonia Bertolino, et al. An orchestrated survey of methodologies for automated software test case generation.Journal of systems and software, 86(8):1978–2001, 2013

    work page 1978

  4. [4]

    Introducing claude 3.5 sonnet, 6 2024

    Anthropic. Introducing claude 3.5 sonnet, 6 2024

    work page 2024

  5. [5]

    Claude 3.7 sonnet and claude code, 2 2025

    Anthropic. Claude 3.7 sonnet and claude code, 2 2025

    work page 2025

  6. [6]

    Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    arXiv 2021

  7. [7]

    Code alpaca: An instruction-following llama model for code generation

    Sahil Chaudhary. Code alpaca: An instruction-following llama model for code generation. https://github.com/sahil280114/codealpaca, 2023

    2023

  8. [8]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    work page 2021

Show all 67 references
  1. [9]

    Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  2. [10]

    An introduction to algorithms and the big o notation.Introduction to Programming with Fortran: With Coverage of Fortran 90, 95, 2003, 2008 and 77, pages 359–364, 2015

    Ian Chivers, Jane Sleightholme, Ian Chivers, and Jane Sleightholme. An introduction to algorithms and the big o notation.Introduction to Programming with Fortran: With Coverage of Fortran 90, 95, 2003, 2008 and 77, pages 359–364, 2015

    2003

  3. [11]

    Mhpp: Exploring the capabilities and limitations of language models beyond basic code generation, 2024

    Jianbo Dai, Jianqiao Lu, Yunlong Feng, Dong Huang, Guangtao Zeng, Rongju Ruan, Ming Cheng, Haochen Tan, and Zhijiang Guo. Mhpp: Exploring the capabilities and limitations of language models beyond basic code generation, 2024

    2024

  4. [12]

    Docker.lınea].[Junio de 2017]

    Inc Docker et al. Docker.lınea].[Junio de 2017]. Disponible en: https://www. docker. com/what- docker, 2020

    2017

  5. [13]

    Stepcoder: Improve code generation with reinforcement learning from compiler feedback.arXiv preprint arXiv:2402.01391, 2024

    Shihan Dou, Yan Liu, Haoxiang Jia, Limao Xiong, Enyu Zhou, Wei Shen, Junjie Shan, Caishuang Huang, Xiao Wang, Xiaoran Fan, et al. Stepcoder: Improve code generation with reinforcement learning from compiler feedback.arXiv preprint arXiv:2402.01391, 2024. 10

    2024 arXiv

  6. [14]

    Mercury: A code efficiency benchmark for code large language models.Advances in Neural Information Processing Systems, 37, 2024

    Mingzhe Du, Luu Anh Tuan, Bin Ji, Qian Liu, and See-Kiong Ng. Mercury: A code efficiency benchmark for code large language models.Advances in Neural Information Processing Systems, 37, 2024

    2024

  7. [15]

    Chapman and Hall/CRC, 1994

    Bradley Efron and Robert J Tibshirani.An introduction to the bootstrap. Chapman and Hall/CRC, 1994

    1994

  8. [16]

    Rlef: Grounding code llms in execution feedback with reinforcement learning.arXiv preprint arXiv:2410.02089, 2024

    Jonas Gehring, Kunhao Zheng, Jade Copet, Vegard Mella, Quentin Carbonneaux, Taco Cohen, and Gabriel Synnaeve. Rlef: Grounding code llms in execution feedback with reinforcement learning.arXiv preprint arXiv:2410.02089, 2024

    2024 arXiv

  9. [17]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  10. [18]

    Measuring coding challenge competence with apps.NeurIPS, 2021

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with apps.NeurIPS, 2021

    2021

  11. [19]

    Codecot: Tackling code syntax errors in cot reasoning for code generation, 2024

    Dong Huang, Qingwen Bu, Yuhao Qing, and Heming Cui. Codecot: Tackling code syntax errors in cot reasoning for code generation, 2024

    2024

  12. [20]

    Effilearner: Enhancing efficiency of generated code via self-optimization.Advances in Neural Information Processing Systems, 37:84482–84522, 2024

    Dong Huang, Jianbo Dai, Han Weng, Puzhen Wu, Yuhao Qing, Heming Cui, Zhijiang Guo, and Jie Zhang. Effilearner: Enhancing efficiency of generated code via self-optimization.Advances in Neural Information Processing Systems, 37:84482–84522, 2024

    2024

  13. [21]

    Effibench: Benchmarking the efficiency of automatically generated code.Advances in Neural Information Processing Systems, 37:11506–11544, 2024

    Dong Huang, Yuhao Qing, Weiyi Shang, Heming Cui, and Jie Zhang. Effibench: Benchmarking the efficiency of automatically generated code.Advances in Neural Information Processing Systems, 37:11506–11544, 2024

    2024

  14. [22]

    Effi-code: Unleashing code efficiency in language models

    Dong Huang, Guangtao Zeng, Jianbo Dai, Meng Luo, Han Weng, Yuhao Qing, Heming Cui, Zhijiang Guo, and Jie M Zhang. Effi-code: Unleashing code efficiency in language models. arXiv preprint arXiv:2410.10209, 2024

    2024 arXiv

  15. [23]

    Bias testing and mitigation in llm-based code generation.ACM Transactions on Software Engineering and Methodology, 2024

    Dong Huang, Jie M Zhang, Qingwen Bu, Xiaofei Xie, Junjie Chen, and Heming Cui. Bias testing and mitigation in llm-based code generation.ACM Transactions on Software Engineering and Methodology, 2024

    2024

  16. [24]

    Rethinking the influence of source code on test case generation.arXiv preprint arXiv:2409.09464, 2024

    Dong Huang, Jie M Zhang, Mingzhe Du, Mark Harman, and Heming Cui. Rethinking the influence of source code on test case generation.arXiv preprint arXiv:2409.09464, 2024

    2024 arXiv

  17. [25]

    Zhang, Michael Luck, Qingwen Bu, Yuhao Qing, and Heming Cui

    Dong Huang, Jie M. Zhang, Michael Luck, Qingwen Bu, Yuhao Qing, and Heming Cui. Agentcoder: Multi-agent-based code generation with iterative testing and optimisation, 2024

    2024

  18. [26]

    Siming Huang, Tianhao Cheng, J. K. Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J. Yang, J. H. Liu, Chenchen Zhang, Linzheng Chai, Ruifeng Yuan, Zhaoxiang Zhang, Jie Fu, Qian Liu, Ge Zhang, Zili Wang, Yuan Qi, Yinghui Xu, and Wei Chu. Opencoder: The open cookbook for top-tier code...

    2024 arXiv

  19. [27]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jia- jun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report.arXiv preprint arXiv:2409.12186, 2024

    2024 arXiv

  20. [28]

    Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez

    Md. Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez. Mapcoder: Multi-agent code generation for competitive problem solving, 2024

    2024

  21. [29]

    Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974, 2024

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  22. [30]

    A survey on large language models for code generation.arXiv preprint arXiv:2406.00515, 2024

    Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation.arXiv preprint arXiv:2406.00515, 2024

    2024 arXiv

  23. [31]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. InProceedings of the ACM SIGOPS 29th Symposium on Operating System...

    2023

  24. [32]

    Coderl: Mastering code generation through pretrained models and deep reinforcement learning

    Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 35:21314–21328, 2022. 11

    2022

  25. [33]

    Competition-level code generation with alphacode.Science, 378(6624):1092–1097, 2022

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode.Science, 378(6624):1092–1097, 2022

    2022

  26. [34]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  27. [35]

    Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  28. [36]

    Evaluating language models for efficient code generation.arXiv preprint arXiv:2408.06450, 2024

    Jiawei Liu, Songrun Xie, Junhao Wang, Yuxiang Wei, Yifeng Ding, and Lingming Zhang. Evaluating language models for efficient code generation.arXiv preprint arXiv:2408.06450, 2024

    2024 arXiv

  29. [37]

    Refining chatgpt-generated code: Characterizing and mitigating code quality issues.ACM Transactions on Software Engineering and Methodology, 33(5):1–26, 2024

    Yue Liu, Thanh Le-Cong, Ratnadira Widyasari, Chakkrit Tantithamthavorn, Li Li, Xuan-Bach D Le, and David Lo. Refining chatgpt-generated code: Characterizing and mitigating code quality issues.ACM Transactions on Software Engineering and Methodology, 33(5):1–26, 2024

    2024

  30. [38]

    Starcoder 2 and the stack v2: The next generation.arXiv preprint arXiv:2402.19173, 2024

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Noua- mane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder 2 and the stack v2: The next generation.arXiv preprint arXiv:2402.19173, 2024

    2024 arXiv

  31. [39]

    Wizardcoder: Empowering code large language models with evol-instruct.arXiv preprint arXiv:2306.08568, 2023

    Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct.arXiv preprint arXiv:2306.08568, 2023

    2023 arXiv

  32. [40]

    The llama 4 herd: The beginning of a new era of natively multimodal ai innovation

    AI Meta. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation. https://ai. meta. com/blog/llama-4-multimodal-intelligence/, checked on, 4(7):2025, 2025

    2025

  33. [41]

    Octopack: Instruction tuning code large language models.arXiv preprint arXiv:2308.07124, 2023

    Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre. Octopack: Instruction tuning code large language models.arXiv preprint arXiv:2308.07124, 2023

    2023 arXiv

  34. [42]

    Codegen: An open large language model for code with multi-turn program synthesis.arXiv preprint arXiv:2203.13474, 2022

    Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis.arXiv preprint arXiv:2203.13474, 2022

    2022 arXiv

  35. [43]

    Introducing openai o3 and o4-mini, 4 2025

    OpenAI. Introducing openai o3 and o4-mini, 4 2025

    2025

  36. [44]

    Zhang, Heming Cui, Siu-Ming Yiu, Dong Huang, See-Kiong Ng, and Luu Anh Tuan

    Yuhao Qing, Boyu Zhu, Mingzhe Du, Zhijiang Guo, Terry Yue Zhuo, Qianru Zhang, Jie M. Zhang, Heming Cui, Siu-Ming Yiu, Dong Huang, See-Kiong Ng, and Luu Anh Tuan. Effibench- x: A multi-language benchmark for measuring efficiency of llm-generated code, 2025

    2025

  37. [45]

    How efficient is llm-generated code? a rigorous & high-standard benchmark.arXiv preprint arXiv:2406.06647, 2024

    Ruizhong Qiu, Weiliang Will Zeng, James Ezick, Christopher Lott, and Hanghang Tong. How efficient is llm-generated code? a rigorous & high-standard benchmark.arXiv preprint arXiv:2406.06647, 2024

    2024 arXiv

  38. [46]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728–53741, 2023

    2023

  39. [47]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  40. [48]

    Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256, 2024

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256, 2024

    2024 arXiv

  41. [49]

    Efficient and green large language models for software engineering: Literature review, vision, and the road ahead.arXiv preprint arXiv:2404.04566, 2024

    Jieke Shi, Zhou Yang, and David Lo. Efficient and green large language models for software engineering: Literature review, vision, and the road ahead.arXiv preprint arXiv:2404.04566, 2024

    2024 arXiv

  42. [50]

    Learning performance-improving code edits.arXiv preprint arXiv:2302.07867, 2023

    Alexander Shypula, Aman Madaan, Yimeng Zeng, Uri Alon, Jacob Gardner, Milad Hashemi, Graham Neubig, Parthasarathy Ranganathan, Osbert Bastani, and Amir Yazdanbakhsh. Learning performance-improving code edits.arXiv preprint arXiv:2302.07867, 2023

    2023 arXiv

  43. [51]

    Coderosetta: Pushing the boundaries of unsupervised code translation for parallel programming.arXiv preprint arXiv:2410.20527, 2024

    Ali TehraniJamsaz, Arijit Bhattacharjee, Le Chen, Nesreen K Ahmed, Amir Yazdanbakhsh, and Ali Jannesari. Coderosetta: Pushing the boundaries of unsupervised code translation for parallel programming.arXiv preprint arXiv:2410.20527, 2024. 12

    arXiv

  44. [52]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  45. [53]

    Ecco: Can we improve model-generated code efficiency without sacrificing functional correctness?arXiv preprint arXiv:2407.14044, 2024

    Siddhant Waghjale, Vishruth Veerendranath, Zora Zhiruo Wang, and Daniel Fried. Ecco: Can we improve model-generated code efficiency without sacrificing functional correctness?arXiv preprint arXiv:2407.14044, 2024

    2024 arXiv

  46. [54]

    Enhancing code llms with reinforcement learning in code generation.arXiv preprint arXiv:2412.20367, 2024

    Junqiao Wang, Zeng Zhang, Yangfan He, Yuyang Song, Tianyu Shi, Yuchen Li, Hengyuan Xu, Kunyu Wu, Guangwu Qian, Qiuwu Chen, et al. Enhancing code llms with reinforcement learning in code generation.arXiv preprint arXiv:2412.20367, 2024

    2024 arXiv

  47. [55]

    Finetuned language models are zero-shot learners.arXiv preprint arXiv:2109.01652, 2021

    Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners.arXiv preprint arXiv:2109.01652, 2021

    2021 arXiv

  48. [56]

    Magicoder: Empow- ering code generation with OSS-instruct

    Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Empow- ering code generation with OSS-instruct. InProceedings of the 41st International Conference on Machine Learning, volume 235 ofProceedings of Machine Learning Research, pages 52632–52657. PMLR...

    2024

  49. [57]

    Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models.arXiv preprint arXiv:2408.00724, 2024

    Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models.arXiv preprint arXiv:2408.00724, 2024

    2024 arXiv

  50. [58]

    Wizardlm: Empowering large language models to follow complex instructions

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023

    2023 arXiv

  51. [59]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  52. [60]

    Llm4effi: Leveraging large language models to enhance code efficiency and correctness

    Tong Ye, Weigang Huang, Xuhong Zhang, Tengfei Ma, Peiyu Liu, Jianwei Yin, and Wenhai Wang. Llm4effi: Leveraging large language models to enhance code efficiency and correctness. arXiv preprint arXiv:2502.18489, 2025

    2025 arXiv

  53. [61]

    Focused-dpo: Enhancing code generation through focused preference optimization on error-prone points.arXiv preprint arXiv:2502.11475, 2025

    Kechi Zhang, Ge Li, Jia Li, Yihong Dong, and Zhi Jin. Focused-dpo: Enhancing code generation through focused preference optimization on error-prone points.arXiv preprint arXiv:2502.11475, 2025

    2025

  54. [62]

    A systematic literature review on large language models for automated program repair

    Quanjun Zhang, Chunrong Fang, Yang Xie, YuXiang Ma, Weisong Sun, Yun Yang, and Zhenyu Chen. A systematic literature review on large language models for automated program repair. arXiv preprint arXiv:2405.01466, 2024

    2024

  55. [63]

    Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x

    Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Lei Shen, Zihan Wang, Andi Wang, Yang Li, et al. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x. InProceedings of the 29th ACM SIGKDD Conference on Knowledge Dis...

    2023

  56. [64]

    Llamafactory: Unified efficient fine-tuning of 100+ language models

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System ...

    2024

  57. [65]

    Debug like a human: A large language model debugger via verifying runtime execution step-by-step, 2024

    Li Zhong, Zilong Wang, and Jingbo Shang. Debug like a human: A large language model debugger via verifying runtime execution step-by-step, 2024

    2024

  58. [66]

    Bigcodebench: Bench- marking code generation with diverse function calls and complex instructions.arXiv preprint arXiv:2406.15877, 2024

    Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Bench- marking code generation with diverse function calls and complex instructions.arXiv preprint arXiv:2406.15877, 2024

    2024 arXiv

  59. [67]

    <thinking> thing_content </thinking> <solution> solution_content </solution>

    Edward E Zukoski and GC Oates. Afterburners.Aerothermodynamics of aircraft engine components, 1985. 13 A Limitations While Afterburner demonstrates effective efficiency optimization for competition-level program- ming tasks, its extension to larger, real-world software enginee...

    1985

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.