Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Large Language Model Guided Self-Debugging Code Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read PyCapsule, a two-agent pipeline of a programmer LLM and a deterministic executor, claims state-of-the-art success rates on HumanEval, HumanEval-ET, and BigCodeBench(full) while using far fewer API calls than multi-agent rivals.

desk verdict PyCapsule is a genuinely efficient two-agent code-generation pipeline, but the SOTA claims rest on baselines the authors themselves couldn't reproduce, so the paper needs rewriting more than rethinking. read the letter →

arxiv 2502.02928 v2 pith:IKZ45LTA submitted 2025-02-05 cs.SE cs.AI

Muntasir Adnan , Zhiwei Xu , Carlos C. N. Kuhn This is my paper · ORCID
classification cs.SEcs.AI
keywords Pythoncodegenerationself-debuggingLLMagentstwo-agentpipelineprogrammingHumanEvalBigBenchexecutionsafety
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that high-accuracy LLM code generation does not require a crowd of cooperating agents. It proposes PyCapsule, a pipeline in which a programmer agent writes and repairs Python functions while a deterministic executor agent runs them in a Docker container and returns structured error feedback. On HumanEval, HumanEval-ET, and BigCodeBench(full), the authors report success rates higher by up to 5.7, 10.3, and 24.4 percentage points than earlier frameworks, with at most six LLM calls per problem. The authors also report that the normalized contribution of each additional self-debugging attempt decays exponentially, which leads them to cap debugging at five attempts. If these results hold, they matter because strong code generation would become cheaper, more lightweight, and more reliably deployed.

What carries the argument

The load-bearing mechanism is the two-agent self-debugging loop: a programmer agent (an LLM prompted with chain-of-thought reasoning and a fixed persona) alternates between generating a first solution and repairing it in fix mode, while an executor agent validates each attempt by running the code in a Docker container against the benchmark's test cases. Three deterministic modules support the loop: a signature converter that derives a function name, signature, and example call from the first test case; an example call detector that deletes embedded function invocations before execution; and an error handler that filters tracebacks, labels the error type, and truncates verbose failures such as recursion errors. The fix-mode prompt is fed only the original problem statement, the prior solution attempt, and the processed error message, a deliberately one-step context choice that the paper says prevents the performance drop observed with longer conversation histories.

What would settle it

Re-running PyCapsule, AgentCoder, MapCoder, and LDB on the same HumanEval, HumanEval-ET, MBPP, and BigCodeBench splits inside the same Docker-based harness and temperature settings would settle the claim; the headline margins collapse if AgentCoder's HumanEval score is re-measured above 96.5, its HumanEval-ET score above 96.3, or if PyCapsule's BigCodeBench result drops toward the 41.0% standalone baseline.

Watch

Extended reading notes

Core claim

The paper's central claim is that PyCapsule achieves state-of-the-art success rates on HumanEval, HumanEval-ET, and BigCodeBench(full) with a much smaller architecture than the multi-agent systems it compares against. The pipeline pairs a programmer agent, an LLM that operates in generation mode or fix mode, with an executor agent that is not an LLM but a Docker container running the generated code against the benchmark's tests. Three deterministic modules replace the extra agents used by other frameworks: the signature converter infers a function signature from the first test case without revealing expected outputs, the example call detector removes stray function calls that could loop or execute unsafely, and the error-handling module compresses tracebacks into short, type-labelled natural-language feedback. The paper also reports a secondary empirical finding: the normalized independent influence of each debugging attempt follows an exponential decay, approaching zero by the third attempt for GPT-4-Preview-1106 and by the fifth for Qwen2.5-Coder-7B-Instruct, which justifies the five-attempt maximum. On MBPP and MBPP-ET the reported numbers are strong but not the highest, a gap the paper attributes to its deliberately uniform prompting policy across datasets.

Load-bearing premise

The headline results assume the previously published success rates of AgentCoder, MapCoder, and LDB are accurate and directly comparable, even though the authors state they could not reproduce AgentCoder's numbers and caution against direct comparisons in their table.

Editorial extensions

If this is right

  • A two-agent design with deterministic modules can match or beat multi-agent frameworks on three widely used benchmarks while making at most six LLM API calls per problem, compared with up to 17 for MapCoder on HumanEval.
  • The exponential decay of per-attempt debugging gains means most of the accuracy improvement comes from the first one or two repairs, so running long debugging loops is wasteful.
  • A 7-billion-parameter model under PyCapsule reaches 94.1% on HumanEval and 65.4% on BigCodeBench(full), exceeding the standalone scores of much larger models, so pipeline structure can substitute for model scale.
  • Deterministic preprocessing and error filtering can replace LLM agents for signature inference, execution-safety checking, and traceback simplification without hurting accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Since the authors could not reproduce AgentCoder's reported scores and advise caution in comparing with them, the headline margins on HumanEval and HumanEval-ET should be read as pending a direct head-to-head rerun; the BigCodeBench gain against the base Qwen model does not depend on unverified baselines.
  • The exponential decay curve the paper fits could serve as a general efficiency metric for self-debugging pipelines: the per-attempt normalized success rate of any agent system can be plotted against attempt number and compared across models and frameworks, though the paper does not itself propose this as a standard measure.
  • The same two-agent loop and deterministic modules should transfer to other base LLMs beyond the three tested, including open-weight models, which would provide a direct test of whether the architecture or the specific model is responsible for the gains.
  • The example-call detector's role as a safety layer suggests the pipeline could extend to executing generated code in untrusted or user-supplied environments, where stripping embedded calls before running tests is a security measure rather than just an accuracy fix.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes PyCapsule, a two-agent framework for Python code generation that combines a programmer LLM agent with a deterministic executor agent and three rule-based modules (signature converter, example call detector, error handler). The system is evaluated on HumanEval, HumanEval-ET, MBPP, MBPP-ET, and BigCodeBench(full) using GPT-3.5-Turbo-1106, GPT-4-Preview-1106, and Qwen2.5-Coder-7B-Instruct. The abstract claims state-of-the-art success rates on HumanEval, HumanEval-ET, and BigCodeBench(full), with improvements of up to 5.7%, 10.3%, and 24.4% respectively, along with reduced token usage and fewer API calls compared to multi-agent baselines. The paper also reports an exponential decay in per-attempt debugging effectiveness, which it uses to justify a maximum of five self-debugging attempts.

Significance. If the empirical claims held, the paper would make a useful contribution: it shows that a lightweight two-agent pipeline with deterministic modules can be competitive with much heavier multi-agent frameworks, and it provides careful measurements of token usage, API call counts, and per-attempt debugging success across three model families. The authors are transparent about the reproducibility trouble with the AgentCoder baseline, which is commendable. However, the central state-of-the-art claim is not supported by the evidence as presented: the key comparisons rely on baseline numbers the authors could not reproduce, the HumanEval lead is within statistical noise, and the BigCodeBench improvement is measured against the base model rather than against a state-of-the-art method. The efficiency analysis and the debugging-decay characterization are interesting and potentially publishable, but the headline contribution needs to be reframed or re-evaluated with controlled baselines.

major comments (4)
  1. [§4.2, Table 1] The headline claim of state-of-the-art results on HumanEval-ET and HumanEval depends on AgentCoder's published numbers, yet the manuscript's own Table 1 footnote states: 'Despite multiple attempts, we were unable to reproduce the reported AgentCoder results... We include their reported numbers for completeness but advise caution in direct comparisons.' Because the 10.3% HumanEval-ET gap (96.3 vs. 86.0) and the HumanEval gap (96.5 vs. 96.3) are both measured against these unverified numbers, the central SOTA claim is unsupported. The authors should either run the baselines in a controlled setting (identical prompts, decoding, and test harness) or explicitly limit the claim to 'competitive with high efficiency' rather than 'state-of-the-art'.
  2. [§4.2, Table 1] The HumanEval result of PyCapsule (96.5±0.7 over three repeats) versus AgentCoder (96.3) is not statistically meaningful: the difference is 0.2 percentage points, well within one standard error of the mean. The abstract's 'up to 5.7% improvement on HumanEval' is not tied to a specific baseline, and the comparison to the listed AgentCoder value does not support such a claim. A significance test or a clear statement that the difference is within noise is needed.
  3. [Abstract, §4.2, Table 1] The claimed 24.4% improvement on BigCodeBench(full) is computed against the Qwen2.5-Coder-7B-Instruct model's zero-shot baseline of 41.0% (Table 1), not against any prior multi-agent or self-debugging method. The abstract's phrasing 'compared to the state-of-art methods' is therefore misleading for this benchmark. The authors should either compare against published BigCodeBench results from other methods or restrict the claim to 'improvement over the base model'.
  4. [§3.1, §4.2] The maximum of five self-debugging attempts is justified by an exponential decay fit to debugging-effectiveness data, but the data were collected under that same five-attempt cap. The fitted curve I(x)=a·e^{-bx} cannot establish that effectiveness 'approaches close to zero after five attempts,' because no attempt beyond five was ever observed. This is a circular justification for a design choice that shapes all reported results. The authors should either present the cap as a pragmatic engineering choice or provide a sensitivity analysis with a larger cap on a subset of problems.
minor comments (5)
  1. [Appendix B, Table 3] Table 3 lists 'MapCoder GPT-3.5 HumanEval 48.1,' which duplicates the 'Direct 48.1' entry in Table 1; either the attribution is inconsistent or the baseline label needs clarification.
  2. [§4.2, Figure 3] The text says the exponential decay fits have R² 'close to one' for every model and dataset, but no R² values are reported. Please include them in Figure 3 or in a table so the claim can be verified.
  3. [Appendix F] The system prompts contain small typos and formatting inconsistencies, e.g., 'pyhton-dotenv' instead of 'python-dotenv' and an extra blank line before the '### Code' section. Please proofread the appendix.
  4. [§3.1] The statement that retaining only the most recent problem-solution pair 'aligns with Markov Decision Process (MDP)' is loose: the design is a context-management heuristic, and the MDP analogy does not by itself justify it. Rephrase to avoid implying a formal MDP formulation.
  5. [Header, References] The manuscript uses an ACM journal template with '2018' in the header and placeholder 'Received XX; revised XX; accepted XX' lines; these should be cleaned up before submission.

Circularity Check

1 steps flagged · score 2.0 of 10

Mild in-sample justification of the five-attempt cap; the central benchmark evaluation remains self-contained.

  1. fitted input called prediction [Section 3.1; Section 4.2 (Influence of Self-debugging Attempts); Algorithm 1 in Appendix D]
    "Based on preliminary experiments that showed diminishing returns for additional debugging iterations, the programmer agent is limited to a maximum of five self-debugging attempts. This limit was empirically determined through exponential decay analysis of debugging effectiveness (see 4.2). ... We fitted the influence data to an exponential decay function and calculated the goodness of fit, obtaining R2, which was close to one for every model and dataset, confirming an exponential decay pattern in debugging effectiveness. ..."

    The decay curve is fitted to I_i = S_i/N_i values computed from runs of Algorithm 1, whose input already sets the maximum number of self-debugging attempts to N=5. Thus the data contain no observations beyond attempt five; the statement that 'effectiveness approaches close to zero after five attempts' is an in-sample extrapolation of the fitted curve, not an independent empirical finding. The same fitted curve is then used as the justification for retaining the five-attempt cap, and the headline Table 1 numbers are the outputs of the pipeline running under that cap. The cap is therefore a parameter derived from the very benchmark runs whose success rates are reported, rather than an independently validated design choice.

full rationale

The paper's central contribution is an empirical system evaluation, not a derivation from premises that include its own conclusions. PyCapsule's benchmark scores are produced by running the stated two-agent pipeline (Algorithm 1) against public datasets; no load-bearing step is justified solely by an author self-citation, and the paper does not invoke any uniqueness theorem. The reported SOTA comparisons do depend on AgentCoder's published numbers, and the footnote explicitly admits these could not be reproduced ('we advise caution in direct comparisons'); that is an external-validity or reproducibility risk, not circularity. The only mild circular step is the five-attempt cap: the exponential decay fit in Section 4.2 is computed from runs that already enforce a five-attempt cap, and the same fit is then used to justify the cap and to claim that debugging effectiveness approaches zero after five attempts. This is an in-sample design justification rather than an independent prediction, and it modestly shapes the reported results, but it does not make the headline benchmark comparison equivalent to its inputs. Overall, no significant circularity; the score is 2 mainly to flag the self-referential cap determination.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim is an empirical engineering result. The ledger shows the main non-benchmark inputs: fitted decay parameters, a capped attempt limit, unstated sampling hyperparameters, and assumptions about test oracles and baseline validity. No speculative physical or conceptual entities are introduced.

free parameters (3)
  • Exponential decay fit parameters a and b = not reported numerically
    Used in I(x) = a·e^{-bx} to model per-attempt debugging effectiveness; fitted to the same data and then used to justify the five-attempt limit and the claim of exponential decay (Section 4.2, Figure 3).
  • Maximum self-debugging attempts (N=5) = 5
    Set from preliminary experiments and the fitted decay curve; caps the loop that produces the reported results (Section 3.1, Algorithm 1).
  • Model sampling settings (temperature, max tokens, top-p) = not reported
    Mentioned in Appendix B as 'optimized implementation details' but no values are given; these choices affect pass rates and are not controlled across the baselines.
assumptions (4)
  • domain assumption Passing the provided test cases is a correct and sufficient oracle for code correctness.
    Success rate is defined as the fraction of problems whose code passes all dataset tests; there is no human verification of hidden failures. Standard for these benchmarks but not flagged.
  • domain assumption Reported baseline results from the cited papers are accurate and comparable.
    Table 1 uses official reported numbers for MapCoder, AgentCoder, LDB, and Direct baselines; the paper itself acknowledges it could not reproduce AgentCoder (footnote 1).
  • domain assumption Retaining only the most recent problem-solution pair preserves the information needed for debugging (MDP assumption).
    Section 3.1 justifies single-pair context with a Markov Decision Process argument; no ablation shows the retained pair is sufficient for fixing diverse errors.
  • domain assumption The error-handling module's filtered messages retain all debugging-relevant information.
    Section 3.2 and E.3 truncate tracebacks and replace them with generic natural-language feedback; there is no ablation confirming fixes are not hindered by lost detail.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Large Language Model Guided Self-Debugging Code Generation." pith.science (2026). https://pith.science/paper/IKZ45LTA

@misc{pith2026250202928,
  author       = {Pith},
  title        = {Pith review of: Large Language Model Guided Self-Debugging Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IKZ45LTA}},
  note         = {Machine review of arXiv:2502.02928}
}
read the original abstract

Automated code generation is gaining significant importance in intelligent computer programming and system deployment. However, current approaches often face challenges in computational efficiency and lack robust mechanisms for code parsing and error correction. In this work, we propose a novel framework, PyCapsule, with a simple yet effective two-agent pipeline and efficient self-debugging modules for Python code generation. PyCapsule features sophisticated prompt inference, iterative error handling, and case testing, ensuring high generation stability, safety, and correctness. Empirically, PyCapsule achieves up to 5.7% improvement of success rate on HumanEval, 10.3% on HumanEval-ET, and 24.4% on BigCodeBench compared to the state-of-art methods. We also observe a decrease in normalized success rate given more self-debugging attempts, potentially affected by limited and noisy error feedback in retention. PyCapsule demonstrates broader impacts on advancing lightweight and efficient code generation for artificial intelligence systems.

Figures

Figures reproduced from arXiv: 2502.02928 by the authors.

Figure 1
Figure 1. PyCapsule framework for Python code generation. The framework comprises iterative code generation, [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 1
Figure 1. This pipeline integrates the programmer and executor agents with the self-debugging [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Distribution of relative success ratios along the self-debugging attempts. The relative success ratio [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figures from the paper (3 more)
Figure 3
Figure 3. Figure 3: The plot illustrates the exponential decay in the normalized independent influence of self-debugging [PITH_FULL_IMAGE:figures/full_fig_p008_3.png]
Figure 4
Figure 4. Figure 4: Visualisation of Debugging Attempts. The left column represents the number of debugging attempts on [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: HumanEval experiment results using GPT-4-1106 across three repeats. Each line represents the [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. From Evaluation to Optimisation: Hierarchy-Aware Training Signals for CWE Prediction in Python

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Using the ALPHA hierarchy penalty as a direct GRPO reward improves CWE prediction under distribution shift, while supervised delivery of the same penalty regresses below zero-shot.

  2. MineValiCoder: Reliable Code Generation with Test Case Quality Mining and Bipartite Graph-Based Mutual Validation

    cs.SE 2026-07 conditional novelty 5.0 of 10

    A closed-loop test-driven framework - filter tests by LLM self-consistency, refine code in parallel, and select the winner by bipartite code-test mutual scoring - reaches 96.34% Pass@1 on HumanEval without human-writt...

Reference graph

Works this paper leans on

40 extracted references · 9 canonical work pages · cited by 2 Pith papers

  1. [1]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021)

    arXiv 2021

  2. [2]

    Mark Burgin and Gordana Dodig Crnkovic. 2009. A Systematic Approach to Artificial Agents. (03 2009)

    work page 2009

  3. [3]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, et al. 2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374 (2021)

    arXiv 2021

  4. [4]

    Xi Chen, XiangWen Deng, Hao Wen, MingKe You, Weizhi Liu, Qi Li, and Jian Li. 2024. Prompt engineering in consistency and reliability with the evidence-based guideline for LLMs. npj Digital Medicine 7 (02 2024). https: //doi.org/10.1038/s41746-024-01029-4

    doi:10.1038/s41746-024-01029-4 2024

  5. [5]

    Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching Large Language Models to Self-Debug. arXiv:2304.05128 [cs.CL] https://arxiv.org/abs/2304.05128

    arXiv 2023

  6. [6]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sashank Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bra...

    work page 2024

  7. [7]

    Yihong Dong, Jiazheng Ding, Xue Jiang, Zhuo Li, Ge Li, and Zhi Jin. 2023. Codescore: Evaluating code generation by learning code execution. arXiv preprint arXiv:2301.09043 (2023)

    arXiv 2023

  8. [8]

    Shihan Dou, Haoxiang Jia, Shenxi Wu, Huiyuan Zheng, Weikang Zhou, Muling Wu, Mingxu Chai, Jessica Fan, Caishuang Huang, Yunbo Tao, Yan Liu, Enyu Zhou, Ming Zhang, Yuhao Zhou, Yueming Wu, Rui Zheng, Ming Wen, Rongxiang Weng, Jingang Wang, Xunliang Cai, Tao Gui, Xipeng Qiu, Qi Zhang, and Xuanjing Huang. 2024. What’s Wrong with Your Code Generated by Large L...

    arXiv 2024

Show all 40 references
  1. [9]

    Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M. Zhang. 2023. Large Language Models for Software Engineering: Survey and Open Problems. In 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineer...

    2023

  2. [10]

    Michael Fu, Chakkrit Tantithamthavorn, Trung Le, Van Nguyen, and Dinh Phung. 2022. VulRepair: a T5-based automated software vulnerability repair. InProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineeri...

    2022

  3. [11]

    Sirui Hong, Mingchen Zhuge, Jiaqi Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber

  4. [12]

    Dong Huang, Qingwen Bu, Yuhao Qing, and Heming Cui. 2024. CodeCoT: Tackling Code Syntax Errors in CoT Reasoning for Code Generation. arXiv:2308.08784 [cs.SE] https://arxiv.org/abs/2308.08784

    2024 arXiv

  5. [13]

    Zhang, Michael Luck, Qingwen Bu, Yuhao Qing, and Heming Cui

    Dong Huang, Jie M. Zhang, Michael Luck, Qingwen Bu, Yuhao Qing, and Heming Cui. 2024. AgentCoder: Multi-Agent- based Code Generation with Iterative Testing and Optimisation. arXiv:2312.13010 [cs.CL] https://arxiv.org/abs/2312. 13010

    2024 arXiv

  6. [14]

    Kai Huang, Xiangxin Meng, Jian Zhang, Yang Liu, Wenjie Wang, Shuhao Li, and Yuqing Zhang. 2023. An Empirical Study on Fine-Tuning Large Language Models of Code for Automated Program Repair.2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) (202...

    2023

  7. [15]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou...

    2024 arXiv

  8. [16]

    Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez

    Md. Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez. 2024. MapCoder: Multi-Agent Code Generation for Competitive Problem Solving. arXiv:2405.11403 [cs.CL] https://arxiv.org/abs/2405.11403 J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018. 111:10 M. Ad...

    2024 arXiv

  9. [17]

    Nicholas Jennings, Katia Sycara, and Michael Wooldridge. 1998. A Roadmap of Agent Research and Development. Autonomous Agents and Multi-Agent Systems 1 (03 1998), 7–38. https://doi.org/10.1023/A:1010090405266

    1998 doi

  10. [18]

    Harshit Joshi, José Cambronero Sanchez, Sumit Gulwani, Vu Le, Ivan Radiček, and Gust Verbruggen. 2023. Repair is nearly generation: multilingual program repair with LLMs. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conferenc...

    2023 doi

  11. [19]

    Nils Knoth, Antonia Tolzin, Andreas Janson, and Jan Marco Leimeister. 2024. AI literacy and its implications for prompt engineering strategies. Computers and Education: Artificial Intelligence (2024)

    2024

  12. [20]

    Kefan Li and Yuan Yuan. 2024. Large Language Models as Test Case Generators: Performance Evaluation and Enhancement. arXiv:2404.13340 [cs.SE] https://arxiv.org/abs/2404.13340

    2024 arXiv

  13. [21]

    Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. 2025. Long-context LLMs Struggle with Long In-context Learning. Transactions on Machine Learning Research (2025). https://openreview.net/forum?id=Cw2xlg0e46

    2025

  14. [22]

    Eric Wong

    Yihao Li, Pan Liu, Haiyang Wang, Jie Chu, and W. Eric Wong. 2025. Evaluating large language models for software testing. Computer Standards & Interfaces 93 (2025), 103942. https://doi.org/10.1016/j.csi.2024.103942

    2025

  15. [23]

    Xiaoli Lian, Shuaisong Wang, Jieping Ma, Xin Tan, Fang Liu, Lin Shi, Cuiyun Gao, and Li Zhang. 2024. Imperfect Code Generation: Uncovering Weaknesses in Automatic Code Generation by Large Language Models (ICSE-Companion ’24). Association for Computing Machinery, New York, NY, ...

    2024

  16. [24]

    Mingxing Liu, Junfeng Wang, Tao Lin, Quan Ma, Zhiyang Fang, and Yanqun Wu. 2024. An Empirical Study of the Code Generation of Safety-Critical Software Using LLMs. Applied Sciences 14, 3 (2024). https://doi.org/10.3390/app14031046

    2024 doi

  17. [25]

    Hyacinth S. Nwana. 1996. Software agents: an overview. The Knowledge Engineering Review 11, 3 (1996), 205–244. https://doi.org/10.1017/S026988890000789X

    1996 doi

  18. [26]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 arXiv

  19. [27]

    Bernstein

    Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative Agents: Interactive Simulacra of Human Behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (San Franc...

    2023

  20. [28]

    Dhavalkumar Patel, Ganesh Raut, Eyal Zimlichman, Satya Cheetirala, Girish Nadkarni, Benjamin Glicksberg, Donald Apakama, Elijah Bell, Robert Freeman, Prem Timsina, and Eyal Klang. 2024. Evaluating prompt engineering on GPT-3.5’s performance in USMLE-style medical calculations ...

    2024 doi

  21. [29]

    Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. ChatDev: Communicative Agents for Software Development. arXiv:2307.07924 [cs.SE] https://arxiv.org/abs/2307.07924

    2024 arXiv

  22. [30]

    Sanka Rasnayaka, Guanlin Wang, Ridwan Shariffdeen, and Ganesh Neelakanta Iyer. 2024. An Empirical Study on Usage and Perceptions of LLMs in a Software Engineering Project. In Proceedings of the 1st International Workshop on Large Language Models for Code (Lisbon, Portugal) (LL...

    2024

  23. [31]

    Murray Shanahan, Kyle McDonell, and Laria Reynolds. 2023. Role play with large language models. Nature 623 (11 2023). https://doi.org/10.1038/s41586-023-06647-8

    2023 doi

  24. [32]

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language Agents with Verbal Reinforcement Learning. arXiv:2303.11366 [cs.AI] https://arxiv.org/abs/2303. 11366

    2023 arXiv

  25. [33]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto. 2018. Reinforcement Learning: An Introduction (second ed.). The MIT Press

    2018

  26. [34]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, F. Xia, Quoc Le, and Denny Zhou. 2022. Chain of Thought Prompting Elicits Reasoning in Large Language Models. ArXiv abs/2201.11903 (2022). https: //api.semanticscholar.org/CorpusID:246411621

    2022 arXiv

  27. [35]

    Michael Wooldridge and Nicholas R Jennings. 1995. Intelligent agents: Theory and practice. The knowledge engineering review 10, 2 (1995), 115–152

    1995

  28. [36]

    Chun Xia and Lingming Zhang. 2022. Less training, more repairing please: revisiting automated program repair via zero-shot learning. Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (2022). http...

    2022

  29. [37]

    Chunqiu Steven Xia and Lingming Zhang. 2022. Less training, more repairing please: revisiting automated program repair via zero-shot learning. In Proceedings of the 30th ACM Joint European Software Engineering Conference and J. ACM, Vol. 37, No. 4, Article 111. Publication dat...

    2022

  30. [38]

    Li Zhong, Zilong Wang, and Jingbo Shang. 2024. Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-step. arXiv:2402.16906 [cs.SE] https://arxiv.org/abs/2402.16906

    2024 arXiv

  31. [39]

    Generation Mode

    Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. 2024. BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions. arXiv preprint arXiv:240...

    2024 arXiv

  32. [2024]

    arXiv:2308.00352 [cs.AI] https: //arxiv.org/abs/2308.00352

    MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework. arXiv:2308.00352 [cs.AI] https: //arxiv.org/abs/2308.00352

    arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.