Pith. sign in

REVIEW 4 major objections 6 minor 57 references

CSR-Bench: Benchmarking LLM Agents in Deployment of Computer Science Research Repositories

T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CSR-Bench measures and improves how well LLM agents can deploy research code repositories, finding that an escalation pipeline raises completion rates for every model family tested.

desk verdict A genuinely useful new benchmark for repository deployment, but its headline numbers rest on an unvalidated LLM judge, so treat the quantitative claims as provisional until the harness ships. read the letter →

arxiv 2502.06111 v2 pith:64BTHF2S submitted 2025-02-10 cs.SE cs.AIcs.LG

Yijia Xiao , Runhui Wang , Luyang Kong , Davor Golac , Wei Wang This is my paper · ORCID
classification cs.SEcs.AIcs.LG
keywords CSR-BenchLLMagentsrepositorydeploymentmulti-agentframeworkbashcommandgenerationcodeautomationretrieval-augmentedrepairbenchmarking
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that large language models can be made usefully reliable at deploying computer science research repositories by turning a README into working bash commands for environment setup, data download, training, inference, and evaluation. It introduces CSR-Bench, a collection of 100 research repositories, and CSR-Agents, a pipeline in which a command drafter, log analyzer, issue retriever, and web searcher cooperate to repair failed commands. With the full pipeline, cumulative success rates rise for every model family tested, reaching about 0.46 for setup and download and 0.15 to 0.29 for training, inference, and evaluation. The paper's claim, read sympathetically, is that iterative error repair with retrieval and web search is the key mechanism, and that full autonomous deployment remains out of reach.

What carries the argument

The carrying mechanism is the CSR-Agents escalation chain. A Command Drafter converts README instructions and the repository's file tree into bash scripts split into five deployment sections. A Script Executor runs those scripts in a standardized containerized environment and, because many bash commands lack a reliable return code, uses an LLM to judge success from standard output and error streams. Failed commands pass to a Log Analyzer for internal repair, then to an Issue Retriever that retrieves relevant repository-issue discussions via BM25 keyword matching on the command, output, and error text, and finally to a Web Searcher that queries the web for external fixes. Each stage feeds refined commands back into the executor, so the architecture converts error logs into increasingly informed repair attempts.

What would settle it

Run the pipeline on a subset of repositories while recording true bash exit codes for every executed command, then tally how often the LLM executor's success judgment disagrees with the real exit code; if disagreement is common, the reported completion rates and escalation gains need revision.

Watch

Extended reading notes

Core claim

CSR-Bench is a benchmark of 100 research repositories with pinned versions, grouped into five deployment stages: environment setup, data and model download, training, inference, and evaluation. The central discovery is that CSR-Agents, an escalation pipeline of four cooperating agents, materially raises the share of commands that execute successfully compared with a single drafting model. The pattern holds across Claude, GPT, Llama, and Mistral families: drafter-only success is roughly 0.23 to 0.31 for setup and download and near zero for the harder stages; adding log analysis raises setup and download to about 0.34 to 0.40; adding issue retrieval pushes complex tasks toward 0.25; and adding web search brings setup and download to about 0.46 and the complex stages to 0.15 to 0.29. The paper reads these results as evidence that LLM agents can meaningfully accelerate repository deployment while still falling far short of a fully autonomous research workflow.

Load-bearing premise

The entire benchmark's numbers assume the Script Executor's LLM is correctly reading standard output and error streams when it decides a command succeeded, and that judgment is never checked against real exit codes.

Editorial extensions

If this is right

  • Every model family in the paper improves monotonically as more agents are added, so the escalation architecture itself, rather than model choice, drives the reported gains.
  • Setup and download commands succeed close to half the time with the full pipeline, while training, inference, and evaluation stay under 30 percent, making near-term automation practical for dependency and environment work but not for full research runs.
  • The issue retriever's contribution implies that repository-specific discussion history is a usable repair signal, so deployments of mature repositories with active issue databases should be easier than deployments of fresh ones.
  • Full autonomy is not achieved: the best cumulative rates leave a large fraction of deployment steps unfinished, so the paper's contribution is a measured baseline and an architecture, not a solved task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the escalation order is the active ingredient, then applying the same pipeline to repositories outside computer science that have an issue database should reproduce the relative gains; the paper leaves that transfer test open.
  • Because the Script Executor's success judgment is never validated against true bash exit codes, the absolute completion rates are best treated as upper bounds; the monotone improvement from adding agents is the sturdier claim.
  • The paper announces efficiency as a benchmark goal but reports no wall-clock or token costs, so a natural extension is measuring time and cost per successful deployment rather than success rate alone.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces CSR-Bench, a benchmark of 100 GitHub repositories from computer science research papers, and CSR-Agents, a multi-agent framework (Command Drafter, Script Executor, Log Analyzer, Issue Retriever, Web Searcher) for automating repository deployment. The authors evaluate several LLM families (Claude, GPT, Llama, Mistral) across five deployment stages (setup, download, training, inference, evaluation) and report that completion rates increase monotonically as more agents are added, reaching approximately 0.46 for setup/download and 0.15--0.29 for the other stages. The central claim is that LLM agents can significantly enhance the repository deployment workflow, while full autonomy remains out of reach.

Significance. If the quantitative results are trustworthy, the paper makes a useful contribution: it defines a new task area (deployment of research repositories, distinct from code generation or issue resolution), provides a curated repository collection with pinned commit IDs and permissive licenses, and proposes a sensible escalation architecture. The diversity of topics and the use of Docker for environment isolation are also strengths. The paper ships no fitted parameters or mathematical derivations, so the standard circularity concern does not apply. However, the empirical claim rests entirely on an unvalidated completion-rate metric, so the significance of the current results is conditional on metric validation; the benchmark itself could become valuable after that validation is supplied.

major comments (4)
  1. [Section 4.2 / Section 3.1] The completion-rate metric depends on an LLM judge that parses stdout/stderr into a success/failure signal, and this judge is never validated against true bash exit codes, human labels, or downstream artifact checks. Section 4.2 states that 'a good quantity of commands do not have return code,' so the executor instructs an LLM to provide feedback based on standard output and error messages, and this feedback is parsed into a return code. Every entry in Tables 1--4 and every curve in Figures 8--11 inherits this signal. Because later-stage commands are generated after observing an error and often take the form of explicit fixes, an LLM judge may systematically label them as successful more often than initial drafts, even when the underlying deployment has not actually progressed. The reported monotonic improvement from adding Log Analyzer, Issue Retriever, and Web Searcher could therefore be an artifact of the measurement instrument. The authors should validate the LLM judge on a sample against true exit codes or human annotations, report agreement rates, and perform a sensitivity analysis with stricter success criteria (e.g., checking that expected artifacts such as checkpoints or output files are produced).
  2. [Section 3.1] The completion-rate definition is ambiguous and potentially misleading. It is defined as 'the ratio between number of successfully executed commands and the total number of commands executed,' which is a command-level rate, not a repository-level deployment success rate. The paper does not specify how this ratio is aggregated over the 100 repositories (pooling all commands versus averaging per-repository rates), and it does not report the number of commands per repository or per stage. Because later pipeline stages may execute additional commands through retries, the denominator changes across the conditions being compared (Drafter, Analyzer, Issue Retriever, Web Searcher), so the monotonic improvements in Tables 1--4 could partly reflect changes in the denominator rather than improvements in deployment success. The authors should report repository-level success rates (e.g., the fraction of repositories where all five deployment stages complete successfully) and clarify the aggregation procedure, ideally with per-repository paired comparisons.
  3. [Section 5] No error bars, confidence intervals, or significance tests are reported for any of the completion rates. With 100 repositories and unknown command counts, the differences in Tables 1--4 (e.g., Setup improving from 0.28 to 0.47 for Claude 3 Sonnet) may be within sampling variability. The paper should provide bootstrap confidence intervals across repositories, or at least report the number of commands per stage and per model. In addition, the evaluation setup is under-specified: the Docker image, hardware (GPU/CPU), timeouts, retry budgets (max_attempt values), and LLM decoding settings are not stated. These details are required to assess whether later-stage gains come from additional compute/time rather than from the agent architecture, and to make the benchmark reproducible.
  4. [Section 4.2 / Section 5.1] The 'success' of a command is not grounded in actual deployment outcomes. The paper reports that the LLM judge decides success from stdout/stderr, but it does not check whether a training run produced model checkpoints, whether an inference command produced output artifacts, or whether an evaluation script generated the expected metrics. For a deployment benchmark, the ground truth should be whether the repository can actually be used for its intended purpose, not merely whether an LLM perceives no error in the log. The authors should augment the evaluation with artifact-based checks or a human-validated subset of repositories to establish that the completion rate measures deployment success rather than command-level plausibility.
minor comments (6)
  1. [Section 4.2] The heading 'CSR-Agent: LLM Agent Design' should be 'CSR-Agents' for consistency, and 'Retriver' in the Issue Retriever description is a typo for 'Retriever'.
  2. [Section 4.3] The heading 'LLM Coorporation Framwfork' contains typos and should read 'LLM Cooperation Framework'.
  3. [Section 5.1] The sentence introducing the table abbreviations says 'D to stand for the owload stage'; this should be 'Download stage'.
  4. [Figures 8-11] The legend labels include 'With Analyzer and Ragger,' but the agent is named 'Issue Retriever'; the legend should use the correct terminology for clarity.
  5. [Appendix A.2] Figure 12 is captioned 'Distribution of Topics in GSRBench100'; this should be 'CSR-Bench' for consistency with the benchmark name.
  6. [Appendix C] The README example in Table 8 contains 'FQA' in the original text; this appears to be a typo for 'FAQ'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: CSR-Bench is an empirical benchmark and the LLM-judged success signal is a measurement-validity concern, not a derivation that reduces to its own inputs.

full rationale

The paper contains no mathematical derivation chain, no fitted parameters, and no quantity that is defined in terms of the thing it is used to predict. The central claim is empirical: adding the Log Analyzer, Issue Retriever, and Web Searcher agents increases the defined completion rate across LLM families and deployment stages (Tables 1-4, Figures 8-11). The completion rate is explicitly defined in Section 3.1 as the ratio of successfully executed commands to total commands executed, and Section 4.2 explains that because bash often provides no return code, the executor uses an LLM to parse standard output and standard error into a return code that is then treated as success or failure. This means the metric inherits an unvalidated LLM judgment, which is a legitimate threat to construct validity: the reported improvements could partly reflect the judge's leniency or bias rather than genuine deployment progress. However, that is a correctness and measurement concern, not circularity under the enumerated patterns. The paper does not fit a parameter to a subset and then 'predict' a closely related quantity; it does not invoke a self-citation chain or a uniqueness theorem from the authors' prior work; and it does not rename an existing result. References are all external prior work rather than load-bearing self-citations. The skeptical worry about LLM-judged labels should be addressed with validation against true exit codes or artifact checks, but it does not make the derivation equivalent to its inputs by construction. Accordingly, the circularity score is 0.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

No mathematically fitted constants or invented entities appear in this paper. The central claim depends on unreported evaluation choices (LLM-based success judging, retry budgets, Docker environment) and on benchmark selection assumptions, both of which are listed above.

free parameters (2)
  • Agent retry budget (max_attempt) = not reported
    Controls how many times the Log Analyzer and Web Searcher can rewrite a command before giving up; it directly affects all success rates and is never given a value in the paper.
  • LLM decoding settings (temperature, sampling) = not reported
    No decoding parameters are reported for any of the four model families; the same model can produce different deployment commands across settings, so the success rates are not pinned to reproducible configurations.
assumptions (5)
  • ad hoc to paper LLM-parsed return codes are a valid proxy for command success
    Section 4.2 replaces bash return codes with an LLM's reading of stdout and stderr and never validates this against true exit codes or human labels; all metrics depend on it.
  • domain assumption Top-star, self-contained GitHub repositories are representative of CS research deployment
    Section 3.2 selects 100 repos by stars, manual README inspection, deployable code, and permissive licenses; this biases the benchmark away from poorly documented or non-standard repos.
  • domain assumption A fixed Docker image with bash, Conda, GCC, Make, and Python suffices for all 100 repositories
    Section 4.1 says a standard Docker image is used for all repos, but no GPU, memory, or timeout configuration is reported; training and inference success rates may be capped by environment limits rather than agent skill.
  • domain assumption GitHub issues and web search contain the information needed to repair most deployment errors
    The Issue Retriever and Web Searcher stages assume the repository's issue database and general web results offer fixes for execution errors; the paper does not measure how often errors actually have retrievable solutions.
  • domain assumption BM25 lexical retrieval is sufficient for finding relevant GitHub issues
    Section 4.2 chooses BM25 over Contriever based on speed and keyword overlap, but no retrieval accuracy evaluation is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CSR-Bench: Benchmarking LLM Agents in Deployment of Computer Science Research Repositories." pith.science (2026). https://pith.science/paper/64BTHF2S

@misc{pith2026250206111,
  author       = {Pith},
  title        = {Pith review of: CSR-Bench: Benchmarking LLM Agents in Deployment of Computer Science Research Repositories},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/64BTHF2S}},
  note         = {Machine review of arXiv:2502.06111}
}
read the original abstract

The increasing complexity of computer science research projects demands more effective tools for deploying code repositories. Large Language Models (LLMs), such as Anthropic Claude and Meta Llama, have demonstrated significant advancements across various fields of computer science research, including the automation of diverse software engineering tasks. To evaluate the effectiveness of LLMs in handling complex code development tasks of research projects, particularly for NLP/CV/AI/ML/DM topics, we introduce CSR-Bench, a benchmark for Computer Science Research projects. This benchmark assesses LLMs from various aspects including accuracy, efficiency, and deployment script quality, aiming to explore their potential in conducting computer science research autonomously. We also introduce a novel framework, CSR-Agents, that utilizes multiple LLM agents to automate the deployment of GitHub code repositories of computer science research projects. Specifically, by checking instructions from markdown files and interpreting repository structures, the model generates and iteratively improves bash commands that set up the experimental environments and deploy the code to conduct research tasks. Preliminary results from CSR-Bench indicate that LLM agents can significantly enhance the workflow of repository deployment, thereby boosting developer productivity and improving the management of developmental workflows.

Figures

Figures reproduced from arXiv: 2502.06111 by the authors.

Figure 1
Figure 1. Conf Distribution of CSR-Bench ML 25% CV 19% LLM 19% NLP 19% Interdisciplinary 19% Gen. & Diff. Models 6% GNNs & Transf. 6% Self-Sup. & Contr. Learn. 6% M odel I n te rp. & Opt. 6% 3 D Vi sio n & Rec. 6% Img Segm. & Enh. 6% Pose Estim. & Obj Det. 6% LLM Dev. & Opt. 6% Infer. & Eff. in LLMs 6% LLM Appl. & Eval. 6% Txt Classif. & Extr. 6% M ulti. & Kn owl. Sys. 6% Tra n sf. Lea rn. & Do m. Ad a p. 6% Health. & Bioinf.… view at source ↗
Figure 4
Figure 4. Number of Files per Repository [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 3
Figure 3. Number of Tokens per README [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 6
Figure 6. Figure 6: Size of Issue Database [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: Workflow of CSR-Bench is provided by bash. We experimented with set￾ting predefined special prompt to bash and parse the return code from the returned output and error message. However, we find that a good quantity of commands do not have return code, making the feedba…
Figure 8
Figure 8. Figure 8: Performance of Claude 3 Sonnet Setup Download Training Inference Evaluation Deployment Stage 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Cumulative Success Rate Execution Type Initial Execution With Analyzer Only With Analyzer and Ragger With Analyzer, Ragger, and Searcher [PITH_FULL…
Figure 10
Figure 10. Figure 10: Performance of Llama 3.1 70B Instruct Setup Download Training Inference Evaluation Deployment Stage 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Cumulative Success Rate Execution Type Initial Execution With Analyzer Only With Analyzer and Ragger With Analyzer, Ragger, and Searcher [P…
Figure 12
Figure 12. Figure 12: Distribution of Topics in GSRBench100 [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 23 canonical work pages

  1. [1]

    Agent-101: A software engineering agent for code assistance developed by ibm research

    2024. Agent-101: A software engineering agent for code assistance developed by ibm research. https://github.com/swe-bench/experiments/blob/main/evaluation/lite/20240612_IBM_Research_Agent101/README.md/

    arXiv 2024

  2. [2]

    Amazon q developer the most capable generative ai–powered assistant for software development

    2024. Amazon q developer the most capable generative ai–powered assistant for software development. https://aws.amazon.com/q/developer/

    work page 2024

  3. [3]

    Devin, ai software engineer

    2024. Devin, ai software engineer. https://www.cognition.ai/introducing-devin

    work page 2024

  4. [4]

    Factory bringing autonomy to software engineering

    2024. Factory bringing autonomy to software engineering. https://www.factory.ai/

    work page 2024

  5. [5]

    Lingma agent

    2024. Lingma agent. https://github.com/swe-bench/experiments/tree/main/evaluation/lite/20240622_Lingma_Agent

    arXiv 2024

  6. [6]

    Marscode code and innovate faster with ai

    2024. Marscode code and innovate faster with ai. https://www.marscode.com/

    work page 2024

  7. [7]

    Opencsg starship

    2024 a . Opencsg starship. https://opencsg.com/product?class=StarShip/

    work page 2024

  8. [8]

    Opendevin: Code less, make more

    2024 b . Opendevin: Code less, make more. https://github.com/OpenDevin/OpenDevin/

    work page 2024

Show all 57 references
  1. [9]

    Swe-bench lite

    2024. Swe-bench lite. https://www.swebench.com/lite.html

    2024

  2. [10]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    2023 arXiv

  3. [11]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. https://arxiv.org/abs/2108.07732 Program synthesis with large language models . Preprint, arXiv:2108.07732

    2021 arXiv

  4. [12]

    Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. 2024. Repairagent: An autonomous, llm-based agent for program repair. arXiv preprint arXiv:2403.17134

    2024 arXiv

  5. [13]

    Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. 2023. Multipl-e: A scalable and polyglot approach to benchmarking neural code generation. IEEE Transaction...

    2023

  6. [14]

    Dong Chen, Shaoxin Lin, Muhan Zeng, Daoguang Zan, Jian-Gang Wang, Anton Cheshkov, Jun Sun, Hao Yu, Guoliang Dong, Artem Aliev, et al. 2024. Coder: Issue resolving with multi-agent and task graphs. arXiv preprint arXiv:2406.01304

    2024 arXiv

  7. [15]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    2021 arXiv

  8. [16]

    Yinlin Deng, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang. 2023. Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models. In 32nd International Symposium on Software Testing and Analysis (ISSTA)

    2023

  9. [17]

    Yinlin Deng, Chunqiu Steven Xia, Chenyuan Yang, Shizhuo Dylan Zhang, Shujing Yang, and Lingming Zhang. 2024. Large language models are edge-case fuzzers: Testing deep learning libraries via fuzzgpt. In 46th International Conference on Software Engineering (ICSE)

    2024

  10. [18]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  11. [19]

    Paul Gauthier. 2024. Aider is ai pair programming in your terminal. https://aider.chat/

    2024

  12. [20]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. 2024. Deepseek-coder: When the large language model meets programming--the rise of code intelligence. arXiv preprint arXiv:2401.14196

    2024 arXiv

  13. [21]

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring coding challenge competence with apps. NeurIPS

    2021

  14. [22]

    Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. 2023. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.00352

    2023 arXiv

  15. [23]

    Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2018. Mapping language to code in programmatic context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1643--1652

    2018

  16. [24]

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint

    2024

  17. [25]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. https://openreview.net/forum?id=VTF8yNQM66 SWE -bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Re...

    2024

  18. [26]

    Sungmin Kang, Juyeon Yoon, and Shin Yoo. 2023. Large language models are few-shot testers: Exploring llm-based general bug reproduction. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 2312--2323. IEEE

    2023

  19. [27]

    Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2023. Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, pages 18319--1...

    2023

  20. [28]

    Caroline Lemieux, Jeevana Priya Inala, Shuvendu K Lahiri, and Siddhartha Sen. 2023. Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models. In 45th International Conference on Software Engineering (ICSE)

    2023

  21. [29]

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. Starcoder: may the source be with you!

    2023

  22. [30]

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, R \'e mi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022 a . Competition-level code generation with alphacode. Science, 378(6624):1092--1097

    2022

  23. [31]

    Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, R \'e mi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal,...

    2022 doi

  24. [32]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023 a . https://openreview.net/forum?id=1qvx610Cu7 Is your code generated by chat GPT really correct? rigorous evaluation of large language models for code generation . In Thirty-seventh Conference on Neural Info...

    2023

  25. [33]

    Tianyang Liu, Canwen Xu, and Julian McAuley. 2023 b . Repobench: Benchmarking repository-level code auto-completion systems. arXiv preprint arXiv:2306.03091

    2023 arXiv

  26. [34]

    Yuliang Liu, Xiangru Tang, Zefan Cai, Junjie Lu, Yichi Zhang, Yanjun Shao, Zexuan Deng, Helan Hu, Zengxian Yang, Kaikai An, et al. 2023 c . Ml-bench: Large language models leverage open-source libraries for machine learning tasks. arXiv e-prints, pages arXiv--2311

    2023

  27. [35]

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Z...

    2024 arXiv

  28. [36]

    Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568

    2023 arXiv

  29. [37]

    Yingwei Ma, Qingping Yang, Rongyu Cao, Binhua Li, Fei Huang, and Yongbin Li. 2024. How to understand whole software repository? arXiv preprint arXiv:2406.01422

    2024 arXiv

  30. [38]

    Martin Monperrus. 2018. The living review on automated program repair. Ph.D. thesis, HAL Archives Ouvertes

    2018

  31. [39]

    Rangeet Pan, Ali Reza Ibrahimzada, Rahul Krishna, Divya Sankar, Lambert Pouguem Wassi, Michele Merler, Boris Sobolev, Raju Pavuluri, Saurabh Sinha, and Reyhaneh Jabbarvand. 2024. Lost in translation: A study of bugs introduced by large language models while translating code. I...

    2024

  32. [40]

    Noah Patton, Kia Rahmani, Meghana Missula, Joydeep Biswas, and I s l Dillig. 2024. Programming-by-demonstration for long-horizon robot tasks. Proceedings of the ACM on Programming Languages, 8(POPL):512--545

    2024

  33. [41]

    Baptiste Rozi \`e re, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, J \'e r \'e my Rapin, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950

    2023 arXiv

  34. [42]

    Baptiste Roziere, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. 2020. Unsupervised translation of programming languages. Advances in neural information processing systems, 33:20601--20611

    2020

  35. [43]

    Baptiste Roziere, Jie M Zhang, Francois Charton, Mark Harman, Gabriel Synnaeve, and Guillaume Lample. 2021. Leveraging automated unit tests for unsupervised code translation. arXiv preprint arXiv:2110.06773

    2021 arXiv

  36. [44]

    Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2023. Magicoder: Source code is all you need. arXiv preprint arXiv:2312.02120

    2023 arXiv

  37. [45]

    Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. 2023. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864

    2023 arXiv

  38. [46]

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2024 a . Agentless: Demystifying llm-based software engineering agents. arXiv preprint arXiv:2407.01489

    2024 arXiv

  39. [47]

    Chunqiu Steven Xia, Matteo Paltenghi, Jia Le Tian, Michael Pradel, and Lingming Zhang. 2024 b . Universal fuzzing via large language models. In 46th International Conference on Software Engineering (ICSE)

    2024

  40. [48]

    Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2023. Automated program repair in the era of large pre-trained language models. In Proceedings of the ACM/IEEE 45th International Conference on Software Engineering, ICSE '23

    2023

  41. [49]

    Chunqiu Steven Xia and Lingming Zhang. 2023. Keep the conversation going: Fixing 162 out of 337 bugs for \ 0.42 each using chatgpt. arXiv preprint arXiv:2304.00385

    2023 arXiv

  42. [50]

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. arXiv preprint arXiv:2405.15793

    2024 arXiv

  43. [51]

    Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, Alex Polozov, and Charles Sutton. 2022. https://arxiv.org/abs/arXiv:2212.09248 Natural language to code generation in interactive da...

    2022 arXiv

  44. [52]

    Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. Repocoder: Repository-level code completion through iterative retrieval and generation. arXiv preprint arXiv:2303.12570

    2023 arXiv

  45. [53]

    Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. https://arxiv.org/abs/2404.05427 Autocoderover: Autonomous program improvement . Preprint, arXiv:2404.05427

    2024 arXiv

  46. [54]

    Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, et al. 2023. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. arXiv preprint arXiv:2303.17568

    2023 arXiv

  47. [55]

    Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. 2024. Opencodeinterpreter: Integrating code generation with execution and refinement. arXiv preprint arXiv:2402.14658

    2024 arXiv

  48. [56]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  49. [57]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.