Pith. sign in

REVIEW 4 major objections 5 minor 45 references

RepoDebug: Repository-Level Multi-Task and Multi-Language Debugging Evaluation of Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RepoDebug, the first multi-task, multi-language repository-level debugging dataset, shows that current LLMs—including the best tested, Claude 3.5 Sonnet—fall short on locating and repairing bugs in real repositories, especially when several

desk verdict RepoDebug is a genuinely new benchmark combination (repo-level, 3 tasks, 8 languages, 22 bug types) but its construct validity is undermined by a syntax-heavy distribution and single-file prompts, so the headline claim about repository-level debugging is not yet supported. read the letter →

arxiv 2509.04078 v2 pith:7MEWTP7U submitted 2025-09-04 cs.SE cs.AI

Jingjing Liu , Zeming Liu , Zihao Cheng , Mengliang He , Xiaoming Shi , Yuhang Guo , Xiangrong Zhu , Yuanfang Guo
show 2 more authors
Yunhong Wang Haifeng Wang
This is my paper
classification cs.SEcs.AI
keywords repository-leveldebuggingLLMevaluationbuglocalizationautomaticprogramrepairabstractsyntaxtreeinjectionmultilingualcodebenchmarkerrortaxonomy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that existing code-debugging benchmarks test LLMs mostly on isolated functions, not on real repository files, and that this hides a major capability gap. To close it, it constructs RepoDebug: bugs injected by modifying abstract syntax trees in 63 real GitHub repositories across 8 languages, covering 22 error subtypes and three tasks—identifying the error type, locating the bug line, and proposing a repair. Evaluations of 10 LLMs show that the best model, Claude 3.5 Sonnet, identifies error types correctly in at most about half of cases and repairs only a small fraction of instances; performance drops sharply with multiple errors and with files longer than 500 tokens. The claim is that repository-level debugging is a distinct and largely unsolved capability for current LLMs, and that RepoDebug is a way to measure progress on it.

What carries the argument

The carrying mechanism is AST-guided bug injection with Tree-sitter: correct code is parsed into an abstract syntax tree, and for each of 22 error subtypes the construction queries relevant node types (calls, assignments, conditionals, brackets), selects nodes randomly, and rewrites source text by string manipulation—replacing identifiers, deleting delimiters, flipping operators, or inserting divide-by-zero. This yields precisely located, controllable faults while preserving surrounding structure; manual checks confirm each edit matches its subtype and harms execution. The evaluation side is carried by four metrics—Bug Identification accuracy, One-Bug's Location, All-Bugs' Location, and repa

What would settle it

Run the repository test suites against every buggy instance in RepoDebug, record the fraction that fail only at compile/link time versus at runtime, and compare model accuracy on those two subsets. If compile-time failures dominate the accuracy signal and models do no better on the rare runtime-failing instances, then the benchmark is measuring syntax-error detection rather than repository-level semantic debugging.

Watch

Extended reading notes

Core claim

RepoDebug is built by taking 63 MIT-licensed GitHub repositories created after January 2022 across eight languages, parsing the correct code with Tree-sitter into abstract syntax trees, and applying rule-based edits that yield 22 bug subtypes grouped as syntax, reference, logic, and multiple errors. Each instance pairs buggy and correct code, the error subtype, and exact bug lines; some bugs cross files (e.g., wrong imports) and a few compile while silently corrupting state. Evaluating ten LLMs on bug identification, localization, and repair, the paper finds that the best model, Claude 3.5 Sonnet, identifies error types only about 36–56% of the time by language, with far lower repair success

Load-bearing premise

The dataset's bugs are generated by rule-based AST modifications, not collected from real developer mistakes; if these synthetic bugs are not representative of the cross-file, semantic defects developers actually debug—especially since about 95% of them fail at compile time—the benchmark may measure syntax-error spotting in long files rather than repository-level debugging.

Editorial extensions

If this is right

  • Bug identification accuracy for the strongest model hovers around 40–56% depending on language, and exact repair (Pass@1) stays below about 17% in every language, so repository-level repair is far from solved.
  • Multiple errors are the hardest category: Claude 3.5 Sonnet identifies only 3.66% of multiple-error instances correctly while reaching 54.15% on syntax errors, making error count a first-order difficulty factor.
  • Longer files hurt: most models lose several points when input grows past 500 tokens, and the gap widens toward 10,000 tokens.
  • Models find one bug among several much more easily than all bugs (OBL rises while ABL collapses for multi-error instances), suggesting partial localization is a realistic intermediate capability.
  • Performance on repositories created before April 2024 is higher than after, indicating data leakage contributes to the measured scores.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because roughly 95% of injected bugs fail at compile time and syntax errors make up 53% of instances, the benchmark likely overweights token-level syntax faults; a version built from failing-test or post-compilation behavior would better isolate semantic, cross-file debugging.
  • The AST-injection method could be extended to mutate across function boundaries and conditionals that compile but change outputs, aligning the benchmark with the silent-logic failures the paper observes in its 5% compilable subset.
  • The OBL-versus-ABL gap suggests a promising agent design: iteratively request one error location and repair, re-run tests, and repeat, rather than asking for all errors at once.
  • The data-leakage result implies that as model training cutoffs move, RepoDebug's post-2024 repositories will age into contamination; benchmarks should re-derive held-out splits from newest repositories.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces RepoDebug, a repository-level debugging benchmark constructed by AST-based injection of 22 error subtypes (syntax, reference, logic, multiple) into files from 63 GitHub repositories across 8 programming languages. It defines three tasks—bug identification, bug localization, and automatic program repair—and evaluates 10 LLMs, reporting low overall performance, with syntax errors being easiest, multiple errors hardest, and performance degrading as code length grows. The paper claims this is the first multi-task, multi-language repository-level debugging dataset.

Significance. If the validity concerns were resolved, RepoDebug would fill a real gap: existing benchmarks are either function-level or single-task, while RepoDebug combines BI, BL, and APR across 8 languages and 22 error subtypes. The evaluation of 10 models on a unified taxonomy is a useful data point for the community. The authors provide detailed appendices and plan to release the dataset. However, the current manuscript contains internal contradictions about dataset size and compilability, and the APR metric is conditional on localization, so the headline conclusions are not yet supported as stated.

major comments (4)
  1. [§3.2, Table 7, Table 1] Instance counts are internally inconsistent. Section 3.2 reports 34,457 train and 5,438 test instances, but summing Table 7 by language gives 43,618 train instances and 5,438 test instances; Table 1 reports 30,696 total instances, which matches neither 34,457+5,438=39,895 nor 43,618+5,438=49,056. The paper must correct these numbers and explain the derivation, because the reported scale and per-language distributions are part of the benchmark description.
  2. [§3.3, Appendix D, §3.5] A load-bearing validity claim is contradicted by the paper's own appendix. Section 3.3 states that AST-guided injection 'ensures that the modified code remains compilable' and that 'all errors are injected using AST-guided methods to ensure code remains structurally valid,' but Appendix D reports that only approximately 5% of injected bugs compile without immediate execution failure, and Section 3.5 states that 53.38% of instances are syntax errors (9 of 22 subtypes). The manual check in Section 3.4 verifies only that a bug exists and matches its subtype, not that the injected defect resembles a real repository-level debugging scenario. As a result, the headline claim that RepoDebug evaluates repository-level debugging may largely reduce to syntax-error detection in long files. Please rebalance the error distribution, report results separately for compilable vs non-compilable bugs, or su
  3. [§F.1, Eqs. (4)–(5)] The APR metrics are computed only over lines on which the predicted and actual bug locations overlap. Edit Similarity and Exact Match are averaged over k in the intersection Li ∩ L*i, and Pass@1 is described in the same conditional way. Consequently, a model that fails to localize an error is not penalized in APR; the reported APR numbers are not end-to-end repair rates and are not comparable to APR results in prior work. Please report unconditional APR (e.g., using the reference patch or test-based pass@1 over the full bug location list) or clearly state and interpret the conditional definition.
  4. [§1, §3.1, Figure 9] The 'repository-level' claim is not supported by the evaluation setup. Although instances are drawn from repositories and some injected bugs have cross-file effects (Appendix C), the prompt in Figure 9 provides only the buggy code file with no repository context; cross-file bugs are introduced as single-line modifications. Models cannot reason about other files, so the task is effectively file-level debugging on code extracted from repositories. Please either provide repository context in the evaluation (e.g., related files or a repository map) or rename the benchmark and soften the 'repository-level debugging' claim.
minor comments (5)
  1. [Contributions vs §4.2] The contributions bullet states '3 open-source and 7 closed-source models', while Section 4.2 says 'three closed-source models and seven open-source models'. The counts are reversed; Table 3 indicates 10 models total, with 3 closed-source (GPT-4o, GPT-4o-mini, Claude 3.5 Sonnet) and 7 open-source.
  2. [§4.5] Section 4.5 (Data Leakage) is empty; the leakage analysis appears in Section 4.6 after the qualitative analysis. Please reorder the content.
  3. [Throughout] The model name 'Claude 3.5 Sonnect' is misspelled throughout; it should be 'Claude 3.5 Sonnet'.
  4. [§F.1] The definition of Pass@1 is not formalized. Equations (4) and (5) define ES and EM, but Pass@1 is only described in prose. Please include the formal definition, especially because the conditional nature of the metric is central.
  5. [Table 3] Some formatting inconsistencies in zero values (e.g., 0.00 vs 0) could be unified for readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: benchmark construction and model evaluations are externally defined; no prediction reduces to an input by construction.

full rationale

RepoDebug is a dataset construction and empirical evaluation paper. The central claims are (a) that the dataset contains 3 tasks, 8 languages, and 22 bug subtypes, and (b) that evaluated LLMs perform poorly on it. Neither claim is derived from the dataset's own parameters. Bug instances are generated by AST-guided injection into code from external GitHub repositories, with labels (subtype, location, correct line) fixed by the construction process and checked manually; model outputs are then scored against those externally fixed labels. There is no fitted parameter that is later renamed as a prediction, no equation in which the target quantity is defined in terms of the input quantity, and no load-bearing self-citation. The paper cites prior work (DebugBench, MdEval, iBiR) for taxonomy and metrics, but these are external and not used to justify the empirical results. The authors do cite their own prior work (Liu et al. 2020; He et al. 2025), but only in passing for methodological conventions (manual sampling checks), not as evidence for the central claim. The closest concern is construct validity, not circularity: Section 3.3 states AST injection 'ensur[es] that the modified code remains compilable,' while Appendix D reports that only about 5% of injected bugs compile without immediate execution failure, and Section 3.5 reports 53.38% syntax errors. This internal inconsistency is a substantive threat to whether the benchmark measures repository-level semantic debugging rather than syntax-error detection, but it does not make the evaluation circular: the model scores are still genuinely measured against labels that were not derived from the models. The finding is therefore a normal non-circular outcome.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The dataset's validity rests on curation choices and domain assumptions listed above. These are not derived from the data itself, and if any fails, the benchmark may not measure what it claims.

free parameters (3)
  • repository star threshold = 100 stars
    Chosen to ensure code quality; arbitrary cutoff that shapes which repositories enter the benchmark.
  • repository creation cutoff = 2022-01-01
    Chosen to mitigate data leakage; assumes models have not seen these repositories.
  • maximum error subtypes per file = 5
    Sampling cap; affects the number of instances generated per code file.
assumptions (4)
  • domain assumption AST-based bug injection produces realistic repository-level bugs that are representative of developer mistakes.
    Stated in Section 3.3, citing iBiR, but not validated against real bugs. The benchmark's validity depends on this.
  • domain assumption Repositories created after January 1, 2022 mitigate data leakage from LLM training corpora.
    Used in Section 3.2 to justify data selection. If false, reported model performance could be inflated.
  • domain assumption The 22 error subtypes cover the common repository-level debugging errors across 8 languages.
    Assumed in Section 3.3 and used to define the Bug Identification task. The taxonomy conflates error count with error type for multiple errors.
  • domain assumption Injected bugs, even when they compile, are detectable by the repository's existing test suites.
    Claimed in Section 3.3 as observability. Only about 5% of bugs compile; the rest cause immediate execution failures, making most instances syntax-error-like.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RepoDebug: Repository-Level Multi-Task and Multi-Language Debugging Evaluation of Large Language Models." pith.science (2026). https://pith.science/paper/7MEWTP7U

@misc{pith2026250904078,
  author       = {Pith},
  title        = {Pith review of: RepoDebug: Repository-Level Multi-Task and Multi-Language Debugging Evaluation of Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7MEWTP7U}},
  note         = {Machine review of arXiv:2509.04078}
}
read the original abstract

Large Language Models (LLMs) have exhibited significant proficiency in code debugging, especially in automatic program repair, which may substantially reduce the time consumption of developers and enhance their efficiency. Significant advancements in debugging datasets have been made to promote the development of code debugging. However, these datasets primarily focus on assessing the LLM's function-level code repair capabilities, neglecting the more complex and realistic repository-level scenarios, which leads to an incomplete understanding of the LLM's challenges in repository-level debugging. While several repository-level datasets have been proposed, they often suffer from limitations such as limited diversity of tasks, languages, and error types. To mitigate this challenge, this paper introduces RepoDebug, a multi-task and multi-language repository-level code debugging dataset with 22 subtypes of errors that supports 8 commonly used programming languages and 3 debugging tasks. Furthermore, we conduct evaluation experiments on 10 LLMs, where Claude 3.5 Sonnect, the best-performing model, still cannot perform well in repository-level debugging.

Figures

Figures reproduced from arXiv: 2509.04078 by the authors.

Figure 1
Figure 1. Illustration of code debugging examples with [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Data construction process of RepoDebug. Firstly, the source data of RepoDebug is collected and filtered from GitHub. Then buggy code is constructed and implanted based on the Abstract Syntax Tree (AST). Additionally, we conduct manual checks to ensure that RepoDebug contains error types, error locations, and pairs of buggy and correct code. files that meet the specified requirements. The code files are then edited u… view at source ↗
Figure 3
Figure 3. Four instances of syntax errors. Red indicates original code; green indicates injected errors. [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Four instances of reference errors. Red indicates original code; green indicates injected errors. [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]
Figure 5
Figure 5. Figure 5: Three instances of logic errors. Red indicates original code; green indicates injected errors. [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]
Figure 6
Figure 6. Figure 6: Two instances of multiple errors. Red indicates original code; green indicates injected errors. [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]
Figure 7
Figure 7. Figure 7: An instances of error spanning across files. Red indicates original code; green indicates injected errors. [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]
Figure 8
Figure 8. Figure 8: An instances of compilable error. Red indicates original code; green indicates injected errors. [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]
Figure 9
Figure 9. Figure 9: Prompt for code debugging evaluation of large language models. [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]
Figure 10
Figure 10. Figure 10: We provide an example of a buggy instance, accompanied by successful and failed model outputs and their associated test results. Red underlines indicate erroneous code, while green underlines denote the corrected code [PITH_FULL_IMAGE:figures/full_fig_p026_10.png]
Figure 11
Figure 11. Figure 11: An example where a model successfully analyzes the error type and suggests a fix, but fails to identify [PITH_FULL_IMAGE:figures/full_fig_p027_11.png]
Figure 12
Figure 12. Figure 12: An example where a model incorrectly identifies multiple errors. [PITH_FULL_IMAGE:figures/full_fig_p028_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

45 extracted references · 11 canonical work pages

  1. [1]

    https://doi.org/10.48550/arXiv.2402.19173 Starcoder 2 and the stack v2: The next generation

    2024. https://doi.org/10.48550/arXiv.2402.19173 Starcoder 2 and the stack v2: The next generation . Preprint, arXiv:2402.19173

    arXiv doi:10.48550/arxiv.2402.19173 2024

  2. [2]

    Md Mahim Anjum Haque, Wasi Uddin Ahmad, Ismini Lourentzou, and Chris Brown. 2023. https://doi.org/10.1109/APR59189.2023.00009 Fixeval: Execution-based evaluation of program fixes for programming problems . In 2023 IEEE/ACM International Workshop on Automated Program Repair (APR), pages 11--18, Melbourne, Australia. IEEE

    work page arXiv 2023

  3. [3]

    Nghi Bui, Yue Wang, and Steven C.H. Hoi. 2022. https://doi.org/10.18653/v1/2022.findings-emnlp.57 Detect-localize-repair: A unified framework for learning to debug with codet5 . In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 812--823, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics

    work page doi:10.18653/v1/2022.findings-emnlp.57 2022

  4. [4]

    Wei Cheng, Yuhan Wu, and Wei Hu. 2024. https://doi.org/10.18653/v1/2024.acl-long.431 Dataflow-guided retrieval augmentation for repository-level code completion . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7957--7977, Bangkok, Thailand. Association for Computational Linguistics

    doi:10.18653/v1/2024.acl-long.431 2024

  5. [5]

    Dekun Dai, MingWei Liu, Anji Li, Jialun Cao, Yanlin Wang, Chong Wang, Xin Peng, and Zibin Zheng. 2025. https://doi.org/10.48550/arXiv.2504.06939 Feedbackeval: A benchmark for evaluating large language models in feedback-driven code repair tasks . Preprint, arXiv:2504.06939

    work page doi:10.48550/arxiv.2504.06939 2025

  6. [6]

    DeepSeek-AI , Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y. Wu, Yukun Li, Huazuo Gao, Shirong Ma, Wangding Zeng, Xiao Bi, Zihui Gu, Hanwei Xu, Damai Dai, Kai Dong, Liyue Zhang, Yishi Piao, Zhibin Gou, Zhenda Xie, Zhewen Hao, Bingxuan Wang, Junxiao Song, Deli Chen, Xin Xie, Kang Guan, Yuxiang You, Aixin Liu, Qiushi Du, Wenjun Ga...

    arXiv doi:10.48550/arxiv.2406.11931 2024

  7. [7]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    arXiv 2025

  8. [8]

    Jiawei Guo, Ziming Li, Xueling Liu, Kaijing Ma, Tianyu Zheng, Zhouliang Yu, Ding Pan, Yizhi LI, Ruibo Liu, Yue Wang, Shuyue Guo, Xingwei Qu, Xiang Yue, Ge Zhang, Wenhu Chen, and Jie Fu. 2024. https://doi.org/10.48550/ARXIV.2404.03543 Codeeditorbench: Evaluating code editing capability of large language models . arXiv preprint

    arXiv doi:10.48550/arxiv.2404.03543 2024

Show all 45 references
  1. [9]

    Mengliang He, Jiayi Zeng, Yankai Jiang, Wei Zhang, Zeming Liu, Xiaoming Shi, and Aimin Zhou. 2025. https://doi.org/10.18653/v1/2025.findings-acl.425 F low2 C ode: Evaluating large language models for flowchart-based code generation capability . In Findings of the Association f...

    2025 doi

  2. [10]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou...

    2024 doi arXiv

  3. [11]

    Faria Huq, Masum Hasan, Md Mahim Anjum Haque, Sazan Mahbub, Anindya Iqbal, and Toufique Ahmed. 2022. https://doi.org/10.1016/j.infsof.2021.106765 Review4repair: Code review aided automatic program repairing . Information and Software Technology, 143:106765

    2022

  4. [12]

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974

    2024 arXiv

  5. [13]

    Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. https://doi.org/10.48550/arXiv.2406.00515 A survey on large language models for code generation . Preprint, arXiv:2406.00515

    2024 doi arXiv

  6. [14]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. https://openreview.net/forum?id=VTF8yNQM66 SWE -bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Re...

    2024

  7. [15]

    Rafael-Michael Karampatsis and Charles Sutton. 2020. https://doi.org/10.1145/3379597.3387491 How often do single-statement bugs occur?: The manysstubs4j dataset . In Proceedings of the 17th International Conference on Mining Software Repositories, MSR ’20, page 573–577. ACM

    2020

  8. [16]

    Bissyand\' e , Jacques Klein, and Yves Le Traon

    Ahmed Khanfir, Anil Koyuncu, Mike Papadakis, Maxime Cordy, Tegawende F. Bissyand\' e , Jacques Klein, and Yves Le Traon. 2023. https://doi.org/10.1145/3542946 ibir: Bug-report-driven fault injection . ACM Trans. Softw. Eng. Methodol., 32(2)

    2023 doi

  9. [17]

    Mizuki Kondo, Daisuke Kawahara, and Toshiyuki Kurabayashi. 2024. https://doi.org/10.18653/v1/2024.naacl-srw.15 Improving repository-level code search with text conversion . In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational...

    2024 doi

  10. [18]

    Claire Le Goues, Michael Pradel, and Abhik Roychoudhury. 2019. https://doi.org/10.1145/3318162 Automated program repair . Commun. ACM, 62(12):56–65

    2019 doi

  11. [19]

    Jia Li, Ge Li, Xuanming Zhang, YunFei Zhao, Yihong Dong, Zhi Jin, Binhua Li, Fei Huang, and Yongbin Li. 2024. Evocodebench: An evolving code generation benchmark with domain-specific evaluations. In Advances in Neural Information Processing Systems, volume 37, pages 57619--576...

    2024

  12. [20]

    Shukai Liu, Linzheng Chai, Jian Yang, Jiajun Shi, He Zhu, Liran Wang, Ke Jin, Wei Zhang, Hualei Zhu, Shuyue Guo, Tao Sun, Jiaheng Liu, Yunlong Duan, Yu Hao, Liqun Yang, Guanglin Niu, Ge Zhang, and Zhoujun Li. 2024 a . https://doi.org/10.48550/ARXIV.2411.02310 Mdeval: Massively...

    2024 doi arXiv

  13. [21]

    Tianyang Liu, Canwen Xu, and Julian McAuley. 2024 b . https://openreview.net/forum?id=pPjZIOuQuF Repobench: Benchmarking repository-level code auto-completion systems . In The Twelfth International Conference on Learning Representations

    2024

  14. [22]

    Zeming Liu, Haifeng Wang, Zheng-Yu Niu, Hua Wu, Wanxiang Che, and Ting Liu. 2020. https://doi.org/10.18653/v1/2020.acl-main.98 Towards conversational recommendation over multi-type dialogs . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguis...

    2020 doi

  15. [23]

    Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shuj...

    2021 doi arXiv

  16. [24]

    OpenAI. 2024. https://doi.org/10.48550/arXiv.2303.08774 Gpt-4 technical report . Preprint, arXiv:2303.08774

    2024 doi arXiv

  17. [25]

    Muhammad Shihab Rashid, Christian Bock, Yuan Zhuang, Alexander Buchholz, Tim Esler, Simon Valentin, Luca Franceschi, Martin Wistuba, Prabhu Teja Sivaprasad, Woo Jung Kim, Anoop Deoras, Giovanni Zappella, and Laurent Callot. 2025. https://doi.org/10.48550/arXiv.2504.08703 Swe-p...

    2025 doi arXiv

  18. [26]

    Baptiste Rozi \`e re, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, J \'e r \'e my Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, ...

    2024 doi arXiv

  19. [27]

    Yuling Shi, Songsong Wang, Chengcheng Wan, and Xiaodong Gu. 2024. https://arxiv.org/abs/2410.01215 From code to correctness: Closing the last mile of code generation with hierarchical debugging . Preprint, arXiv:2410.01215

    2024

  20. [28]

    Disha Shrivastava, Denis Kocetkov, Harm de Vries, Dzmitry Bahdanau, and Torsten Scholak. 2023. https://doi.org/10.48550/arXiv.2306.10998 Repofusion: Training code models to understand your repository . Preprint, arXiv:2306.10998

    2023 doi arXiv

  21. [29]

    Jan Strich, Florian Schneider, Irina Nikishina, and Chris Biemann. 2024. https://doi.org/10.18653/v1/2024.acl-srw.28 On improving repository-level code QA for large language models . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vo...

    2024 doi

  22. [30]

    Runchu Tian, Yining Ye, Yujia Qin, Xin Cong, Yankai Lin, Yinxu Pan, Yesai Wu, Hui Haotian, Liu Weichuan, Zhiyuan Liu, and Maosong Sun. 2024 a . https://doi.org/10.18653/v1/2024.findings-acl.247 D ebug B ench: Evaluating debugging capability of large language models . In Findin...

    2024 doi

  23. [31]

    Runchu Tian, Yining Ye, Yujia Qin, Xin Cong, Yankai Lin, Yinxu Pan, Yesai Wu, Haotian Hui, Weichuan Liu, Zhiyuan Liu, and Maosong Sun. 2024 b . https://doi.org/10.48550/arXiv.2401.04621 Debugbench: Evaluating debugging capability of large language models . Preprint, arXiv:2401.04621

    2024 doi arXiv

  24. [32]

    Hanbin Wang, Zhenghao Liu, Shuo Wang, Ganqu Cui, Ning Ding, Zhiyuan Liu, and Ge Yu. 2024. https://doi.org/10.18653/v1/2024.findings-acl.124 INTERVENOR : Prompting the coding ability of large language models with the interactive chain of repair . In Findings of the Association ...

    2024 doi

  25. [33]

    Weishi Wang, Yue Wang, Steven Hoi, and Shafiq Joty. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.430 Towards low-resource automatic program repair with meta-learning and pretrained language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Lan...

    2023 doi

  26. [34]

    Jian Yang, Jiajun Zhang, Jiaxi Yang, Ke Jin, Lei Zhang, Qiyao Peng, Ken Deng, Yibo Miao, Tianyu Liu, Zeyu Cui, et al. 2024 a . https://arxiv.org/abs/2412.11990 Execrepobench: Multi-level executable code completion evaluation . Preprint, arXiv:2412.11990

    2024 arXiv

  27. [35]

    Weiqing Yang, Hanbin Wang, Zhenghao Liu, Xinze Li, Yukun Yan, Shuo Wang, Yu Gu, Minghe Yu, Zhiyuan Liu, and Ge Yu. 2024 b . https://doi.org/10.48550/arXiv.2408.05006 Enhancing the code debugging ability of llms via communicative agent based data refinement . Preprint, arXiv:2408.05006

    2024 doi arXiv

  28. [36]

    Michihiro Yasunaga and Percy Liang. 2021. https://proceedings.mlr.press/v139/yasunaga21a.html Break-it-fix-it: Unsupervised learning for program repair . In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Rese...

    2021

  29. [37]

    Daoguang Zan, Ailun Yu, Wei Liu, Dong Chen, Bo Shen, Wei Li, Yafen Yao, Yongshun Gong, Xiaolin Chen, Bei Guan, Zhiguang Yang, Yongji Wang, Qianxiang Wang, and Lizhen Cui. 2024. https://doi.org/10.48550/arXiv.2403.16443 Codes: Natural language to code repository via multi-layer...

    2024 doi arXiv

  30. [38]

    Chenyuan Zhang, Hao Liu, Jiutian Zeng, Kejing Yang, Yuhong Li, and Hui Li. 2024 a . https://doi.org/10.1145/3639478.3643065 Prompt-enhanced software vulnerability detection using chatgpt . In Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineerin...

    2024

  31. [39]

    Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.151 R epo C oder: Repository-level code completion through iterative retrieval and generation . In Proceedings of t...

    2023 doi

  32. [40]

    Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. 2024 b . https://doi.org/10.18653/v1/2024.acl-long.737 C ode A gent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges . In Proceedings of the 62nd Annual Meeting of th...

    2024 doi

  33. [41]

    Yuze Zhao, Zhenya Huang, Yixiao Ma, Rui Li, Kai Zhang, Hao Jiang, Qi Liu, Linbo Zhu, and Yu Su. 2024. https://doi.org/10.18653/v1/2024.findings-acl.973 R e P air: Automated program repair with process-based feedback . In Findings of the Association for Computational Linguistic...

    2024 doi

  34. [42]

    Li Zhong, Zilong Wang, and Jingbo Shang. 2024 a . https://doi.org/10.18653/v1/2024.findings-acl.49 Debug like a human: A large language model debugger via verifying runtime execution step by step . In Findings of the Association for Computational Linguistics: ACL 2024, pages 8...

    2024 doi

  35. [43]

    Zhiyuan Zhong, Sinan Wang, Hailong Wang, Shaojin Wen, Hao Guan, Yida Tao, and Yepang Liu. 2024 b . https://doi.org/10.48550/ARXIV.2410.09414 Advancing bug detection in fastjson2 with large language models driven unit test generation . arXiv preprint

    2024 doi

  36. [44]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  37. [45]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.