Pith. sign in

REVIEW 4 major objections 6 minor 73 references

From Empirical Evaluation to Context-Aware Enhancement: Repairing Regression Errors with LLMs

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Supplying the bug-inducing change to LLM prompts lifts correct repairs from 9 to 16 on a 99-bug regression benchmark.

desk verdict A much-needed Java regression-bug benchmark with a clean validation pipeline, but the headline BIC-context result is confounded by a simultaneous change in prompt instruction and needs an ablation before the causal claim can stand. read the letter →

arxiv 2506.13182 v2 pith:HAUZVPAP submitted 2025-06-16 cs.SE cs.AI

Anh Ho , Thanh Le-Cong , Bach Le , Christine Rizkallah This is my paper
classification cs.SEcs.AI
keywords regressionbugsautomatedprogramrepairlargelanguagemodelsbug-inducingchangesbenchmarkJavapromptengineeringempiricalstudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that large language model-based automated program repair (APR) performs poorly on real-world regression bugs when it receives only the buggy function and failing test, but improves markedly when the prompt also includes the bug-inducing change (BIC): the diff and commit message of the commit that first broke the behavior. To test this, the authors build RegMiner4APR, a benchmark of 99 Java regression bugs mined from 32 widely used GitHub repositories, each with its bug-inducing and bug-fixing commits, four version snapshots, and a regression-witnessing test. In their experiments, traditional APR tools fix zero bugs, while the best LLM-based baselines fix 9; augmenting ChatGPT-4o's conversational repair with BIC context raises correct repairs to 16, about 1.8x more. The paper's qualitative analysis shows the BIC context helps the model locate the root cause and decide whether to fully or partially revert prior code.

What carries the argument

The regression bug instance is the central object: four snapshots (bug-inducing commit, its parent, bug-fixing commit, its parent), a regression-witnessing test, and the minimized bug-inducing change (BIC) diff with its commit message. The mechanism is prompt-based APR: ChatGPT is told it is an Automated Program Repair Tool, given the buggy function, the failing test names and error messages, and either the BIC diff or, when the buggy function was unchanged, a note pointing to the commit message, followed by a chain-of-thought instruction to find the root cause and provide a correct function. A validation tool checks that the migrated test passes before the BIC and fails after it, and that the fix commit passes all tests, which makes the 16-versus-9 comparison interpretable.

What would settle it

Re-run the best BIC-augmented configuration (ChatGPT-4o with conversation) on all 99 bugs, replacing each true bug-inducing diff with a same-sized diff from an unrelated commit of the same project; if the repair count remains near 16 instead of dropping toward the 9 of the no-BIC baseline, the claimed benefit is not caused by the semantic content of the BIC. As a second check, run the 16 patches judged correct against a held-out second regression test for the same bug that was not used during repair; any failures would indicate the single-test oracle allowed test overfitting.

Watch

Extended reading notes

Core claim

The central claim is that incorporating bug-inducing change (BIC) information into LLM prompts significantly improves the repair of regression bugs. On the new RegMiner4APR benchmark, which contains 99 real-world Java regression bugs from 32 GitHub repositories, ChatGPT-4o with conversational prompting and BIC context correctly repairs 16 bugs versus 9 without BIC, and the union of both ChatGPT models with BIC repairs 18 bugs versus 9 without. The BIC context supplied to the model is the extracted, minimized diff of the bug-inducing commit plus its commit message, along with an instruction to reason about the root cause from that change. The paper interprets this as evidence that regression-specific version-history context helps LLMs fix regressions more effectively than test-feedback alone.

Load-bearing premise

The load-bearing premise is that the mining and validation pipeline correctly identifies the true bug-inducing commit for each of the 99 regressions, and that the single regression-witnessing test is strong enough to judge whether a patch is genuinely correct rather than merely passing that one test.

Editorial extensions

If this is right

  • APR pipelines for regressions can be extended to mine the bug-inducing commit (e.g., from CI/build history) and feed its diff and message into the prompt, achieving a roughly 1.8x gain without retraining.
  • The benchmark's four-snapshot structure with grounded BIC information provides a reusable testbed for regression-specific repair, updating an area where the previous Java dataset (CIBugs) lacked bug-inducing commit information.
  • The frequent 'revert to previous statement' pattern suggests that template-based APR could add a version-history-aware reversion operator, provided it can judge when reversion is semantically justified.
  • Combining BIC-augmented prompting with regression-specific fine-tuning (e.g., fine-tuning on the benchmark's bugs) is a natural next step that could push the 16-of-99 correct rate higher.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A randomized ablation that replaces the true BIC diff with an unrelated same-size diff from the same project could isolate whether the 16-vs-9 gain comes from the semantic content of the bug-inducing change or simply from extra input tokens.
  • Because 77 of the 99 regressions are local (the BIC directly touches the buggy code), the reported gain may concentrate there; remote and unmask regressions may need additional context beyond the commit message.
  • The BIC prompt acts as a coarse fault localizer by directing attention to the changed lines, so the same technique could apply to any bug whose introducing commit can be identified, not only to regressions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents RegMiner4APR, a benchmark of 99 Java regression bugs mined with RegMiner and validated with a custom four-snapshot tool, and uses it to evaluate traditional APR tools, fine-tuned LLMs, and prompt-based LLMs. It reports that traditional tools fix no bugs, that RepairLLaMA and ChatGPT-4o with conversation each fix 9 bugs, and that augmenting the prompt with bug-inducing change (BIC) information raises ChatGPT-4o conversation repairs from 9 to 16 (1.8x). A qualitative analysis discusses full/partial reversion, fault localization, and plausible-but-incorrect patches.

Significance. If the central claim holds, the result is useful for the APR community: it provides a reproducible, up-to-date Java regression benchmark, a transparent validation pipeline, and evidence that regression-specific context can improve LLM-based repair. The paper's strengths include the detailed benchmark construction funnel (Section 3.2), the explicit four-snapshot regression validation condition, the broad comparison across 15 repair methods, and the public artifacts for benchmark, framework, and repair prompts. The qualitative case studies, especially RegressionBug-2, 69, 84, and 99, give concrete insight into how BIC context affects model reasoning. However, the headline result is currently underdetermined by the experimental design because the BIC prompt differs from the baseline in instruction content as well as in context, and the abstract/body inconsistency weakens the paper's presentation.

major comments (4)
  1. [Abstract vs. Sections 3.1, 6.1] The abstract provided with the paper states that the benchmark includes 200 Java and Python regression bugs and that BIC enhancement yields 1.6x more successful repairs, while the body consistently reports a 99-bug Java-only benchmark and a 1.8x improvement (16 vs. 9 for ChatGPT-4o conversation). These are different claims about the benchmark's composition and the size of the effect. This is not a minor wording issue: readers and reviewers cannot tell which experiment was actually conducted. The abstract must be aligned with the body, and if a separate 200-bug Java/Python version exists, its results and construction must be described in the paper; otherwise the abstract should be corrected.
  2. [Sections 3.3.3, 3.3.4, RQ3 (Section 6.1)] The BIC treatment is confounded with the reasoning instruction. The baseline prompt ends with "Let's think step by step to fix the bug. Please provide a correct function," while the BIC prompt ends with "Please think step by step to identify the root cause from the regression-inducing change information... Then, provide a correct version of the function." Thus the 9-to-16 improvement could be caused by the more directive chain-of-thought instruction, by the added BIC content, or by their interaction. The paper reports no control condition that keeps the instruction constant and varies only the BIC content. This is an internal-validity threat to the central claim of RQ3; an instruction-matched ablation is needed before attributing the gain to BIC information.
  3. [Section 3.2.3, Section 3.4, Section 6.1] The regression-witnessing test is a single migrated test, and Step 3 deliberately requires that the buggy version fails only that test. This makes test-passing a very weak behavioral oracle: a patch that passes the single test may overfit the test while remaining incorrect. The authors mitigate this with manual semantic-equivalence assessment, which is a reasonable practice, but the manual assessment is the only evidence separating correct from plausible patches for the 16-vs-9 comparison. The paper should report inter-rater agreement or an explicit second-validation procedure for the BIC condition, and should discuss how many of the 16 BIC fixes are distinguished from plausible patches solely by human judgment rather than by any additional behavioral test.
  4. [Section 6.1, Table 5] The phrase "significantly improves" is used without any statistical analysis. The main comparison is 16 of 99 versus 9 of 99 for one model/configuration pair; even ignoring the prompt confound, this is a small difference on a small sample, and the paper does not report confidence intervals, a paired test, or a correction for multiple comparisons. Either add appropriate statistical analysis or soften the language to "improves in this sample" and let the qualitative examples carry the strength of the claim.
minor comments (6)
  1. [Section 4.2.1 and Section 4.2.2] The text says bug IDs range from 1 to 100 (Section 4.2.1) and later refers to "100 patches" (Section 4.2.2), but the benchmark is stated to contain 99 bugs. This inconsistency should be fixed, and the numbering convention should be clarified if one ID is intentionally absent.
  2. [Section 2.2.2 and Table 4] The text mentions GenProg and Kali, while Table 4 lists jGenProg and jKali; please make the tool names consistent throughout and note that the j-prefixed tools are the Java ASTOR variants used through Cerberus.
  3. [Section 3.3.2] The names "Incoder" and "InCoder" are used interchangeably; please standardize the spelling and the capitalization.
  4. [Figures 2, 4, 16, 18, 20] Several figures contain diff text with inconsistent spacing and line-wrapping artifacts (for example, Figure 2 and Figure 4), making the actual changed lines hard to read. Please regenerate these figures with a monospaced, syntax-highlighted diff format.
  5. [Section 3.3.3] The sentence "We observe that there are four main cases in which feedback is provided to the model" is followed by a list, but the conversational strategy is described earlier as stopping once a plausible patch is found; clarify whether feedback is also generated when the model returns no code or times out, and how those cases interact with the maximum conversation length.
  6. [Section 7.3.2] The external-validity discussion of data leakage cites prior work but does not check the specific 2024-2025 models and repositories used in this paper; a brief note on whether any of the 32 projects or their bug-fix commits postdate the training cutoff would strengthen the argument.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the BIC gain is a measured experimental outcome, not a construction-derived equivalence.

full rationale

The paper is an empirical evaluation rather than a derivation. The central RQ3 result—16 versus 9 correct repairs for ChatGPT-4o with conversation—is a measured count produced by running LLMs over the 99 benchmark bugs and validating patches against compiled tests plus manual semantic equivalence checks (Sections 3.4 and 6.1). Nothing in the benchmark construction (Section 3.2.3) numerically forces that count: Step 2 checks pass/fail conditions on four snapshots, and Step 3 filters for single-failing-test utility, but neither defines the number of future repairs nor encodes the BIC prompt outcome. No fitted parameter is renamed as a prediction, and no equation is defined in terms of its own output. The authors' use of RegMiner, FLAMES, and their own validation tool is instrumental and does not form a load-bearing self-citation chain. The skeptic's observation that the BIC prompt also changed the closing chain-of-thought instruction (Section 3.3.4 versus 3.3.3) identifies a real internal-validity confound for causal attribution, but it is not circularity: the 16-versus-9 difference is not equivalent by construction to the prompt text. I therefore find no circular step; the low score reflects only the self-referential nature of authors building and evaluating their own benchmark, which is a validity concern rather than circular reasoning.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The central claim rests on the validity of the mined regression instances, the adequacy of a single test as an oracle, the manual correctness assessment, and the assumption that LLM training data leakage is not significant. The experimental hyperparameters are hand-chosen but held constant across the key comparison, so they do not by themselves explain the BIC gain.

free parameters (4)
  • Patch sampling budget = 10 candidates per bug
    Set for both fine-tuning-based and prompt-based methods to control API and compute cost. It is held constant across compared configurations, so it does not explain the BIC improvement, but it does bound the number of plausible patches that can be discovered.
  • Conversation length threshold = 5 rounds
    Maximum number of prompt-response exchanges for conversational APR. The 16-vs-9 headline comparison uses this threshold, and larger values could change the result.
  • Generation temperature = 1.0
    Temperature for all LLM generation, set to control output diversity. This is a hand-chosen experimental setting that can affect success counts.
  • Beam size = 10
    Beam size for fine-tuned models, set to avoid out-of-memory crashes and to keep generation cost manageable. It constrains the candidate space.
assumptions (4)
  • domain assumption RegMiner's mining heuristics correctly identify bug-inducing and bug-fixing commit pairs.
    Sections 2.1.3 and 3.2.2: the benchmark is built on RegMiner's measurement heuristic and test migration algorithm. If a mined pair is wrong, BIC context is misleading and the regression characterization fails.
  • domain assumption The four-snapshot validation condition is a sufficient definition of a true regression bug.
    Section 3.2.3 Step 2: a bug is accepted if a single migrated test passes before the bug-inducing commit and on the fixing commit, and fails on the bug-inducing commit and before the fixing commit. This excludes regression bugs that affect more than one observable behavior.
  • domain assumption Manual semantic equivalence assessment accurately separates correct from merely plausible patches.
    Sections 3.4 and 7.3.1: correct patches are determined by the authors' manual inspection following prior practice. This is a known subjective step and directly affects all correct-patch counts.
  • domain assumption LLM training data leakage is not a significant threat for this APR evaluation.
    Section 7.3.2: the paper relies on prior work [20] to argue that training corpora lack aligned bug-fix pairs. If leakage exists, the measured BIC improvement could be inflated.
invented entities (1)
  • RegMiner4APR benchmark independent evidence
    purpose: Evaluation dataset of 99 real-world Java regression bugs with bug-inducing and fixing snapshots, plus a reproduction framework.
    A new resource introduced by the paper and publicly hosted on GitHub. It is not a theoretical entity, but it is a new artifact that the entire empirical study depends on.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Empirical Evaluation to Context-Aware Enhancement: Repairing Regression Errors with LLMs." pith.science (2026). https://pith.science/paper/HAUZVPAP

@misc{pith2026250613182,
  author       = {Pith},
  title        = {Pith review of: From Empirical Evaluation to Context-Aware Enhancement: Repairing Regression Errors with LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HAUZVPAP}},
  note         = {Machine review of arXiv:2506.13182}
}
read the original abstract

[...] Since then, various APR approaches, especially those leveraging the power of large language models (LLMs), have been rapidly developed to fix general software bugs. Unfortunately, the effectiveness of these advanced techniques in the context of regression bugs remains largely unexplored. This gap motivates the need for an empirical study evaluating the effectiveness of modern APR techniques in fixing real-world regression bugs. In this work, we conduct an empirical study of APR techniques on regression bugs. To facilitate our study, we introduce RegressionBug4APR, a high-quality benchmark of Java and Python regression bugs integrated into a framework designed to facilitate APR research. The current benchmark includes 200 regression bugs collected from widely used real-world GitHub repositories. We begin by conducting an in-depth analysis of the benchmark, demonstrating its diversity and quality. Building on this foundation, we empirically evaluate the capabilities of APR to regression bugs by assessing both traditional APR tools and advanced LLM-based APR approaches. Our experimental results show that classical APR tools fail to repair any bugs, while LLM-based APR approaches exhibit promising potential. Motivated by these results, we investigate impact of incorporating bug-inducing change information into LLM-based APR approaches for fixing regression bugs. We further conduct an ablation study to disaggregate the contribution of each contextual element within the bug-inducing change information. Our results highlight that this context-aware enhancement significantly improves the performance of LLM-based APR, yielding 1.6x more successful repairs compared to using LLM-based APR without such context. Moreover, our findings are consistent across both Java and Python benchmarks, providing preliminary evidence for the generalizability of our findings.

Figures

Figures reproduced from arXiv: 2506.13182 by the authors.

Figure 1
Figure 1. An issue reported requiring developers to resolve; however, after fixing it, another bug is inadvertently introduced. See the } else { return concatChunks( [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 3
Figure 3. An issue reported requiring developers to resolve. This issue is caused by the fix of a problem introduced two years prior (see + public static boolean isInvisibleChar(int c) { - + return CharactergetType(c) == 16 [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Regression-fixing changes made by developers to resolve the regression bug. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (18 more)
Figure 5
Figure 5. Figure 5: Overview of the RegMiner4APR benchmark construction. 3.2.1 Project Selection. In this study, we focus on Java regression bugs for two reasons. First, Java remains one of the most widely used programming languages today, as consistently ranked in the TIOBE Index 3 . Sec…
Figure 6
Figure 6. Figure 6: The input and output representation of Incoder and CodeGen. private static boolean isAssignableFrom(Class<?> target, KType source) { Type parameterType = ReflectJvmMapping.getJavaType(source); // buggy code // if (parameterType instanceof Class) { // return target.isAs…
Figure 8
Figure 8. Figure 8: Structure of our prompt-based APR query. The code snippet and error messages are omitted for clarity. The expected response [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Prompt templates designed for prompt-based APR with regression-inducing change information. [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Number of bugs filtered out during the regression validation process. [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Time distribution of the regression bugs in our benchmark [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Distribution of repair operators on different types of regression bug. Note that, we abbreviated the names of the repair action [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: Occurrence of repair operators in RegMiner4APR. Note that, we abbreviated the names of the repair action groups following [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]
Figure 14
Figure 14. Figure 14: Detailed examination of a repair operator, namely [PITH_FULL_IMAGE:figures/full_fig_p023_14.png]
Figure 15
Figure 15. Figure 15: Distribution of the number of repair operators per patch. [PITH_FULL_IMAGE:figures/full_fig_p023_15.png]
Figure 17
Figure 17. Figure 17: Excerpt from ChatGPT-4o’s response under the zero-shot prompting configuration augmented with bug-inducing change [PITH_FULL_IMAGE:figures/full_fig_p028_17.png]
Figure 19
Figure 19. Figure 19: Excerpt from ChatGPT-4o’s response under the conversation configuration augmented with bug-inducing change information Observation: Fully or Partially reverting [PITH_FULL_IMAGE:figures/full_fig_p030_19.png]
Figure 21
Figure 21. Figure 21: Excerpt from ChatGPT-4o’s response under the conversation configuration augmented with bug-inducing change information [PITH_FULL_IMAGE:figures/full_fig_p031_21.png]
Figure 22
Figure 22. Figure 22: Regression-inducing changes in RegressionBug-69 To illustrate this behavior, we present RegressionBug-69, where the developers attempted to improve the performance of JSONObject.toJavaObject by introducing optimizations 12, as shown in [PITH_FULL_IMAGE:figures/full_f…
Figure 23
Figure 23. Figure 23: Excerpts from ChatGPT-4o’s response under the conversational configuration for repairing [PITH_FULL_IMAGE:figures/full_fig_p033_23.png]
Figure 24
Figure 24. Figure 24: Patches generated by ChatGPT-4o for RegressionBug-69 under different configurations. Manuscript submitted to ACM [PITH_FULL_IMAGE:figures/full_fig_p033_24.png]
Figure 25
Figure 25. Figure 25: Excerpt from ChatGPT-4o’s response under zero-shot prompting configuration augmented with bug-inducing change [PITH_FULL_IMAGE:figures/full_fig_p034_25.png]
Figure 26
Figure 26. Figure 26: Token cost of patch generation per bug across different methods. [PITH_FULL_IMAGE:figures/full_fig_p036_26.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

73 extracted references · 59 canonical work pages

  1. [1]

    RegMiner4APR Benchmark

    2025. RegMiner4APR Benchmark. https://github.com/brojackvn/RegMiner4APR-Benchmark

    work page 2025

  2. [2]

    RegMiner4APR Framework

    2025. RegMiner4APR Framework. https://github.com/brojackvn/RegMiner4APR-Framework

    work page 2025

  3. [3]

    RegMiner4APR Homepage

    2025. RegMiner4APR Homepage. https://brojackvn.github.io/RegMiner4APR-Homepage

    work page 2025

  4. [4]

    RegRepair: Prompt-based Program Repair for Regression Bugs

    2025. RegRepair: Prompt-based Program Repair for Regression Bugs. https://github.com/brojackvn/Program-Repair-Framework

    work page 2025

  5. [5]

    Regression validation tool

    2025. Regression validation tool. https://github.com/brojackvn/RegMiner4APR-Framework/blob/main/regression-validation

    work page 2025

  6. [6]

    Abdulaziz Alhefdhi, Hoa Khanh Dam, Thanh Le-Cong, Bach Le, and Aditya Ghose. 2025. Adversarial patch generation for automated program repair. Software Quality Journal 33, 1 (Jan. 2025), 23 pages. doi:10.1007/s11219-025-09709-4

    work page doi:10.1007/s11219-025-09709-4 2025

  7. [7]

    Gabin An, Jingun Hong, Naryeong Kim, and Shin Yoo. 2023. Fonte: Finding bug inducing commits from failures. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 589–601

    work page 2023

  8. [8]

    Earl T Barr, Yuriy Brun, Premkumar Devanbu, Mark Harman, and Federica Sarro. 2014. The plastic surgery hypothesis. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering . 306–317

    work page 2014

Show all 73 references
  1. [9]

    Moritz Beller, Georgios Gousios, and Andy Zaidman. 2017. Travistorrent: Synthesizing travis ci and github for full-stack research on continuous integration. In 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR) . IEEE, 447–450

    2017

  2. [10]

    Marcel Böhme, Bruno C d S Oliveira, and Abhik Roychoudhury. 2013. Regression tests to expose change interaction errors. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering . 334–344

    2013

  3. [11]

    Marcel Böhme and Abhik Roychoudhury. 2014. Corebench: Studying complexity of regression errors. In Proceedings of the 2014 international symposium on software testing and analysis . 105–115

    2014

  4. [12]

    Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. 2024. Repairagent: An autonomous, llm-based agent for program repair. arXiv preprint arXiv:2403.17134 (2024)

    2024 arXiv

  5. [13]

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems 36 (2023), 10088–10115

    2023

  6. [14]

    Thomas Durieux and Rui Abreu. 2019. Critical review of bugswarm for fault localization and program repair. arXiv preprint arXiv:1905.09375 (2019)

    2019 arXiv

  7. [15]

    Emelie Engström and Per Runeson. 2010. A qualitative survey of regression testing practices. In Product-Focused Software Process Improvement: 11th International Conference, PROFES 2010, Limerick, Ireland, June 21-23, 2010. Proceedings 11 . Springer, 3–16

    2010

  8. [16]

    Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. 2022. Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204.05999 (2022)

    2022 arXiv

  9. [17]

    Xiang Gao, Yannic Noller, and Abhik Roychoudhury. 2022. Program repair. arXiv preprint arXiv:2211.12787 (2022)

    2022 arXiv

  10. [18]

    Péter Gyimesi, Béla Vancsics, Andrea Stocco, Davood Mazinanian, Arpád Beszédes, Rudolf Ferenc, and Ali Mesbah. 2019. Bugsjs: a benchmark of javascript bugs. In 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST) . IEEE, 90–101

    2019

  11. [19]

    Kai Huang, Jian Zhang, Xinlei Bao, Xu Wang, and Yang Liu. 2025. Comprehensive Fine-Tuning Large Language Models of Code for Automated Program Repair. IEEE Transactions on Software Engineering (2025)

    2025

  12. [20]

    Nan Jiang, Kevin Liu, Thibaud Lutellier, and Lin Tan. 2023. Impact of code language models on automated program repair. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 1430–1442

    2023

  13. [21]

    Nan Jiang, Thibaud Lutellier, Yiling Lou, Lin Tan, Dan Goldwasser, and Xiangyu Zhang. 2023. Knod: Domain knowledge distilled tree decoder for automated program repair. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 1251–1263

    2023

  14. [22]

    Nan Jiang, Thibaud Lutellier, and Lin Tan. 2021. Cure: Code-aware neural machine translation for automatic program repair. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) . IEEE, 1161–1173. Manuscript submitted to ACM From Empirical Evaluation to ...

    2021

  15. [23]

    René Just, Darioush Jalali, and Michael D Ernst. 2014. Defects4J: A database of existing faults to enable controlled testing studies for Java programs. In Proceedings of the 2014 international symposium on software testing and analysis . 437–440

    2014

  16. [24]

    Vinay Kabadi, Dezhen Kong, Siyu Xie, Lingfeng Bao, Gede Artha Azriadi Prana, Tien-Duy B Le, Xuan-Bach D Le, and David Lo. 2023. The Future Can’t Help Fix The Past: Assessing Program Repair In The Wild. In 2023 IEEE International Conference on Software Maintenance and Evolution...

    2023

  17. [25]

    Xuan-Bach D Le, Duc-Hiep Chu, David Lo, Claire Le Goues, and Willem Visser. 2017. S3: syntax-and semantic-guided repair synthesis via programming by examples. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering . 593–604

    2017

  18. [26]

    Le, David Lo, and Claire Le Goues

    Xuan Bach D. Le, David Lo, and Claire Le Goues. 2016. History Driven Program Repair. In 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER) , Vol. 1. 213–224. doi:10.1109/SANER.2016.76

    2016 doi

  19. [27]

    Xuan-Bach D Le, Ferdian Thung, David Lo, and Claire Le Goues. 2018. Overfitting in semantics-based automated program repair. In Proceedings of the 40th international conference on software engineering . 163–163

    2018

  20. [28]

    Thanh Le-Cong, Bach Le, and Toby Murray. 2024. Semantic-guided Search for Efficient Program Repair with Large Language Models. arXiv preprint arXiv:2410.16655 (2024)

    2024

  21. [29]

    Claire Le Goues, Neal Holtschulte, Edward K Smith, Yuriy Brun, Premkumar Devanbu, Stephanie Forrest, and Westley Weimer. 2015. The ManyBugs and IntroClass benchmarks for automated repair of C programs. IEEE Transactions on Software Engineering 41, 12 (2015), 1236–1256

    2015

  22. [30]

    Claire Le Goues, ThanhVu Nguyen, Stephanie Forrest, and Westley Weimer. 2011. Genprog: A generic method for automatic software repair. Ieee transactions on software engineering 38, 1 (2011), 54–72

    2011

  23. [31]

    Claire Le Goues, Michael Pradel, and Abhik Roychoudhury. 2019. Automated program repair. Commun. ACM 62, 12 (2019), 56–65

    2019

  24. [32]

    Fengjie Li, Jiajun Jiang, Jiajun Sun, and Hongyu Zhang. 2024. Hybrid automated program repair by combining large language models and program analysis. ACM Transactions on Software Engineering and Methodology (2024)

    2024

  25. [33]

    Kui Liu, Anil Koyuncu, Dongsun Kim, and Tegawendé F Bissyandé. 2019. TBar: Revisiting template-based automated program repair. In Proceedings of the 28th ACM SIGSOFT international symposium on software testing and analysis . 31–42

    2019

  26. [34]

    Kui Liu, Li Li, Anil Koyuncu, Dongsun Kim, Zhe Liu, Jacques Klein, and Tegawendé F Bissyandé. 2021. A critical review on the evaluation of automated program repair systems. Journal of Systems and Software 171 (2021), 110817

    2021

  27. [35]

    Yue Liu, Thanh Le-Cong, Ratnadira Widyasari, Chakkrit Tantithamthavorn, Li Li, Xuan-Bach D Le, and David Lo. 2024. Refining chatgpt-generated code: Characterizing and mitigating code quality issues. ACM Transactions on Software Engineering and Methodology 33, 5 (2024), 1–26

    2024

  28. [36]

    Bissyandé, Haoye Tian, and Xuan Bach D

    Wenqiang Luo, Jacky Keung, Boyang Yang, He Ye, Claire Le Goues, Tegawendé F. Bissyandé, Haoye Tian, and Xuan Bach D. Le. 2025. When Fine-Tuning LLMs Meets Data Privacy: An Empirical Study of Federated Learning in LLM-Based Program Repair. ACM Trans. Softw. Eng. Methodol. (May ...

    2025 doi

  29. [37]

    Fernanda Madeiral, Simon Urli, Marcelo Maia, and Martin Monperrus. 2019. Bears: An extensible java bug benchmark for automatic program repair studies. In 2019 IEEE 26th international conference on software analysis, evolution and reengineering (SANER) . IEEE, 468–478

    2019

  30. [38]

    Matias Martinez and Martin Monperrus. 2016. Astor: A program repair library for java. In Proceedings of the 25th international symposium on software testing and analysis . 441–444

    2016

  31. [39]

    Matias Martinez and Martin Monperrus. 2018. Ultra-large repair search space with automatically mined templates: The cardumen mode of astor. In Search-Based Software Engineering: 10th International Symposium, SSBSE 2018, Montpellier, France, September 8-9, 2018, Proceedings 10 ...

    2018

  32. [40]

    Sergey Mechtaev, Jooyong Yi, and Abhik Roychoudhury. 2016. Angelix: Scalable multiline program patch synthesis via symbolic analysis. In Proceedings of the 38th international conference on software engineering . 691–701

    2016

  33. [41]

    Martin Monperrus. 2018. The living review on automated program repair . Ph. D. Dissertation. HAL Archives Ouvertes

    2018

  34. [42]

    Hoang Duong Thien Nguyen, Dawei Qi, Abhik Roychoudhury, and Satish Chandra. 2013. Semfix: Program repair via semantic analysis. In 2013 35th International Conference on Software Engineering (ICSE) . IEEE, 772–781

    2013

  35. [43]

    Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474 (2022)

    2022 arXiv

  36. [44]

    Dor Nir, Shmuel Tyszberowicz, and Amiram Yehudai. 2008. Locating regression bugs. In Hardware and Software: Verification and Testing: Third International Haifa Verification Conference, HVC 2007, Haifa, Israel, October 23-25, 2007. Proceedings 3 . Springer, 218–234

    2008

  37. [45]

    Akira K Onoma, Wei-Tek Tsai, Mustafa Poonawala, and Hiroshi Suganuma. 1998. Regression testing in an industrial environment. Commun. ACM 41, 5 (1998), 81–86

    1998

  38. [46]

    Kai Pan, Sunghun Kim, and E James Whitehead. 2009. Toward an understanding of bug fix patterns. Empirical Software Engineering 14 (2009), 286–315

    2009

  39. [47]

    Zichao Qi, Fan Long, Sara Achour, and Martin Rinard. 2015. An analysis of patch plausibility and correctness for generate-and-validate patch generation systems. In Proceedings of the 2015 International Symposium on Software Testing and Analysis . 24–36

    2015

  40. [48]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al

  41. [49]

    Haifeng Ruan, Yuntong Zhang, and Abhik Roychoudhury. 2024. Specrover: Code intent extraction via llms. arXiv preprint arXiv:2408.02232 (2024)

    2024 arXiv

  42. [50]

    Ripon K Saha, Yingjun Lyu, Wing Lam, Hiroaki Yoshida, and Mukul R Prasad. 2018. Bugs. jar: A large-scale, diverse dataset of real-world java bugs. In Proceedings of the 15th international conference on mining software repositories . 10–13. Manuscript submitted to ACM 40 Ho et al

    2018

  43. [51]

    Ridwan Shariffdeen, Martin Mirchev, Yannic Noller, and Abhik Roychoudhury. 2023. Cerberus: a Program Repair Framework. In2023 IEEE/ACM 45th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) . IEEE, 73–77

    2023

  44. [52]

    André Silva, Sen Fang, and Martin Monperrus. 2023. Repairllama: Efficient representations and fine-tuned adapters for program repair. arXiv preprint arXiv:2312.15698 (2023)

    2023 arXiv

  45. [53]

    Edward K Smith, Earl T Barr, Claire Le Goues, and Yuriy Brun. 2015. Is the cure worse than the disease? overfitting in automated program repair. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering . 532–543

    2015

  46. [54]

    Victor Sobreira, Thomas Durieux, Fernanda Madeiral, Martin Monperrus, and Marcelo de Almeida Maia. 2018. Dissection of a bug dataset: Anatomy of 395 patches from defects4j. In 2018 IEEE 25th international conference on software analysis, evolution and reengineering (SANER) . I...

    2018

  47. [55]

    Xuezhi Song, Yun Lin, Siang Hwee Ng, Yijian Wu, Xin Peng, Jin Song Dong, and Hong Mei. 2022. Regminer: towards constructing a large regression dataset from code evolution history. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis . 314–326

    2022

  48. [56]

    Shin Hwei Tan and Abhik Roychoudhury. 2015. relifix: Automated repair of software regressions. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering , Vol. 1. IEEE, 471–482

    2015

  49. [57]

    Shin Hwei Tan, Jooyong Yi, Sergey Mechtaev, Abhik Roychoudhury, et al. 2017. Codeflaws: a programming competition benchmark for evaluating automated program repair tools. In 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C) . IEEE, 180–182

    2017

  50. [58]

    David A Tomassi, Naji Dmeiri, Yichen Wang, Antara Bhowmick, Yen-Chuan Liu, Premkumar T Devanbu, Bogdan Vasilescu, and Cindy Rubio- González. 2019. Bugswarm: Mining and continuously growing a dataset of reproducible failures and fixes. In 2019 IEEE/ACM 41st International Confer...

    2019

  51. [59]

    Ratnadira Widyasari, Sheng Qin Sim, Camellia Lok, Haodi Qi, Jack Phan, Qijin Tay, Constance Tan, Fiona Wee, Jodie Ethelda Tan, Yuheng Yieh, et al

  52. [60]

    Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2023. Automated program repair in the era of large pre-trained language models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 1482–1494

    2023

  53. [61]

    Chunqiu Steven Xia and Lingming Zhang. 2024. Automated program repair via conversation: Fixing 162 out of 337 bugs for $0.42 each using ChatGPT. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis . 819–831

    2024

  54. [62]

    Jiahong Xiang, Xiaoyang Xu, Fanchu Kong, Mingyuan Wu, Zizheng Zhang, Haotian Zhang, and Yuqun Zhang. 2024. How far can we go with practical function-level program repair? arXiv preprint arXiv:2404.12833 (2024)

    2024 arXiv

  55. [63]

    Junjielong Xu, Ying Fu, Shin Hwei Tan, and Pinjia He. 2024. Aligning the Objective of LLM-based Program Repair. arXiv preprint arXiv:2404.08877 (2024)

    2024 arXiv

  56. [64]

    Jifeng Xuan, Matias Martinez, Favio Demarco, Maxime Clement, Sebastian Lamelas Marcote, Thomas Durieux, Daniel Le Berre, and Martin Monperrus. 2016. Nopol: Automatic repair of conditional statement bugs in java programs. IEEE Transactions on Software Engineering 43, 1 (2016), 34–55

    2016

  57. [65]

    Boyang Yang, Haoye Tian, Jiadong Ren, Shunfu Jin, Yang Liu, Feng Liu, and Bach Le. 2025. Enhancing Repository-Level Software Repair via Repository-Aware Knowledge Graphs. arXiv:2503.21710 [cs.SE] https://arxiv.org/abs/2503.21710

    2025

  58. [66]

    He Ye, Matias Martinez, and Martin Monperrus. 2022. Neural program repair with execution-based backpropagation. In Proceedings of the 44th international conference on software engineering . 1506–1518

    2022

  59. [67]

    Xin Yin, Chao Ni, Shaohua Wang, Zhenhao Li, Limin Zeng, and Xiaohu Yang. 2024. Thinkrepair: Self-directed automated program repair. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis . 1274–1286

    2024

  60. [68]

    Zuoning Yin, Ding Yuan, Yuanyuan Zhou, Shankar Pasupathy, and Lakshmi Bairavasundaram. 2011. How do fixes become bugs?. In Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering . 26–36

    2011

  61. [69]

    Yuan Yuan and Wolfgang Banzhaf. 2018. Arja: Automated repair of java programs via multi-objective genetic programming. IEEE Transactions on software engineering 46, 10 (2018), 1040–1067

    2018

  62. [70]

    Andreas Zeller. 1999. Yesterday, my program worked. Today, it does not. Why? ACM SIGSOFT Software engineering notes 24, 6 (1999), 253–267

    1999

  63. [71]

    Qihao Zhu, Zeyu Sun, Yuan-an Xiao, Wenjie Zhang, Kang Yuan, Yingfei Xiong, and Lu Zhang. 2021. A syntax-guided edit decoder for neural program repair. In Proceedings of the 29th ACM joint meeting on European software engineering conference and symposium on the foundations of s...

    2021

  64. [2020]

    In Proceedings of the 28th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering

    Bugsinpy: a database of existing bugs in python programs to enable controlled testing and debugging studies. In Proceedings of the 28th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering . 1556–1560

  65. [2023]

    arXiv preprint arXiv:2308.12950 (2023)

    Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023)

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.