Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

The paper claims that repository-level program repair improves when historical fixes are reused as backward-distilled, outcome-conditioned reasoning plans, turning verified patches into in-context guidance that steers fault localization and

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 06:08 UTC pith:STPRX7EZ

load-bearing objection A sensible inference-time repair method with consistent gains, but the exemplar pool likely leaks other SWE-Bench gold patches, so the headline numbers are not yet trustworthy. the 4 major comments →

arxiv 2601.23257 v2 pith:STPRX7EZ submitted 2026-01-30 cs.SE

From Historical Patches to Repair Plans: Outcome-Conditioned Reasoning for Repository-Level Program Repair

Chenglin Li , Yisen Xu , Zehao Wang , Shin Hwei Tan , Tse-Hsun (Peter) Chen This is my paper
classification cs.SE
keywords automated program repairrepository-level repairoutcome-conditioned reasoningbackward reasoning distillationin-context learningfault localizationSWE-Benchtest-time scaling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that repository-level program repair can be made more effective and cheaper by reusing historical fixes as backward-distilled, outcome-conditioned reasoning plans, rather than re-solving each issue from scratch or exploring forward with search. The proposed O-CRD retrieves a same-repository resolved issue, filters it through a conservative 'Exemplar Guardian', and distills three stage-wise plans (file localization, function localization, patch synthesis) that are conditioned on the verified ground-truth patch and injected as in-context prompts at inference time. On SWE-Bench Lite, O-CRD raises Pass@1 by 10.4 points with GPT-4o, 8.6 with DeepSeek-V3, and 10.3 with GPT-5 over the Agentless scaffold, with statistical significance, and beats a controlled MCTS forward-search baseline by 13.0 points while using 9.71x fewer LLM calls. A sympathetic reader would care because it suggests that backward conditioning on verified outcomes is a scalable inference-time alternative to expensive search-based test-time scaling for long-horizon code repair.

Core claim

The central claim is that conditioning repair reasoning on a known-correct terminal state—the verified patch—produces reusable, causally consistent plans that improve downstream localization and patch generation, whereas forward exploration that reasons under outcome uncertainty drifts and wastes compute. O-CRD operationalizes this through Backward Reasoning Distillation: for each sub-task, it generates a step-wise plan under an autoregressive conditioning objective P(t_i | t_<i, x, y_k, guidance), where y_k is the ground-truth outcome; then it refines each step by re-conditioning on the full outcome set and selects the best variant. The finalized plans are injected into the corresponding st

What carries the argument

The central object is the outcome-conditioned repair plan S* = {S*_file, S*_func, S*_patch}, produced by Backward Reasoning Distillation. The mechanism is the conditional-next-token objective P(t_i | t_<i, x, y_k, guidance), which narrows reasoning to trajectories compatible with a verified fix; each step is generated and then refined by ranking against rewrite candidates under constraints of local context, global outcome consistency, and specificity. The other load-bearing component is the Exemplar Guardian, a conservative LLM-based filter with a five-dimension rubric (root-cause similarity, causal-chain transferability, fix-strategy applicability, contextual alignment, debugging-technique

Load-bearing premise

The load-bearing premise is that the repository-specific exemplar pool, built from all resolved historical issues, contains no benchmark test instances whose ground-truth patches leak into prompts, and that the LLM-based Exemplar Guardian reliably identifies transferable exemplars; if leakage exists or the Guardian passes misleading matches, the reported gains are inflated.

What would settle it

Inspect the constructed exemplar dataset for the 13 SWE-Bench Lite repositories and check whether any test issue's ground-truth patch appears as a historical exemplar. Or, in a held-out sample, remove the Exemplar Guardian and compare Pass@1: if the drop is not statistically significant, or if a matched-budget MCTS with the same number of LLM calls achieves comparable Pass@1 to O-CRD, the central claim of backward-outcome-conditioning superiority would be contradicted.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Repository-level repair can be improved at inference time by reusing same-repository historical fixes as in-context guidance, without fine-tuning or search.
  • Fault localization accuracy improves alongside Pass@1 (e.g., 59.3% vs 51.0% function-level accuracy for GPT-4o), suggesting the guidance steers earlier decisions, not just patch text.
  • The gains hold across three LLMs (GPT-4o, GPT-5, DeepSeek-V3), so the method is not tied to a single model's prompting behavior.
  • Backward-distilled plans require far fewer LLM calls and tokens than a controlled MCTS forward-search baseline (9.71x fewer calls, 2.47x fewer tokens) while achieving higher Pass@1.
  • The framework can be adapted to other agentic scaffolds, such as SWE-agent, by distilling plans in Thought/Action/Observation format.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If backward conditioning is the real driver, the same recipe could be applied to other long-horizon code tasks—feature additions, refactors, test generation—where a verified terminal state exists.
  • The Exemplar Guardian's LLM-based rubric is a cost center; a frozen embedding or small classifier might filter exemplars more cheaply, and a testable variant is to replace it and re-measure Pass@1.
  • The same-repository restriction limits applicability to projects with rich fix histories; a natural extension is cross-repository exemplar retrieval, which the paper's own limitation section identifies as a weak spot for sparse-history repositories.
  • The MCTS comparison uses a fixed budget; a matched-budget MCTS (equal LLM calls/tokens) could test whether forward search closes the gap, a comparison the paper leaves for future work.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces O-CRD (also called ConRAD in the abstract and Figure 2), an inference-time method for repository-level automated program repair. For each repository, the method builds a historical exemplar pool from resolved issues with verified patches, retrieves a semantically compatible exemplar for a target issue, filters it with an LLM-based Exemplar Guardian, and then backward-distills a stage-wise repair plan conditioned on the exemplar's ground-truth patch and issue description. The resulting file-, function-, and patch-level plans are injected into an Agentless-style pipeline to guide localization and patch generation. On SWE-Bench Lite, the authors report Pass@1 gains over Agentless of +10.4% (GPT-4o), +8.6% (DeepSeek-V3), and +10.3% (GPT-5), together with ablations of the Guardian and a fixed-budget MCTS forward-search baseline.

Significance. If the main empirical claim holds after a clean-pool re-evaluation, the contribution is significant: it demonstrates a cheap, training-free alternative to search-based reasoning for APR, with consistent gains across three backbones and no fine-tuning. The backward-distillation idea is distinct from prior retrieval/memory/CoT methods and the ablation design is thoughtful. The paper does not ship code or data, so reproducibility currently rests on the textual description. The central risk is benchmark overlap in the exemplar pool; this concern lands on the manuscript as written and must be resolved before the reported gains can be interpreted as valid.

major comments (4)
  1. [Appendix A.1; Section 3.3; Figure 1] The exemplar pool as described appears to include other SWE-Bench evaluation instances. Appendix A.1 states that the pool is constructed by collecting 'all historical issues that have been successfully resolved and are associated with verified pull requests' from the same 13 SWE-Bench Lite repositories. SWE-Bench Lite/Verified instances are themselves resolved issues with verified PRs in those repositories. The only stated exclusion is temporal (exemplar resolved strictly before the target's creation time), which does not remove other benchmark instances that were resolved earlier. For such an exemplar, Stage 3 conditions the distilled plan S* on the exemplar's ground-truth y_patch, so the injected prompt contains reasoning derived from the gold patch of another held-out evaluation instance. The Exemplar Guardian's rubric (Stage 2) has no benchmark-membership criterion, so it cannot reje
  2. [Sections 4.3, 4.4; Table 1] The GPT-4o baseline numbers are taken from prior papers rather than rerun in the same environment. O-CRD and the rerun Agentless variants may use different prompt templates, API versions, or sampling settings, so the 'outperforms the strongest competitor by 6.7%' claim is not a fully controlled comparison. In addition, no confidence intervals or run-to-run variance are reported for the main Pass@1 values. The McNemar tests are useful, but they only compare counts within one run and do not address systematic setup differences. Please rerun at least Agentless and one strong baseline under the same environment, report the exact model/settings, and provide CIs or per-seed variance.
  3. [Section 5.2; Table 4; Appendix B] The conclusion that outcome-conditioned distillation can 'replace forward exploration' is supported only by a fixed-budget MCTS variant (B=3, R=20, depth 4) evaluated on 100 samples. This is not a state-of-the-art search method such as SWE-Search, and Appendix B correctly concedes that the comparison is not matched-budget. The 9.71x call and 2.47x token ratios are descriptive of this particular configuration, not a general cost-effectiveness result. Please calibrate the budget to a stronger MCTS baseline or soften the claim to 'outperforms a fixed-budget forward-search variant'.
  4. [Table 2; Section 5.1] The Guardian accepts only 195/300 exemplars with GPT-4o, 286/300 with GPT-5, and 257/300 with DeepSeek-V3. For the rejected targets, the pipeline presumably falls back to the plain Agentless scaffold. The aggregate gains in Table 1 could therefore be driven mostly by the accepted subset, which may be easier on average. Please report Pass@1 and localization accuracy separately for accepted vs rejected targets, and compare these subgroups against the Agentless baseline, so that the improvement can be attributed to the injected plans rather than to selection of easier cases.
minor comments (6)
  1. [Throughout] The method name is inconsistent: the abstract and Figure 2 use 'ConRAD', while the body uses 'O-CRD' and Appendix A.2 refers to 'Reverse-Engineered Reasoning Distillation' as well as 'Backward Reasoning Distillation' (BRD). Please unify the terminology.
  2. [Figure 1] The figure contains an internal Chinese-language annotation: 'Historical dataset怎么来的？ -- issue fix dataset不需要分数s，迷惑 Stage 3 突出创新点？'. This appears to be an unresolved author note and must be removed. It is also directly relevant to the data-provenance concern raised above.
  3. [Section 3.3] The conditioning equation is unnumbered and the notation Pθ(ti | t<i) → Pθ(ti | t<i, x, yk, guidance) is used without a precise definition of yk or the generation procedure. Please number the equation and define all symbols.
  4. [Section 5.2; Appendix B] Table 4 and Table 5 report results on a 100-sample subset, but the text does not state whether the same 100 random issues are used in the MCTS and exemplar-similarity ablations. Please clarify the sample overlap and report the sample IDs or a seed.
  5. [Section 5.1] For the McNemar tests, the paper reports only p < 0.001. Please report the exact p-values and the test statistic, and note whether any multiple-testing correction was applied.
  6. [Appendix A.1] The paper does not include a reproducibility statement, code release, or dataset release. Given that the exemplar pool is central to the method, please release the pool construction script, the retrieved exemplar IDs, and the Guardian decisions, or explain why this is not possible.

Circularity Check

0 steps flagged

No circularity: target Pass@1 is evaluated externally; exemplar plans are derived from different historical issues.

full rationale

The derivation chain is self-contained with respect to the benchmark target. Stage 1 retrieves an exemplar issue from the same repository using embedding similarity and an LLM judge; Stage 2 filters it by transferability; Stage 3 constructs the repair plan by backward conditioning on the exemplar's own verified patch (x, y_k). The Pass@1 evaluation is computed on the target issue's held-out tests, and the target gold patch is never used to construct the injected plan. Thus the claimed improvement is not forced by construction: the plan is an input-derived artifact for a different issue, and the target outcome is external. The only near concern is data contamination: Appendix A.1 says the pool is built from 'all historical issues that have been successfully resolved and are associated with verified pull requests' following the SWE-Bench protocol, and does not state exclusion of other SWE-Bench test instances, so another benchmark instance's gold patch could enter the exemplar pool and be distilled. This is a leakage/validity threat, not a circularity step under the definition used here: it does not make the target prediction equal to an input by construction. One minor self-citation ([36], by co-author Shin Hwei Tan) supports the Guardian's dimension checklist but is not load-bearing for the central result. No circular step found.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 1 invented entities

No parameters are fitted to the benchmark; all thresholds and budgets are hand-chosen. The central claim rests on several domain assumptions: verified patches are trustworthy, same-repository history transfers, the Guardian filters well, backward generation produces usable plans, and temporal ordering avoids leakage. No new physical entity is introduced; the only invented component is the Guardian pipeline element.

free parameters (4)
  • Top-5 retrieval candidates = 5
    Number of textually similar historical issues retained before LLM-as-judge selection; hand-chosen and affects recall/noise trade-off (Section 3.1).
  • Maximum modified files per exemplar = at most 3 files
    Filter applied to historical patches when building the exemplar dataset; large patches excluded to keep plans coherent; affects pool composition and results (Appendix A.1).
  • Refinement candidates per step = 3 rewrites
    Step-wise refinement generates three candidate rewrites and selects the best; hand-chosen and affects plan quality and cost (Section 3.3, Step 2).
  • MCTS baseline budget = branching 3, 20 rollouts, depth 4
    Configuration for the controlled forward-search ablation; not claimed to represent state-of-the-art MCTS and directly affects the reported 13.0-point comparison (Appendix B).
axioms (6)
  • domain assumption Verified historical patches are correct repair outcomes
    Used as ground-truth y_k for distillation; patches are only test-verified, not formally proven correct (Section 3, Appendix A.1).
  • domain assumption Same-repository historical issues provide transferable repair guidance
    Core premise of the retrieval stage; assumed to preserve conventions and coding patterns (Section 3.1).
  • domain assumption LLM-as-a-judge Guardian can reliably detect transferable vs misleading exemplars
    Stage 2 relies on a five-criteria rubric and one-step self-reflection; only run-to-run stability is reported, not human validation (Section 3.2, Table 3).
  • domain assumption Backward generation conditioned on a verified outcome yields causally consistent, transferable plans
    Section 3.3 posits that conditioning on the gold patch narrows reasoning to globally consistent steps; support is indirect, via downstream Pass@1.
  • domain assumption Temporal ordering prevents leakage
    Only exemplars resolved strictly before the target issue's creation time are eligible; assumed to eliminate leakage, but the pool may still contain other SWE-Bench test instances (Appendix A.1).
  • domain assumption SWE-Bench Lite is representative of repository-level repair
    Used as the sole evaluation benchmark; limitations acknowledge Python-only coverage (Sections 4.1, 6).
invented entities (1)
  • Exemplar Guardian no independent evidence
    purpose: LLM-based compatibility filter that labels retrieved historical issues as Transferable, Non-transferable, or Misleading before reasoning distillation.
    A new software component without external validation; its reliability is only assessed by run-to-run agreement in Table 3, not against human labels or an independent oracle.

pith-pipeline@v1.3.0-alltime-deepseek · 14309 in / 14739 out tokens · 173651 ms · 2026-08-03T06:08:00.397739+00:00 · methodology

0 comments
read the original abstract

Repository-level automated program repair (APR) requires long-horizon reasoning over interdependent decisions. However, most LLM-based approaches reconstruct repair reasoning independently for each issue, failing to reuse successful patterns from prior repairs, even though real-world repositories contain many related issues with shared structure or constraints. Existing methods typically rely on forward exploration, which operates under outcome uncertainty, incurs substantial inference-time overhead, and can drift from the final correct patch. We propose Conditional Reasoning Distillation (ConRAD), which leverages in-repository resolved issues by reconstructing repair reasoning backward from verified patches and distilling outcome-consistent, stage-wise repair reasoning plans. Injected at inference time, these plans guide fault localization and patch generation, replacing open-ended exploration with constrained inference without fine-tuning or search. On SWE-Bench Lite, ConRAD improves Pass@1 by 10.4\% (GPT-4o), 8.6\% (DeepSeek-V3), and 10.3\% (GPT-5), demonstrating a scalable inference-time alternative to forward exploration for long-horizon APR.

Figures

Figures reproduced from arXiv: 2601.23257 by Chenglin Li, Shin Hwei Tan, Tse-Hsun (Peter) Chen, Yisen Xu, Zehao Wang.

Figure 1
Figure 1. Figure 1: O-CRD overview: Stage 1 retrieves in-repository exemplars and selects one via an LLM￾based judge. Stage 2 filters candidates into Transferable, Non-transferable, Misleading. Stage 3 distills outcome-conditioned plans S ∗ f ile, S ∗ func, S ∗ patch from the exemplar’s ground-truth fix and injects them as in-context guidance during inference. repair agent [20, 22], often incorporating search-style refinement… view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of Pass@1 results on tasks grouped by different difficulty levels. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: SWE-agent official trajectory format [35]. [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Bug Report Specification Refinement with Trajectory Guidance for Automated Program Repair

    cs.SE 2026-07 conditional novelty 6.0

    Refining bug reports with hierarchical evidence from an unverified agent trajectory plus repository review raises Pass@1 on SWE-Bench Lite from 41% to 59.67% (GPT-5-mini) and similarly for other agents.

Reference graph

Works this paper leans on

57 extracted references · 23 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Otter: Generating tests from issues to validate SWE patches

    Toufique Ahmed, Jatin Ganhotra, Rangeet Pan, Avraham Shinnar, Saurabh Sinha, and Mar- tin Hirzel. Otter: Generating tests from issues to validate SWE patches. InForty-second International Conference on Machine Learning, 2025

    2025

  2. [2]

    Swe-search: Enhancing software agents with monte carlo tree search and iterative refinement

    Antonis Antoniades, Albert Örwall, Kexun Zhang, Yuxi Xie, Anirudh Goyal, and William Wang. Swe-search: Enhancing software agents with monte carlo tree search and iterative refinement. arXiv preprint arXiv:2410.20285, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    Masai: Modular architecture for software- engineering ai agents.arXiv preprint arXiv:2406.11638, 2024

    Daman Arora, Atharv Sonwane, Nalin Wadhwa, Abhav Mehrotra, Saiteja Utpala, Ramakrishna Bairi, Aditya Kanade, and Nagarajan Natarajan. Masai: Modular architecture for software- engineering ai agents.arXiv preprint arXiv:2406.11638, 2024

    Pith/arXiv arXiv 2024

  4. [4]

    Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    Mark Chen. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    Pith/arXiv arXiv 2021

  5. [5]

    Incoder: A generative model for code infilling and synthesis.arXiv preprint arXiv:2204.05999, 2022

    Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis.arXiv preprint arXiv:2204.05999, 2022

    Pith/arXiv arXiv 2022

  6. [6]

    Trae agent: An llm-based agent for software engineering with test-time scaling.arXiv preprint arXiv:2507.23370, 2025

    Pengfei Gao, Zhao Tian, Xiangxin Meng, Xinchen Wang, Ruida Hu, Yuanan Xiao, Yizhou Liu, Zhao Zhang, Junjie Chen, Cuiyun Gao, et al. Trae agent: An llm-based agent for software engineering with test-time scaling.arXiv preprint arXiv:2507.23370, 2025. 10

    Pith/arXiv arXiv 2025

  7. [7]

    Causal reasoning in software quality assurance: A systematic review.Information and Software Technology, 178:107599, 2025

    Luca Giamattei, Antonio Guerriero, Roberto Pietrantuono, and Stefano Russo. Causal reasoning in software quality assurance: A systematic review.Information and Software Technology, 178:107599, 2025

    2025

  8. [8]

    Zhang, Yang Liu, and Yun Ma

    Yaoqi Guo, Zhenpeng Chen, Jie M. Zhang, Yang Liu, and Yun Ma. Personality-guided code generation using large language models. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors,Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1068–1080, Vienna, Austr...

    2025

  9. [9]

    Reason- ing with language model is planning with world model

    Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong, Zhen Wang, Daisy Wang, and Zhiting Hu. Reason- ing with language model is planning with world model. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8154–8173, 2023

    2023

  10. [10]

    Aprmcts: Improving llm-based automated program repair with iterative tree search.arXiv preprint arXiv:2507.01827, 2025

    Haichuan Hu, Congqing He, Hao Zhang, Xiaochen Xie, and Quanjun Zhang. Aprmcts: Improving llm-based automated program repair with iterative tree search.arXiv preprint arXiv:2507.01827, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Large language models cannot self-correct reasoning yet.arXiv preprint arXiv:2310.01798, 2023

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet.arXiv preprint arXiv:2310.01798, 2023

    Pith/arXiv arXiv 2023

  12. [12]

    Shaping program repair space with existing patches and similar code

    Jiajun Jiang, Yingfei Xiong, Hongyu Zhang, Qing Gao, and Xiangqun Chen. Shaping program repair space with existing patches and similar code. InProceedings of the 27th ACM SIGSOFT international symposium on software testing and analysis, pages 298–309, 2018

    2018

  13. [13]

    Swe-bench: Can language models resolve real-world github issues?arXiv preprint arXiv:2310.06770, 2023

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?arXiv preprint arXiv:2310.06770, 2023

    Pith/arXiv arXiv 2023

  14. [14]

    PatchPilot: A cost-efficient software engineering agent with early attempts on formal verification

    Hongwei Li, Yuheng Tang, Shiqi Wang, and Wenbo Guo. PatchPilot: A cost-efficient software engineering agent with early attempts on formal verification. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors,Proceedings of the 42nd International Conference on Machine Learning, ...

    2025

  15. [15]

    Structured chain-of-thought prompting for code generation.ACM Transactions on Software Engineering and Methodology, 34(2):1–23, 2025

    Jia Li, Ge Li, Yongmin Li, and Zhi Jin. Structured chain-of-thought prompting for code generation.ACM Transactions on Software Engineering and Methodology, 34(2):1–23, 2025

    2025

  16. [16]

    Contrastive decoding: Open-ended text generation as optimization

    Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors,Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon...

    2023

  17. [17]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    Reliable fix patterns inferred from static checkers for automated program repair.ACM Transactions on Software Engineering and Methodology, 32(4):1–38, 2023

    Kui Liu, Jingtang Zhang, Li Li, Anil Koyuncu, Dongsun Kim, Chunpeng Ge, Zhe Liu, Jacques Klein, and Tegawendé F Bissyandé. Reliable fix patterns inferred from static checkers for automated program repair.ACM Transactions on Software Engineering and Methodology, 32(4):1–38, 2023

    2023

  19. [19]

    Relrepair: Enhancing automated program repair by retrieving relevant code.arXiv preprint arXiv:2509.16701, 2025

    Shunyu Liu, Guangdong Bai, Mark Utting, and Guowei Yang. Relrepair: Enhancing automated program repair by retrieving relevant code.arXiv preprint arXiv:2509.16701, 2025

    arXiv 2025

  20. [20]

    Lingma swe-gpt: An open development-process-centric language model for automated software improvement.arXiv preprint arXiv:2411.00622, 2024

    Yingwei Ma, Rongyu Cao, Yongchang Cao, Yue Zhang, Jue Chen, Yibo Liu, Yuchen Liu, Binhua Li, Fei Huang, and Yongbin Li. Lingma swe-gpt: An open development-process-centric language model for automated software improvement.arXiv preprint arXiv:2411.00622, 2024. 11

    Pith/arXiv arXiv 2024

  21. [21]

    Thinking longer, not larger: Enhancing software engineering agents via scaling test-time compute.arXiv preprint arXiv:2503.23803, 2025

    Yingwei Ma, Yongbin Li, Yihong Dong, Xue Jiang, Rongyu Cao, Jue Chen, Fei Huang, and Binhua Li. Thinking longer, not larger: Enhancing software engineering agents via scaling test-time compute.arXiv preprint arXiv:2503.23803, 2025

    Pith/arXiv arXiv 2025

  22. [22]

    Alibaba lingmaagent: Improving automated issue resolution via comprehensive repository exploration

    Yingwei Ma, Qingping Yang, Rongyu Cao, Binhua Li, Fei Huang, and Yongbin Li. Alibaba lingmaagent: Improving automated issue resolution via comprehensive repository exploration. InProceedings of the 33rd ACM International Conference on the Foundations of Software Engineering, pages 238–249, 2025

    2025

  23. [23]

    Sorft: Issue resolving with subtask-oriented reinforced fine-tuning.arXiv preprint arXiv:2502.20127, 2025

    Zexiong Ma, Chao Peng, Pengfei Gao, Xiangxin Meng, Yanzhen Zou, and Bing Xie. Sorft: Issue resolving with subtask-oriented reinforced fine-tuning.arXiv preprint arXiv:2502.20127, 2025

    Pith/arXiv arXiv 2025

  24. [24]

    Note on the sampling error of the difference between correlated proportions or percentages.Psychometrika, 12(2):153–157, 1947

    Quinn McNemar. Note on the sampling error of the difference between correlated proportions or percentages.Psychometrika, 12(2):153–157, 1947

    1947

  25. [25]

    Moatless tools

    Moatless Tool Team. Moatless tools. https://github.com/aorwall/ moatless-tools, 2024. Accessed: 2025-11-18

    2024

  26. [26]

    Experepair: Dual- memory enhanced llm-based repository-level program repair.arXiv preprint arXiv:2506.10484, 2025

    Fangwen Mu, Junjie Wang, Lin Shi, Song Wang, Shoubin Li, and Qing Wang. Experepair: Dual- memory enhanced llm-based repository-level program repair.arXiv preprint arXiv:2506.10484, 2025

    Pith/arXiv arXiv 2025

  27. [27]

    In-context example selection with influences.arXiv preprint arXiv:2302.11042, 2023

    Tai Nguyen and Eric Wong. In-context example selection with influences.arXiv preprint arXiv:2302.11042, 2023

    Pith/arXiv arXiv 2023

  28. [28]

    Hello gpt-4o

    OpenAI. Hello gpt-4o. https://openai.com/index/hello-gpt-4o/, 2024. Ac- cessed: 2025-12-12

    2024

  29. [29]

    Introducing swe-bench verified

    OpenAI. Introducing swe-bench verified. https://openai.com/index/ introducing-swe-bench-verified/, 2024. Accessed: 2025-12-18

    2024

  30. [30]

    Introducing gpt-5

    OpenAI. Introducing gpt-5. https://openai.com/index/introducing-gpt-5/,

  31. [31]

    Training software engineering agents and verifiers with swe-gym.arXiv preprint arXiv:2412.21139, 2024

    Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with swe-gym.arXiv preprint arXiv:2412.21139, 2024

    Pith/arXiv arXiv 2024

  32. [32]

    An analysis of patch plausibility and correctness for generate-and-validate patch generation systems

    Zichao Qi, Fan Long, Sara Achour, and Martin Rinard. An analysis of patch plausibility and correctness for generate-and-validate patch generation systems. InProceedings of the 2015 international symposium on software testing and analysis, pages 24–36, 2015

    2015

  33. [33]

    Sentence-bert: Sentence embeddings using siamese bert- networks.arXiv preprint arXiv:1908.10084, 2019

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks.arXiv preprint arXiv:1908.10084, 2019

    Pith/arXiv arXiv 1908

  34. [34]

    Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36:8634–8652, 2023

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36:8634–8652, 2023

    2023

  35. [35]

    Swe-agent output files

    SWE-agent Team. Swe-agent output files. https://swe-agent.com/latest/usage/ trajectories/, 2025. Accessed: 2025-12-18

    2025

  36. [36]

    Crossfix: Resolution of github issues via similar bugs recommendation.Journal of Software: Evolution and Process, 36(4):e2554, 2024

    Shin Hwei Tan, Ziqiang Li, and Lu Yan. Crossfix: Resolution of github issues via similar bugs recommendation.Journal of Software: Evolution and Process, 36(4):e2554, 2024

    2024

  37. [37]

    Intervenor: Prompt the coding ability of large language models with the interactive chain of repairing.CoRR, 2023

    Hanbin Wang, Zhenghao Liu, Shuo Wang, Ganqu Cui, Ning Ding, Zhiyuan Liu, and Ge Yu. Intervenor: Prompt the coding ability of large language models with the interactive chain of repairing.CoRR, 2023

    2023

  38. [38]

    Reverse-engineered reasoning for open-ended generation.arXiv preprint arXiv:2509.06160, 2025

    Haozhe Wang, Haoran Que, Qixin Xu, Minghao Liu, Wangchunshu Zhou, Jiazhan Feng, Wanjun Zhong, Wei Ye, Tong Yang, Wenhao Huang, et al. Reverse-engineered reasoning for open-ended generation.arXiv preprint arXiv:2509.06160, 2025. 12

    Pith/arXiv arXiv 2025

  39. [39]

    Swe- mirror: Scaling issue-resolving datasets by mirroring issues across repositories.arXiv preprint arXiv:2509.08724, 2025

    Junhao Wang, Daoguang Zan, Shulin Xin, Siyao Liu, Yurong Wu, and Kai Shen. Swe- mirror: Scaling issue-resolving datasets by mirroring issues across repositories.arXiv preprint arXiv:2509.08724, 2025

    Pith/arXiv arXiv 2025

  40. [40]

    Openhands: An open platform for ai software developers as generalist agents.arXiv preprint arXiv:2407.16741, 2024

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents.arXiv preprint arXiv:2407.16741, 2024

    Pith/arXiv arXiv 2024

  41. [41]

    Mcts-refined cot: High-quality fine-tuning data for llm-based repository issue resolution.arXiv preprint arXiv:2506.12728, 2025

    Yibo Wang, Zhihao Peng, Ying Wang, Zhao Wei, Hai Yu, and Zhiliang Zhu. Mcts-refined cot: High-quality fine-tuning data for llm-based repository issue resolution.arXiv preprint arXiv:2506.12728, 2025

    Pith/arXiv arXiv 2025

  42. [42]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  43. [43]

    Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution.arXiv preprint arXiv:2502.18449, 2025

    Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I Wang. Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution.arXiv preprint arXiv:2502.18449, 2025

    Pith/arXiv arXiv 2025

  44. [44]

    Demystifying llm-based software engineering agents.Proc

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Demystifying llm-based software engineering agents.Proc. ACM Softw. Eng., 2(FSE), June 2025

    2025

  45. [45]

    Automated program repair via conversation: Fixing 162 out of 337 bugs for $0.42 each using chatgpt

    Chunqiu Steven Xia and Lingming Zhang. Automated program repair via conversation: Fixing 162 out of 337 bugs for $0.42 each using chatgpt. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 819–831, 2024

    2024

  46. [46]

    Swe-fixer: Training open-source llms for effective and efficient github issue resolution.arXiv preprint arXiv:2501.05040, 2025

    Chengxing Xie, Bowen Li, Chang Gao, He Du, Wai Lam, Difan Zou, and Kai Chen. Swe-fixer: Training open-source llms for effective and efficient github issue resolution.arXiv preprint arXiv:2501.05040, 2025

    Pith/arXiv arXiv 2025

  47. [47]

    Enhancing repository-level software repair via repository-aware knowledge graphs.arXiv preprint arXiv:2503.21710, 2025

    Boyang Yang, Jiadong Ren, Shunfu Jin, Yang Liu, Feng Liu, Bach Le, and Haoye Tian. Enhancing repository-level software repair via repository-aware knowledge graphs.arXiv preprint arXiv:2503.21710, 2025

    arXiv 2025

  48. [48]

    Swe-agent: Agent-computer interfaces enable automated software engineering.Advances in Neural Information Processing Systems, 37:50528–50652, 2024

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering.Advances in Neural Information Processing Systems, 37:50528–50652, 2024

    2024

  49. [49]

    Lingxi: Repository-level issue resolution framework enhanced by procedural knowledge guided scaling.arXiv preprint arXiv:2510.11838, 2025

    Xu Yang, Jiayuan Zhou, Michael Pacheco, Wenhan Zhu, Pengfei He, Shaowei Wang, Kui Liu, and Ruiqi Pan. Lingxi: Repository-level issue resolution framework enhanced by procedural knowledge guided scaling.arXiv preprint arXiv:2510.11838, 2025

    arXiv 2025

  50. [50]

    Compositional ex- emplars for in-context learning

    Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. Compositional ex- emplars for in-context learning. InInternational Conference on Machine Learning, pages 39818–39833. PMLR, 2023

    2023

  51. [51]

    Thinkrepair: Self- directed automated program repair

    Xin Yin, Chao Ni, Shaohua Wang, Zhenhao Li, Limin Zeng, and Xiaohu Yang. Thinkrepair: Self- directed automated program repair. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 1274–1286, 2024

    2024

  52. [52]

    Rest-mcts*: Llm self-training via process reward guided tree search.Advances in Neural Information Processing Systems, 37:64735–64772, 2024

    Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mcts*: Llm self-training via process reward guided tree search.Advances in Neural Information Processing Systems, 37:64735–64772, 2024

    2024

  53. [53]

    Repair ingredients are all you need: Improving large language model-based program repair via repair ingredients search

    Jiayi Zhang, Kai Huang, Jian Zhang, Yang Liu, and Chunyang Chen. Repair ingredients are all you need: Improving large language model-based program repair via repair ingredients search. arXiv preprint arXiv:2506.23100, 2025

    Pith/arXiv arXiv 2025

  54. [54]

    Meta prompting for ai systems.arXiv preprint arXiv:2311.11482, 2023

    Yifan Zhang, Yang Yuan, and Andrew Chi-Chih Yao. Meta prompting for ai systems.arXiv preprint arXiv:2311.11482, 2023

    arXiv 2023

  55. [55]

    Autocoderover: Au- tonomous program improvement

    Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. Autocoderover: Au- tonomous program improvement. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 1592–1604, 2024. 13

    2024

  56. [56]

    Calls/issue

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in neural information processing systems, 36:46595–46623, 2023. 14 A Appendix:Implementation Details A.1 Data Construction. For each repository R among the ...

    2023

  57. [2025]

    Accessed: 2025-12-13

    2025