Pith. sign in

REVIEW 5 major objections 7 minor 7 cited by

SWE-Debate: Competitive Multi-Agent Debate for Software Issue Resolution

T0 review · 5 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that competitive debate among specialized agents over candidate fault-propagation paths lifts Pass@1 to 41.4% on SWE-Bench-Verified without upgrading the underlying model.

desk verdict A promising framework whose headline SWE-bench result is unverifiable as written because the paper says it never ran the testbed. read the letter →

arxiv 2507.23348 v1 pith:BLSOYEZT submitted 2025-07-31 cs.SE cs.CLcs.LG

Han Li , Yuling Shi , Shaoxin Lin , Xiaodong Gu , Heng Lian , Xin Wang , Yantao Jia , Tao Huang
show 1 more author
Qianxiang Wang
This is my paper · ORCID
classification cs.SEcs.CLcs.LG
keywords multi-agentdebatefaultlocalizationsoftwareissueresolutiondependencygraphSWE-benchMonteCarloTreeSearchLLMagentsrepository-levelrepair
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SWE-Debate argues that the bottleneck in automated software repair is not the reasoning model but what it can see: a single agent exploring a codebase independently tends to lock onto one plausible location and miss faults that span the architecture. The paper's remedy is to generate many candidate fault propagation traces by walking a static dependency graph from entities mentioned in the issue, then to run a three-round competitive debate in which specialized agents rank those traces, propose and critique modification plans, and a discriminator synthesizes a consolidated fix plan that seeds an MCTS patch generator. On SWE-Bench-Verified the framework resolves 207 of 500 issues (41.4% Pass@1), 2.6 points above the strongest baseline using the same DeepSeek-V3-0324 model, and reaches 81.67% file-level localization accuracy on SWE-Bench-Lite. The ablation studies credit the largest share of the gain to generating multiple chains rather than to the debate itself: the paper's message is that localization breadth, not model strength, is the decisive lever.

What carries the argument

The load-bearing object is the fault propagation trace: a chain of code entities (functions, classes, modules) linked by static dependency edges — calls, inheritance, imports, variable references — that traces how an issue's defect may spread through the repository. The framework builds $K \times W$ such chains by matching issue text to entry entities, expanding along issue-relevant neighbors, and running depth-limited graph traversal; a three-round debate then ranks the chains, has $N$ specialized agents propose and competitively refine modification plans, and a discriminator agent synthesizes a consolidated fix plan. That plan initializes an MCTS patch generator whose UCT-guided search, value function, and test execution produce the final edit. The chain is the structuring device: it converts 'which of many plausible locations?' from open-ended search into a ranked choice among concrete alternatives.

What would settle it

Collect the SWE-Bench issues whose gold patches touch files that are not reachable from the issue-mentioned entities through any static call, inheritance, import, or reference edge; if SWE-Debate still resolves a large share of them, the chains cannot be the mechanism the paper claims. A second check: replace the competitive debate with a non-competitive reranker that selects the chain with the most issue-relevant terms and compare Pass@1 — if the gap to the full system disappears, the argumentation stage is not carrying the stated load.

Watch

Extended reading notes

Core claim

The paper's central claim is that competitive, multi-perspective reasoning over multiple graph-derived localization hypotheses resolves more real GitHub issues than any tested single-agent pipeline using the same model: SWE-Debate attains 41.4% Pass@1 on SWE-Bench-Verified, improving 2.6 points over the strongest same-model baseline (SWE-Agent and OpenHands at 38.8% with DeepSeek-V3-0324), and 81.67% file-level localization accuracy on SWE-Bench-Lite, 3.93 points above the strongest baseline across all models, including Claude-3.5 Sonnet. The paper further claims that the diversity of candidate fault-propagation chains, not the raw reasoning power of the model, carries the improvement: removing chain generation costs 10.0 points, removing the edit plan costs 6.0, and removing the debate costs 4.2. On this basis the authors state that the framework sets a new reported state of the art among open-source agent frameworks on the benchmark, and that the localization module can be dropped into existing pipelines such as SWE-Search and Agentless to lift their performance.

Load-bearing premise

The static dependency graph, built by parsing the code's syntax, actually captures the routes along which a bug's effects spread — if the true root cause is only reachable through runtime behavior, dynamic dispatch, or relationships the static edges miss, no candidate chain can carry the debate to it.

Editorial extensions

If this is right

  • With the identical model, debate-guided localization beats independent exploration: 41.4% versus 38.8% for SWE-Agent and OpenHands, and versus 35.4% for SWE-Search on DeepSeek-V3-0324.
  • Localization quality is the dominant lever on resolution: generating multiple propagation chains contributes +10.0 points, more than the edit plan (+6.0) or the debate itself (+4.2).
  • Because the localization module is separable, the paper expects it to improve existing pipelines such as SWE-Search and Agentless without retraining or model changes.
  • Chain depth around 5 maximizes localization accuracy (86.7% on the sampled subset); deeper chains add distracting context and slightly degrade the debate's decisions.
  • The reported numbers position the framework as a new state of the art among open-source agent frameworks on SWE-Bench-Verified.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: the static dependency graph bounds the method's reach — issues whose root cause is only visible through dynamic dispatch, runtime registration, or monkey-patching may never appear on any chain; the authors acknowledge this, and testing on a multilingual or non-Python benchmark would expose it.
  • My inference: the ablation pattern suggests the debate's marginal value is smaller than the chain generation's; if base models keep improving, the 2.6-point same-model gap may shrink, while the 10-point chain-generation effect may persist.
  • My inference: because all debate participants share one model with different prompts, true model heterogeneity is untested; an equivalent single-agent reranker that simply picks the most issue-relevant chain would reveal how much of the gain really comes from argumentation.
  • My inference: the large gap between localization (81.67%) and resolution (41.4%) points to patch generation as the next bottleneck; a natural extension is pairing the debate-produced plan with stronger edit-time verification or test-driven feedback.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. SWE-Debate proposes a three-stage pipeline for repository-level issue resolution: (1) a static dependency graph over code entities plus semantic matching to generate multiple fault propagation chains; (2) a three-round competitive multi-agent debate that selects a chain and synthesizes a modification plan; and (3) an MCTS-based editing agent initialized with that plan. The paper reports 41.4% Pass@1 on SWE-Bench-Verified (Table 1), claiming a 2.6-point improvement over the strongest same-model baseline (SWE-Agent with DeepSeek-V3-0324 at 38.8%), and 81.67% file-level localization accuracy on SWE-Bench-Lite (Table 3). Ablations attribute +10.0 points to multiple chain generation, +4.2 points to multi-agent debate, and +6.0 points to the edit plan (Table 2). The code and data are released, and the method is evaluated on an external benchmark rather than on self-constructed tasks.

Significance. If the reported results are reproducible, SWE-Debate offers a distinctive combination of static graph-guided fault localization, competitive debate, and MCTS-based editing, and the ablation design is a useful decomposition of the pipeline. The paper ships code and data and uses the standard SWE-Bench external test suite, which are strengths. However, the central empirical claims currently rest on single point estimates with no explicit patch-validation protocol, no variance or significance reporting, and an ambiguously defined ablation condition. The claimed 2.6-point improvement and the component attribution therefore remain conditional on evaluation details that the manuscript does not supply.

major comments (5)
  1. [§4.5, §4.4, Table 1] The manuscript states in §4.5: "Due to unsuccessful testbed setup, we did not utilize it in our experiments." SWE-bench Pass@1 is operationally defined as applying a generated patch to the base commit and running the repository's FAIL_TO_PASS and PASS_TO_PASS tests in a testbed. No alternative patch-validation procedure, test-execution harness, or explanation of how the 207/500 resolved instances were determined is given anywhere. If the sentence refers only to an MCTS-internal component, the external evaluation harness is still omitted. This makes the headline 41.4% and the 2.6-point comparison to same-model baselines unverifiable as written. Please provide the exact evaluation harness, patch application and test execution commands, and confirm that the reproduced same-model baselines were evaluated under the same protocol. Please also clarify what "first attempt" means when the MCTS stage internally performs many edit operations.
  2. [§5.1, §5.2, §5.3, Figure 3] All effectiveness and ablation results are reported as single point estimates with no number of runs, seeds, confidence intervals, or significance tests. For stochastic LLM pipelines, differences of 2.6–10.0 percentage points can fall within run-to-run variation. For example, the w/o Multiple Chain Generation result of 31.4% in Table 2 could overlap with the full model's 41.4% under seed variation. Please report multiple independent runs (or at least bootstrap intervals over the 500 instances) and, where possible, paired comparisons of ablations. Figure 3 likewise shows a single curve with no error bars or trial count, so the depth-5 peak is hard to interpret.
  3. [§5.2, Table 2] The ablation "w/o Multi-Agent Debate" removes the entire debate module, but the debate module is also what synthesizes the edit plan; the separate "w/o Edit plan" row removes the plan itself. The manuscript does not state what remains in the "w/o Multi-Agent Debate" condition—for example, whether the chain-selection debate is retained, how the final plan is produced without debate, and whether the same MCTS initialization is used. Consequently, the 4.2-point debate contribution and the 6.0-point edit-plan contribution are not cleanly separable, and the claim that debate alone contributes 4.2 points is not supported. Please specify the exact configurations of both ablation conditions and, if feasible, add an ablation that removes only the competitive refinement rounds while keeping a plan-generation step.
  4. [§5.3, Table 3, §4.4] The localization result of 81.67% Acc@1(File) on SWE-Bench-Lite is not backed by a description of how the localization prediction is produced and evaluated. The metric is defined as success only when all required modification points are included in the top-1 predicted locations, but the manuscript never specifies the ground-truth source for "required modification points" (gold patch files? test failure traces?) or whether the localization is computed by the pipeline's own chain-selection mechanism rather than by an external oracle. Since the localization metric is produced by the pipeline itself, this is also a potential bias concern. Please document the exact localization evaluation script, the ground-truth definition, and whether the same K, W, L, m, N values were used.
  5. [§4.5, §5.4, Appendix A] Hyperparameters (K, W, L, m, N) are said to be set on "a held out set in the full SWE-Bench dataset," but the composition of that set and its overlap with SWE-Bench-Verified are not given. In §5.4, Finding 4 selects chain depth L=5 using a 75-instance subset (SWE-Bench-Verified-S) even though L was already set from the held-out set. This makes the depth recommendation potentially circular or at least double-tuned on a small sample. Please provide the held-out split details and report depth sensitivity on a distinct evaluation set, or explicitly state that L=5 was fixed a priori and only confirmed on the subset.
minor comments (7)
  1. [Throughout] There are several typos and grammar issues, including "offical" (§4.5), "a held out set" (§4.5), "framewwork" (§4.3), "referecing" (§5.3), and "representing an 3.93 percentage point" (§5.3). A thorough proofreading pass is needed.
  2. [Table 1, §4.3] The baseline list in §4.3 does not describe "Moatless Tools," which appears in Table 1 without a citation. Please either add it to the baseline descriptions or remove it from the table.
  3. [Table 3, §4.3] The baseline name is written as "KGComposs" in §4.3 but "KGCompass" in Table 3 and elsewhere; please use one spelling consistently and verify the reference.
  4. [References] References [30] and [31] appear to be the same paper with the same authors and the same arXiv identifier; please deduplicate.
  5. [Figure 3] The figure has no error bars or indication of the number of independent runs; if the curve is from a single run, the caption should say so explicitly.
  6. [Appendix C] The prompt templates contain rendering artifacts from two-column formatting (e.g., "iss ue_des cri pt io n" and "se l e ct e d _n e i gh b o rs"). The final version should use a verbatim or monospaced environment so the prompts are readable and directly usable.
  7. [§6.2] The limitation that static analysis "may miss dynamic relationships and runtime behaviors" is acknowledged, but given that the ablation identifies multiple chain generation as the largest component, it would be useful to quantify how often the graph traversal fails to include the golden patch file, even if only on a small sample.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: SWE-Debate's headline results are external benchmark evaluations; self-citations are related-work only and no derived quantity is defined by its own inputs.

full rationale

SWE-Debate is an empirical systems paper. Its central claims — 41.4% Pass@1 on SWE-Bench-Verified and 81.67% Acc@1(File) on SWE-Bench-Lite — are benchmark results computed against an external test suite and external gold patches; they are not derived from the method's own fitted parameters. The hyperparameters (K, W, L, m, N) are tuned on a held-out set of SWE-Bench and then reported on the verified/lite test subsets; this is conventional model selection, not a definitional reduction. The localization metric is defined with respect to gold modification points, so the pipeline's self-produced chains are measured against external ground truth. The only self-referential elements are (i) the same DeepSeek model both proposes and critiques plans, which is an independence/bias concern rather than a formal circularity, and (ii) self-citations to CodeR [3] and earlier work by Shi and Gu [34,35], which appear only in related-work contexts and are not load-bearing for the framework's design or the empirical claims. Section 4.5's statement that the testbed was not utilized is a serious verifiability gap for the Pass@1 metric — the paper does not describe how patches were executed and FAIL_TO_PASS/PASS_TO_PASS tests were run — but a missing evaluation harness is not an equivalence between input and output, so under the circularity rules it is not scored as a circular step. No fitted quantity is renamed as a prediction, and no uniqueness theorem from the authors' prior work is invoked.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The paper's central contribution is an engineering framework; there are no invented physical entities. The main free parameters are framework hyperparameters tuned on a held-out set and inherited MCTS defaults. The key assumptions are about the sufficiency of static dependency analysis, the reliability of LLM extraction, the diversity of prompt-simulated agents, and the cleanliness of the tuning/evaluation split.

free parameters (6)
  • K (number of entry entities) = 5
    Set based on held-out SWE-bench set (Section 4.5); controls number of seed entities for chain construction.
  • W (breadth-first expansion width) = 4
    Set in Section 4.5; controls how many neighboring entities are considered at each expansion.
  • L (maximum chain depth) = 5
    Set in Section 4.5 and supported by chain depth experiment on 75-instance SWE-Bench-Verified-S (Section 5.4).
  • m (number of chains for competitive ranking) = 6
    Set in Section 4.5; number of candidate localization chains passed to the debate.
  • N (number of debate agents) = 5
    Set in Section 4.5; number of specialized agents in refinement phase.
  • MCTS hyperparameters (c_param, max_iterations, max_depth, thresholds) = c_param=1.41, max_iterations=20, max_depth=20; see Appendix B
    Inherited from SWE-Search defaults; not fitted here, but these choices affect patch generation and resolution rate.
assumptions (4)
  • domain assumption Static dependency graph from Python AST captures relevant fault propagation relationships.
    Section 3.2 constructs G=(V,E) from function calls, inheritance, imports, variable references; dynamic behavior is not modeled and this is acknowledged as a limitation in Section 6.2.
  • domain assumption LLM semantic matching extracts the correct entry entities from issue text.
    Section 3.2 and Prompt 1/2: the pipeline relies on the LLM to identify explicit entities; if extraction misses the root location, all chains are wrong.
  • domain assumption Prompt-simulated agents provide genuinely diverse reasoning perspectives.
    Section 3.3: all agents use the same DeepSeek-V3-0324 base model with different prompts; the paper assumes this yields orthogonal analytical views.
  • ad hoc to paper The held-out set used for hyperparameter tuning does not leak into SWE-Bench-Verified.
    Section 4.5 says parameters are set on a held-out set in the full SWE-Bench dataset, but the overlap with the 500 SWE-Bench-Verified instances is not stated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SWE-Debate: Competitive Multi-Agent Debate for Software Issue Resolution." pith.science (2026). https://pith.science/paper/BLSOYEZT

@misc{pith2026250723348,
  author       = {Pith},
  title        = {Pith review of: SWE-Debate: Competitive Multi-Agent Debate for Software Issue Resolution},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BLSOYEZT}},
  note         = {Machine review of arXiv:2507.23348}
}
read the original abstract

Issue resolution has made remarkable progress thanks to the advanced reasoning capabilities of large language models (LLMs). Recently, agent-based frameworks such as SWE-agent have further advanced this progress by enabling autonomous, tool-using agents to tackle complex software engineering tasks. While existing agent-based issue resolution approaches are primarily based on agents' independent explorations, they often get stuck in local solutions and fail to identify issue patterns that span across different parts of the codebase. To address this limitation, we propose SWE-Debate, a competitive multi-agent debate framework that encourages diverse reasoning paths and achieves more consolidated issue localization. SWE-Debate first creates multiple fault propagation traces as localization proposals by traversing a code dependency graph. Then, it organizes a three-round debate among specialized agents, each embodying distinct reasoning perspectives along the fault propagation trace. This structured competition enables agents to collaboratively converge on a consolidated fix plan. Finally, this consolidated fix plan is integrated into an MCTS-based code modification agent for patch generation. Experiments on the SWE-bench benchmark show that SWE-Debate achieves new state-of-the-art results in open-source agent frameworks and outperforms baselines by a large margin.

Figures

Figures reproduced from arXiv: 2507.23348 by the authors.

Figure 1
Figure 1. Motivating example of multi-agent debate. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overview of SWE-Debate framework. Identifying Entry Nodes via Semantic Matching. Next, the algorithm identifies the top-𝐾 entities 𝐸𝑝 = 𝑒1, . . . , 𝑒𝐾 that are most relevant to the issue description through semantic matching. We employ a language model-based approach to extract structural identifiers that are explicitly referenced in the issue text from the dependency graph represented as an adjacency matrix and ent… view at source ↗
Figure 3
Figure 3. shows the impact of varying chain depth in the com￾petitive debate process on this dataset. We observe that increasing the chain depth from 1 to 5 consistently boosts file-level localiza￾tion accuracy, reaching a peak Acc@1(File) of 86.7%. This suggests that deeper reasoning chains enable more effective exploration of the code graph, capturing complex dependencies and contextual signals that shallow chains often mis… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Case study of SWE-Debate with instance sympy-13974. methods using identical models, SWE-Debate demonstrates substan￾tial improvements: 14.67% over SWE-Agent with DeepSeek-V3-0324 and 8.31% over SWE-Search with GPT-4o , representing an 3.93 percentage point improvement …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LLM Agents Can See Code Repositories

    cs.SE 2026-06 unverdicted novelty 7.0 of 10

    Visual graphs of repository structure added to text inputs for multimodal LLM agents reduce token consumption by up to 26% while maintaining or improving issue-resolution accuracy.

  2. Repository-Aware Metamorphic Relation Generation for Augmented Reality Applications using Large Language Models

    cs.SE 2026-07 conditional novelty 6.0 of 10

    A context-aware LLM pipeline generates metamorphic relations for 142 AR repositories; hierarchical context plus agentic deliberation yields 3,760 refined MRs that human raters judge mostly valid and testable, with 5 c...

  3. PhoenixRepair: Rethinking Repair Strategy Exploration in Software Agents

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A multi-agent repair framework that samples multiple edit locations and iteratively reflects on patch attempts reaches 76.0% Pass@1 on SWE-bench-Verified, up to a 7.8% relative gain over SWE-agent.

  4. Know Before Fix: QA-Driven Repository Knowledge Acquisition for Software Issue Resolution

    cs.SE 2026-07 conditional novelty 6.0 of 10

    QA-driven pre-repair knowledge acquisition (Questioner + Answerer, then Resolver) raises SWE-bench Verified Pass@1 by up to 4.4 points over Mini-SWE-Agent and other pre-repair methods.

  5. Free-MAD: Consensus-Free Multi-Agent Debate

    cs.AI 2025-09 conditional novelty 6.0 of 10

    Free-MAD picks the winning answer by scoring the full trajectory of agents' answers across debate rounds, beating majority voting with fewer rounds.

  6. Detection Time Distribution Predicted Using Absorbing Boundary Conditions and Imaginary Potentials

    quant-ph 2026-03 unverdicted novelty 5.0 of 10

    Three inequivalent quantum detection-time proposals yield waveguide arrival distributions showing partial reflection, spin-orientation independence (for tested parameters), and width dependence when spin couples to th...

  7. Readability-Robust Code Summarization via Meta Curriculum Learning

    cs.SE 2026-01 conditional novelty 4.0 of 10

    A meta-curriculum fine-tuning method improves summary quality on obfuscated Python code while slightly improving quality on clean code.

Reference graph

Works this paper leans on

117 extracted references · 53 canonical work pages · cited by 7 Pith papers

  1. [1]

    Antonis Antoniades, Albert Örwall, Kexun Zhang, Yuxi Xie, Anirudh Goyal, and William Wang. 2024. SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement. arXiv:2410.20285

    arXiv 2024

  2. [2]

    Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate. In The Twelfth International Conference on Learning Representations

    2023

  3. [3]

    Dong Chen, Shaoxin Lin, Muhan Zeng, Daoguang Zan, Jian-Gang Wang, Anton Cheshkov, Jun Sun, Hao Yu, Guoliang Dong, Artem Aliev, Jie Wang, Xiao Cheng, Guangtai Liang, Yuchi Ma, Pan Bian, Tao Xie, and Qianxiang Wang. 2024. CodeR: Issue Resolving with Multi-Agent and Task Graphs. arXiv:2406.01304 [cs]

    arXiv 2024

  4. [4]

    Yuxiao Chen, Jingzheng Wu, Xiang Ling, Changjiang Li, Zhiqing Rui, Tianyue Luo, and Yanjun Wu. 2024. When Large Language Models Confront Repository- Level Automatic Program Repair: How Well They Done?

    2024

  5. [5]

    Zhi Chen, Wei Ma, and Lingxiao Jiang. 2025. Unveiling Pitfalls: Un- derstanding Why AI-driven Code Agents Fail at GitHub Issue Resolution. arXiv:2503.12374 [cs]

    arXiv 2025

  6. [6]

    Zhaoling Chen, Xiangru Tang, Gangda Deng, Fang Wu, Jialong Wu, Zhiwei Jiang, Viktor Prasanna, Arman Cohan, and Xingyao Wang. 2025. LocAgent: Graph-Guided LLM Agents for Code Localization. arXiv:2503.09089 [cs]

    arXiv 2025

  7. [7]

    Gonzalez

    Alejandro Cuadron, Dacheng Li, Wenjie Ma, Xingyao Wang, Yichuan Wang, Siyuan Zhuang, Shu Liu, Luis Gaspar Schroeder, Tian Xia, Huanzhi Mao, Nicholas Thumiger, Aditya Desai, Ion Stoica, Ana Klimovic, Graham Neubig, and Joseph E. Gonzalez. 2025. The Danger of Overthinking: Examining the Reasoning-Action Dilemma in Agentic Tasks

    2025

  8. [8]

    DeepSeek-AI. 2025. DeepSeek-V3 Technical Report. arXiv:2412.19437 [cs]

    arXiv 2025

Show all 117 references
  1. [9]

    Tenenbaum, and Igor Mor- datch

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mor- datch. 2024. Improving Factuality and Reasoning in Language Models through Multiagent Debate. In Proceedings of the 41st International Conference on Machine Learning (ICML’24, Vol. 235). JMLR.org, Vienna...

    2024

  2. [10]

    Ryan Ehrlich, Bradley Brown, Jordan Juravsky, Ronald Clark, Christopher Ré, and Azalia Mirhoseini. 2025. CodeMonkeys: Scaling Test-Time Compute for Software Engineering

    2025

  3. [11]

    Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber

  4. [12]

    Zhang, Michael Luck, and Heming Cui

    Dong Huang, Qingwen Bu, Jie M. Zhang, Michael Luck, and Heming Cui. 2023. AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation. arXiv:2312.13010

    2023 arXiv

  5. [13]

    Mingjian Jiang, Yangjun Ruan, Luis Lastras, Pavan Kapanipathi, and Tatsunori Hashimoto. 2025. Putting It All into Context: Simplifying Agents with LCLMs. arXiv:2505.08120 [cs]

    2025 arXiv

  6. [14]

    Zhonghao Jiang, Xiaoxue Ren, Meng Yan, Wei Jiang, Yong Li, and Zhongxin Liu

  7. [15]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world Github Issues?. In ICLR

    2024

  8. [16]

    Jones and Mary Jean Harrold

    James A. Jones and Mary Jean Harrold. 2005. Empirical Evaluation of the Taran- tula Automatic Fault-Localization Technique. InProceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering . ACM, Long Beach CA USA, 273–282

    2005

  9. [17]

    Xia Li, Wei Li, Yuqun Zhang, and Lingming Zhang. 2019. DeepFL: Integrating Multiple Fault Diagnosis Dimensions for Deep Fault Localization. InProceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis . ACM, Beijing China, 169–180

    2019

  10. [18]

    Yizhou Liu, Pengfei Gao, Xinchen Wang, Jie Liu, Yexuan Shi, Zhao Zhang, and Chao Peng. 2024. MarsCode Agent: AI-native Automated Bug Fixing. arXiv:2409.00899 [cs]

    2024 arXiv

  11. [19]

    Yuhan Liu, Yuxuan Liu, Xiaoqing Zhang, Xiuying Chen, and Rui Yan. 2025. The Truth Becomes Clearer Through Debate! Multi-Agent Systems with Large Language Models Unmask Fake News. arXiv:2505.08532 [cs]

    2025 arXiv

  12. [20]

    Weijie Lv, Xuan Xia, and Sheng-Jun Huang. 2024. CodeACT: Code Adaptive Compute-efficient Tuning Framework for Code LLMs. arXiv:2408.02193 [cs]

    2024 arXiv

  13. [21]

    Yingwei Ma, Rongyu Cao, Yongchang Cao, Yue Zhang, Jue Chen, Yibo Liu, Yuchen Liu, Binhua Li, Fei Huang, and Yongbin Li. 2024. Lingma SWE-GPT: An Open Development-Process-Centric Language Model for Automated Software Improve- ment. arXiv:2411.00622 [cs]

    2024 arXiv

  14. [22]

    Yingwei Ma and Yue Liu. 2025. Improving Automated Issue Resolution via Comprehensive Repository Exploration. In ICLR 2025 Third Workshop on Deep Learning for Code

    2025

  15. [23]

    Yingwei Ma, Qingping Yang, Rongyu Cao, Binhua Li, Fei Huang, and Yongbin Li

  16. [24]

    Xiangxin Meng, Xu Wang, Hongyu Zhang, Hailong Sun, and Xudong Liu. 2022. Improving Fault Localization and Program Repair with Deep Semantic Features and Transferred Knowledge. In Proceedings of the 44th International Conference on Software Engineering. ACM, Pittsburgh Pennsylv...

    2022

  17. [25]

    Niels Mündler, Mark Niklas Mueller, Jingxuan He, and Martin Vechev. 2024. SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  18. [26]

    OpenAI. 2024. Introducing SWE-bench Verified. https://openai.com/index/introducing-swe-bench-verified/

    2024

  19. [27]

    Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. 2024. Training Software Engineering Agents and Verifiers with SWE-Gym

    2024

  20. [28]

    Mike Papadakis and Yves Le Traon. 2015. Metallaxis-FL: Mutation-based Fault Localization. Software Testing, Verification and Reliability 25, 5-7 (Aug. 2015), 605–628

    2015

  21. [29]

    Minh V. T. Pham, Huy N. Phan, Hoang N. Phan, Cuong Le Chi, Tien N. Nguyen, and Nghi D. Q. Bui. 2025. SWE-Synth: Synthesizing Verifiable Bug-Fix Data to En- able Large Language Models in Resolving Real-World Bugs. arXiv:2504.14757 [cs]

    2025

  22. [30]

    Chen Qian, Zihao Xie, YiFei Wang, Wei Liu, Kunlun Zhu, Hanchen Xia, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, Zhiyuan Liu, and Maosong Sun

  23. [31]

    Chen Qian, Zihao Xie, YiFei Wang, Wei Liu, Kunlun Zhu, Hanchen Xia, Yu- fan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, Zhiyuan Liu, and Maosong Sun. 2025. Scaling Large Language Model-based Multi-Agent Collaboration. arXiv:2406.07155 [cs]

    2025 arXiv

  24. [32]

    Yihao Qin, Shangwen Wang, Yiling Lou, Jinhao Dong, Kaixin Wang, Xiaoling Li, and Xiaoguang Mao. 2025. AgentFL: Scaling LLM-based Fault Localization to Project-Level Context. arXiv:2403.16362 [cs]

    2025 arXiv

  25. [33]

    Yuchen Shao, Yuheng Huang, Jiawei Shen, Lei Ma, Ting Su, and Chengcheng Wan

  26. [34]

    In The Thirteenth International Conference on Learning Representations

    Scaling Large Language Model-based Multi-Agent Collaboration. In The Thirteenth International Conference on Learning Representations . Conference’17, July 2017, Washington, DC, USA Li et al

    2017

  27. [35]

    Yuling Shi, Hongyu Zhang, Chengcheng Wan, and Xiaodong Gu. 2024. Between Lines of Code: Unraveling the Distinct Patterns of Machine and Human Program- mers. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE Computer Society, 51–62

    2024

  28. [36]

    Jeongju Sohn and Shin Yoo. 2017. FLUCCS: Using Code and Change Metrics to Improve Fault Localization. In Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis . ACM, Santa Barbara CA USA, 273–283

    2017

  29. [37]

    Xinchen Wang, Pengfei Gao, Xiangxin Meng, Chao Peng, Ruida Hu, Yun Lin, and Cuiyun Gao. 2024. AEGIS: An Agent-based Framework for General Bug Reproduction from Issue Descriptions. arXiv:2411.18015 [cs]

    2024 arXiv

  30. [38]

    In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE)

    Are LLMs Correctly Integrated into Software Systems?. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE) . IEEE, 1178–1190

    2025

  31. [39]

    Yuling Shi, Songsong Wang, Chengcheng Wan, and Xiaodong Gu. 2024. From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debugging. arXiv:2410.01215 [cs]

    2024

  32. [40]

    Zefan Wang, Zichuan Liu, Yingying Zhang, Aoxiao Zhong, Jihong Wang, Fengbin Yin, Lunting Fan, Lingfei Wu, and Qingsong Wen. 2024. RCAgent: Cloud Root Cause Analysis by Autonomous Agents with Tool-Augmented Large Language Models. In Proceedings of the 33rd ACM International Con...

    2024

  33. [41]

    White, Doug Burger, and Chi Wang

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W. White, Doug Burger, and Chi Wang. 2023. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation. arXiv:23...

    2023 arXiv

  34. [42]

    Zhang, Mike Papadakis, Mark Harman, and Yong Liu

    Yonghao Wu, Zheng Li, Jie M. Zhang, Mike Papadakis, Mark Harman, and Yong Liu. 2023. Large Language Models in Fault Localisation. arXiv:2308.15276 [cs]

    2023 arXiv

  35. [43]

    Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brenna...

    2024 arXiv

  36. [44]

    Solved Issues

    You Wang, Michael Pradel, and Zhongxin Liu. 2025. Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Study

    2025

  37. [45]

    Boyang Yang, Haoye Tian, Jiadong Ren, Shunfu Jin, Yang Liu, Feng Liu, and Bach Le. 2025. Enhancing Repository-Level Software Repair via Repository-Aware Knowledge Graphs. arXiv preprint arXiv:2503.21710 (2025)

    2025

  38. [46]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R. Narasimhan, and Ofir Press. 2024. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  39. [47]

    Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang

    John Yang, Kilian Leret, Carlos E. Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. 2025. SWE-smith: Scaling Data for Software Engineering Agents

    2025

  40. [48]

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2024. Agent- less: Demystifying LLM-based Software Engineering Agents. arXiv:2407.01489

    2024 arXiv

  41. [49]

    Aidan Z. H. Yang, Claire Le Goues, Ruben Martins, and Vincent Hellendoorn

  42. [50]

    In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE ’24)

    Large Language Models for Test-Free Fault Localization. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE ’24) . Association for Computing Machinery, New York, NY, USA, 1–12

  43. [51]

    Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, Elsie Nallipogu, Qingwei Lin, Yingnong Dang, Saravan Rajmohan, and Dongmei Zhang. 2025. SWE-bench Goes Live! arXiv:2505.23419 [cs]

    2025 arXiv

  44. [52]

    Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. Au- toCodeRover: Autonomous Program Improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2024). Association for Computing Machinery, New York, N...

    2024

  45. [53]

    Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. 2024. Gptswarm: Language Agents as Optimizable Graphs. In Forty-First International Conference on Machine Learning . A SWE-BENCH-VERIFIED-S SWE-Bench-verified-mini4 is a sub...

    2024

  46. [54]

    Zhongming Yu, Hejia Zhang, Yujie Zhao, Hanxian Huang, Matrix Yao, Ke Ding, and Jishen Zhao. 2025. OrcaLoca: An LLM Agent Framework for Software Issue Localization. arXiv:2502.00350 [cs]

    2025

  47. [55]

    Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, Siyao Liu, Yongsheng Xiao, Liangqiang Chen, Yuyu Zhang, Jing Su, Tianyu Liu, Rui Long, Kai Shen, and Liang Xiang. 2025. Multi-SWE-bench: A Multilingual Benchm...

    2025 arXiv

  48. [56]

    Guibin Zhang, Yanwei Yue, Xiangguo Sun, Guancheng Wan, Miao Yu, Junfeng Fang, Kun Wang, Tianlong Chen, and Dawei Cheng. 2025. G-Designer: Archi- tecting Multi-agent Communication Topologies via Graph Neural Networks. In ICLR 2025 Workshop on Foundation Models in the Wild

    2025

  49. [60]

    Analyze the issue description to identify : - ** Classes **: e . g . , ` UserAuthenticator `, ` PaymentProcessor ` - ** Methods / Functions **: e . g . , ` v a l i d a t e _ c r e d e n t i a l s ()`, ` process_payment () ` - ** Variables / Parameters **: e . g . , ` user_id `...

  50. [61]

    ** Focus on direct mentions **: Only include entities that are clearly referenced in the issue

  51. [62]

    the payment handler

    ** Avoid redundancy **: If multiple terms refer to the same entity ( e . g . , " the payment handler " and ` PaymentProcessor `) , pick the most precise name

  52. [63]

    ** Prioritize key components **: Rank entities by how central they are to the issue

  53. [64]

    ** Return only names **: Do not include paths , modules , or extra descriptions

  54. [65]

    entity_name1

    ** Limit to { max_entities } entities **: Select only the { max_entities } most relevant and important entities for this issue . ** Output Format :** Return a JSON list of exactly { max_entities } entity names in order of relevance ( most relevant first ) : [" entity_name1 " ,...

  55. [66]

    Count " ,

    ** Issue Description :** Query syntax error with condition and distinct combination Description : A Count annotation containing both a Case condition and a distinct = True param produces a query error on Django 2.2 ( whatever the db backend ) . A space is missing at least (......

  56. [67]

    After upgrading to v2 .0 , the ` UserSession ` class sometimes fails to store session data in Redis , causing login loops

    ** Issue Description :** " After upgrading to v2 .0 , the ` UserSession ` class sometimes fails to store session data in Redis , causing login loops ." ** Output ( if max_entities =2) :** [" UserSession " , " Redis "]

  57. [68]

    The ` c a lc u l at e _d i s co u n t ()` function applies incorrect discounts for bulk orders when ` customer_type = ' wholesale ' `

    ** Issue Description :** " The ` c a lc u l at e _d i s co u n t ()` function applies incorrect discounts for bulk orders when ` customer_type = ' wholesale ' `." ** Output ( if max_entities =3) :** [" c a lc u l at e _ di s c ou n t " , " customer_type " , " wholesale "] Note...

    2017

  58. [69]

    Analyze the problem statement to understand what needs to be fixed / implemented

  59. [70]

    Review the code snippets to identify relevant entities

  60. [71]

    ** PRIORITIZE DIVERSITY **: Select entities from different files whenever possible to ensure comprehensive coverage

  61. [72]

    ** BALANCE RELEVANCE AND DIVERSITY **: Choose entities that are both highly relevant to the issue AND come from different modules / files

  62. [73]

    Avoid selecting multiple entities from the same file unless absolutely necessary

  63. [74]

    Select exactly 4 entities that collectively provide the best coverage for solving the issue

  64. [75]

    entity_id

    For each entity , provide the exact entity ID in the format expected by the codebase ** Selection Strategy :** - First priority : High relevance to the problem + Different file locations - Second priority : High relevance to the problem ( even if some files overlap ) - Ensure ...

    2017

  65. [76]

    ** Relevance to Issue **: How likely is this neighbor to contain code related to solving the issue ?

  66. [77]

    ** Diversity **: Avoid selecting too many entities from the same file or with similar names

  67. [78]

    ** Strategic Value **: Prioritize entities that could lead to discovering the root cause or solution

  68. [79]

    ** Entity Type Variety **: Balance between files , classes , and functions when possible ** Instructions :**

  69. [80]

    Analyze each neighbor entity ID to understand what it likely represents

  70. [81]

    Consider file paths , entity names , and types to assess relevance

  71. [82]

    Ensure diversity by avoiding redundant selections from the same file / module

  72. [83]

    Select entities that complement each other in exploring different aspects of the issue

  73. [84]

    se l e ct e d _n e i gh b o rs

    Return exactly the entity IDs that should be explored further ( up to { max_selection }) ** Output Format :** Return a JSON object with your selection : ``` json {{ " se l e ct e d _n e i gh b o rs ": [ " n e i g h b o r _ e n t i t y _ i d _ 1 " , " n e i g h b o r _ e n t i ...

  74. [85]

    Analyze how each neighbor might relate to solving the issue

  75. [86]

    Consider the traversal depth and whether we should continue or stop

  76. [87]

    Evaluate which neighbor is most likely to contain relevant code for the solution

  77. [88]

    should_continue

    Return your decision on whether to continue exploration and which neighbor to select ** Output Format :** Return a JSON object with your decision : ``` json {{ " should_continue ": true / false , " s el ec te d_n ei gh bo r ": " n ei g h bo r _ en t i ty _ i d or null " , " re...

  78. [89]

    ** Problem Location Accuracy **: Does this chain contain the actual location where the bug / issue manifests ?

  79. [90]

    ** Modification Impact **: How directly would changes to this code path affect the described problem ?

  80. [91]

    ** Code Modifiability **: Is the code in this chain well - structured and safe to modify ?

  81. [92]

    ** Solution Completeness **: Would fixing this chain likely resolve the entire issue , not just symptoms ? Conference’17, July 2017, Washington, DC, USA Li et al

    2017

  82. [93]

    ** Risk Assessment **: What are the risks of modifying this particular code path ? ** Key Questions to Consider :** - Which chain contains the root cause rather than just related functionality ? - Where would a developer most likely need to make changes to fix this specific is...

  83. [94]

    For each chain , analyze whether modifying its code would directly address the issue

  84. [95]

    Consider the logical flow : which chain is most likely to contain the problematic code ?

  85. [96]

    Evaluate implementation feasibility : which chain would be safest and most effective to modify ?

  86. [97]

    Vote for exactly ONE chain that represents the best modification target

  87. [98]

    voted_chain_id

    Focus on where to make changes , not just what ' s related to the issue ** Output Format :** Return a JSON object with your vote : ``` json {{ " voted_chain_id ": " chain_X " , " confidence ": 85 , " reasoning ": " Detailed explanation of why this chain is the best modificatio...

  88. [99]

    Examine each entity in the localization chain and its code

  89. [100]

    Identify which specific parts of the code are causing the issue or need enhancement

  90. [101]

    Determine the precise locations where modifications should be made

  91. [102]

    Explain why each location needs modification and what type of change is required

  92. [103]

    Prioritize the modifications by importance ( most critical first )

  93. [104]

    m o d i f i c a t i o n _ l o c a t i o n s

    For each modification , provide DETAILED implementation instructions with specific code examples ** Output Format :** Return a JSON object with your analysis : ``` json {{ " m o d i f i c a t i o n _ l o c a t i o n s ": [ {{ SWE-Debate: Competitive Multi-Agent Debate for Soft...

    2017

  94. [105]

    Review your initial analysis and the analyses from other agents

  95. [106]

    Identify common patterns and disagreements in the proposed modifications

  96. [107]

    Synthesize the best insights from all analyses

  97. [108]

    Refine your modification recommendations based on collective wisdom

  98. [109]

    Provide a more comprehensive and well - reasoned final recommendation

  99. [110]

    r e f i n e d _ m o d i f i c a t i o n _ l o c a t i o n s

    Ensure each s u g ge s te d _ ap p r oa c h contains exhaustive implementation details ** Output Format :** Return a JSON object with your refined analysis : ``` json {{ " r e f i n e d _ m o d i f i c a t i o n _ l o c a t i o n s ": [ {{ " entity_id ": " s pe c i fi c _ en t...

    2017

  100. [111]

    Analyze all the expert recommendations and identify the most reliable and consistent suggestions

  101. [112]

    Resolve any conflicts between different expert opinions using technical merit

  102. [113]

    Create a prioritized , step - by - step modification plan with ONLY concrete modification actions

  103. [114]

    Ensure the plan is practical , safe , and addresses the root cause of the issue

  104. [115]

    Include specific instructions for each modification

  105. [116]

    The output context should be as detailed as possible

  106. [117]

    Add " ,

    Use action verbs like : " Add " , " Modify " , " Replace " , " Insert " , " Update " , " Change " , " Remove " , " Implement " ** Output Format :** Return a comprehensive modification plan : ``` json {{ " final_plan ": {{ " summary ": " High - level summary of the modification...

    2017

  107. [2023]

    arXiv:2308.00352

    MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework. arXiv:2308.00352

    arXiv

  108. [2024]

    How to Understand Whole Software Repository? arXiv:2406.01422 [cs]

    arXiv

  109. [2025]

    arXiv:2503.22424 [cs]

    CoSIL: Software Issue Localization via LLM-Driven Code Repository Graph Searching. arXiv:2503.22424 [cs]

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.