SWE-Adept: An LLM-Based Agentic Framework for Deep Codebase Analysis and Structured Issue Resolution

Kang He; Kaushik Roy

REVIEW 3 major objections 4 minor 1 cited by

This paper claims that a two-agent design—lightweight depth-first code search plus Git-based step-by-step checkpointing—raises end-to-end resolve rates by up to 4.3 percentage points on SWE-Bench Pro.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 19:38 UTC pith:VJMU26BZ

load-bearing objection A coherent systems contribution with a genuinely useful checkpointing design, but the headline margins rest on unverifiable evaluation artifacts; worth referee time, not unconditional trust. the 3 major comments →

arxiv 2603.01327 v2 pith:VJMU26BZ submitted 2026-03-01 cs.SE cs.CLcs.LG

SWE-Adept: An LLM-Based Agentic Framework for Deep Codebase Analysis and Structured Issue Resolution

Kang He , Kaushik Roy This is my paper

classification cs.SE cs.CLcs.LG

keywords LLM agentssoftware engineeringissue localizationissue resolutioncodebase navigationGit checkpointingworking memorySWE-Bench

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

SWE-Adept tries to establish that repository-level bug fixing can be made substantially more reliable by giving LLMs two specialized roles rather than one free-form loop. The paper proposes a localization agent that searches code with depth-first, dependency-aware traversal, receiving only compact previews until a final filtering stage, and a resolution agent that plans hypotheses, tracks to-dos, and checkpoints every edit with Git-based version control, storing code states in working memory indexed by execution steps. On SWE-Bench Lite and a 200-instance SWE-Bench Pro subset, it reports function-level localization accuracy gains of up to 6.2% and end-to-end resolve-rate gains of up to 4.3%, while using fewer tokens than graph-based search methods. A sympathetic reader would care because the results suggest that scaffolding—context management, structured planning, and reversible state—matters as much as model choice for real software engineering.

Core claim

SWE-Adept consistently outperforms prior approaches in both issue localization and issue resolution on SWE-Bench Lite and a 200-instance SWE-Bench Pro subset. The localization agent traverses a code-structure tree with agent-directed depth-first search, receiving only lightweight structural previews until a two-stage filter loads full code for shortlisted candidates, which improves function-level localization accuracy by up to 6.2%. The resolution agent uses hypothesis_plan and hypothesis_git tool families—adaptive to-do lists and Git-based checkpoints stored in working memory indexed by semantic steps—to branch, revert failed edits, and merge the best fix, raising end-to-end resolve rates b

What carries the argument

The load-bearing machinery is the combination of (1) a definition-level code index with a code-structure tree (nodes are code units like functions and classes; edges are 'contains' or 'invokes' relations), stored as lightweight adjacency lists in each unit's metadata; (2) agent-directed depth-first traversal, where the localization agent selectively follows one dependency path at a time and search tools return only previews, skeletons, and child-unit identifiers; (3) two-stage filtering, which defers full-code loading to the final re-ranking step; and (4) a tool-memory interface for the resolution agent, in which hypothesis_git commands wrap raw Git operations and working memory stores check

Load-bearing premise

The headline improvements assume the 200-instance SWE-Bench Pro subset, sampled 'randomly' without a published seed, is representative; if the sample is skewed, the 4.3-point resolve-rate gain and per-language breakdowns may not generalize.

What would settle it

Run SWE-Adept and the strongest prior agent on the full SWE-Bench Pro test set or the published instance list; if the 4.3-percentage-point resolve-rate advantage does not reproduce, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

If the reported gains reproduce, restructuring an agent loop around context-lean search and explicit state checkpoints can be an effective alternative to scaling models.
Improved function-level localization directly raises resolve rates, so keeping issue-irrelevant content out of context may matter more than more sophisticated retrieval.
Semantic-step checkpointing makes long-horizon repair trajectories safer, opening the door to multi-hypothesis exploration for complex multi-file bugs.
Token reductions from preview-based search lower per-instance cost, improving feasibility for larger repositories.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The two-agent split may matter less than the tooling: a single agent equipped with the same preview-based search and checkpointed Git tools might match the gains, which would isolate the contribution of separation.
The preview-and-filter design is worth testing on repositories larger than SWE-Bench, where context overflow is more severe; the token savings could translate into even larger accuracy gains.
The checkpointing mechanism could transfer to other long-horizon agent tasks, such as data pipeline debugging or scientific computation fixes, where reverting to a known-good state is valuable.
Because the Pro subset is stratified by language (60 Python, 60 JavaScript, 60 Go, 20 TypeScript) but sampled without a published seed, the per-language numbers are the most fragile results; a fixed-seed rerun would strengthen them.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A coherent systems contribution with a genuinely useful checkpointing design, but the headline margins rest on unverifiable evaluation artifacts; worth referee time, not unconditional trust.

read the letter

Short version: this is a real systems contribution, but the exact numbers are not yet trustworthy. The two-agent split is not new by itself, but the specific combination is: a localization agent doing depth-first traversal over a definition-level code-structure tree with lightweight previews and deferred full-code loading, and a resolution agent that manages hypotheses, to-do lists, and Git checkpoints through a semantic tool-memory interface. The second piece is the most interesting. Wrapping raw Git operations into high-level commands and indexing checkpoints by to-do step directly addresses a known failure mode: agents lose track of code state over long edit trajectories. Table 6 is honest evidence for this—raw Git prompted into SWE-agent doesn't reproduce the gains and even hurts on Pro.

The ablations are the strongest part of the paper. Each agent is tested with and without the proposed module, and the results point in the expected direction. The token counts also support the context-management claim, though the savings are modest and the comparison is not controlled for quality.

Where I part company with the paper is the evaluation precision. The margins are small: 4.3 points on 200 instances is about 9 instances; the Lite gains are 8-9 instances out of 300. The Pro subset is sampled with language quotas but no seed or instance IDs, so no one can check whether it is representative. Two runs at temperature 0.1, averaged, with no variance or significance tests, cannot support the word "consistently." The paper's own ablation shows the variance issue is real: a 3.0-point improvement on Lite can flip to -2.0 on Pro in the raw-Git ablation. On top of that, no code or trajectories are released, so the subset, the baselines, and the runs are all unverifiable from the text. These are evaluation-validity problems, not architecture flaws. I don't see a circularity or math issue—the evidence is external benchmark performance, and the framework is not defined in terms of the benchmarks.

Bottom line: the framework deserves referee time, and I'd be happy to see it in a venue that values systems work, but acceptance should be conditional on releasing the evaluation artifacts and reporting variance. The paper is worth a serious referee; the current numbers are not.

Who's it for: people building repo-level coding agents. The semantic-checkpointing design is worth stealing. I'd cite the design, not the margins.

Referee Report

3 major / 4 minor

Summary. The paper proposes SWE-Adept, a two-agent LLM framework for repository-level issue resolution. A localization agent performs agent-directed depth-first traversal over a tree-sitter-based definition-level code-structure tree, using lightweight previews and two-stage filtering to identify issue-relevant functions/files. A resolution agent, built on SWE-agent, uses the hypothesis_plan and hypothesis_git tool families with a JSON-serialized working memory to maintain code-state checkpoints indexed by to-do steps, enabling branching, reverting, and merging. On SWE-Bench Lite and a 200-instance SWE-Bench Pro subset, the authors report consistent improvements over several baselines, e.g., up to 6.2% function-level Acc@5 and up to 4.3% resolve rate. The paper includes per-component ablations, action-pattern analyses, and an error analysis.

Significance. SWE-Adept addresses a practically important problem, and the architectural ideas—separate contexts for localization and resolution, dependency-guided DFS with deferred full-code loading, and semantic-step Git checkpoints—are plausible and clearly described. If the empirical claims are robust, the framework is a useful advance for agentic software engineering. The paper's strengths include detailed tool/prompt documentation, internally consistent tables, and ablations that isolate the localization and resolution components. However, the headline quantitative claims rest on an unseeded 200-instance subset and two-run averages with no variance reporting; the claimed margins (2.8–4.3 points on 200–300 instances) are comparable to only a few resolved instances. The evidence is therefore conditional: the architecture story is supported, but the exact performance margins are not yet statistically grounded.

major comments (3)

[Appendix A.1 and Tables 2-3] The SWE-Bench Pro evaluation uses a 200-instance subset constructed by sampling 60 Python, 60 JavaScript, 60 Go, and all 20 TypeScript instances, but no seed or instance IDs are provided. This subset is not the standard benchmark, so the 'SWE-Bench Pro' results are not reproducible and may depend on a favorable random draw. The language-wise breakdowns in Table 3 are especially sensitive: with n=60 per language, a 3–5-point difference is only 2–3 instances, and with n=20 for TypeScript the 10-point gains are just 2 instances. Please publish the seed and instance IDs (or the full subset), and ideally evaluate on the full public test set or report uncertainty over multiple random subsets.
[Appendix A.2 and Tables 4-6] All results are averaged over two runs at temperature 0.1, with no per-run values, variance, confidence intervals, or significance tests. On the 200 Pro instances, the headline 4.3% gap is about 8.6 instances; on the 60-instance language strata, most Table 3 gaps are 2–6 instances. The same issue affects the ablations: for example, Table 5 shows a 1.5-point Lite gain for the proposed context management, and Table 6 shows a 2.3-point raw-Git loss on Lite, both within plausible run-to-run variation. Please report run-level numbers, provide bootstrap confidence intervals or significance tests, and increase the number of runs where feasible. Without this, the claim that SWE-Adept 'consistently outperforms' baselines is not statistically supported.
[General (all experiments)] No code, prompts, or trajectory logs are released. The paper describes the method in detail, but the tool implementations, working-memory serialization, baseline reproduction settings, and the exact candidate sets are not independently verifiable. The behavioral statistics in Figures 4-5 and the error analysis in Appendix B rely on trajectories that are not available. This is a load-bearing reproducibility gap for the empirical claims, although not a formal correctness error. I would expect a public artifact (code, subset IDs, and logs) as part of a revised submission.

minor comments (4)

[§4.2] The text says SWE-Adept 'mostly consumes fewer tokens than graph-based approaches' immediately after Table 1, but Table 2 shows that end-to-end token consumption is higher than several baselines (e.g., Claude-Lite: 3119k vs RepoGraph 2968k and OrcaLoca 1875k). Please clarify that the token-efficiency claim applies to localization only, and discuss the resolution-stage cost tradeoff.
[Figures 4-5] These figures report distributions and accuracy/resolve-rate lines without error bars or sample sizes per bin. Adding n and confidence intervals would strengthen the behavioral claims.
[Limitations section] The Limitations section discusses proprietary models and lack of self-evolution, but does not mention the evaluation-validity limitations: the unseeded Pro subset, the two-run averages, or the absence of released artifacts. These should be acknowledged.
[Table 5] The label 'Search returns code preview + specialized filtering' is verbose; consider a short name. Also, the table shows that the proposed localization design sometimes yields only a small gain (e.g., GPT-5.2 Lite: 69.3% vs 70.8%); the text should temper the causal claim accordingly.

Circularity Check

0 steps flagged

No significant circularity: the paper's central claims rest on external benchmark comparisons, not on self-referential derivation or fitted predictions.

full rationale

SWE-Adept is an empirical systems paper; its load-bearing evidence is external benchmark performance on SWE-Bench Lite and SWE-Bench Pro against independent published baselines (SWE-agent, RepoGraph, OpenHands, SWE-Search, OrcaLoca, LocAgent). No equation defines a target quantity in terms of a fitted parameter, and no prediction is a re-labeling of calibration data. The design components (two-agent split, DFS traversal, two-stage filtering, semantic-step checkpointing) are engineering choices evaluated by ablations in Tables 4-6, not derived from a self-cited theorem. The only self-citation, He and Roy 2025, appears in a related-work sentence about prior work having limited structure for systematic problem solving; it is background context and does not support any reported result. Appendix A.1's unseeded 200-instance SWE-Bench Pro subset and Appendix A.2's two-run averaging without variance or error bars are legitimate reproducibility and generalization concerns, but they concern statistical validity, not circularity. The limitations section similarly asserts scope limits rather than circular dependence. Therefore no circular step is present; with only a minor non-load-bearing self-citation, the appropriate finding is no significant circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

This is a systems/empirical paper, so the ledger contains hand-set hyperparameters and benchmark assumptions rather than fitted physical constants or new postulated entities. The central claim depends on the chosen subset and on the two-run average being representative.

free parameters (4)

Code chunk size = 200 lines
Remaining code outside definitions is segmented into 200-line chunks; this granularity affects code-unit indexing and preview context.
Maximum localization iterations = 20
The localization agent is limited to 20 action iterations per instance, capping search depth.
Sampling temperature = 0.1
Both models use temperature 0.1; results are averaged over two runs, but no variance is reported.
Per-instance cost limit for resolution = $5
The resolution agent stops at $5 per instance, limiting how many hypotheses and edits can be explored.

axioms (4)

domain assumption SWE-Bench Lite and the 200-instance SWE-Bench Pro subset are valid proxies for real repository-level issue resolution.
All resolve-rate claims are measured against these benchmarks; if the subset is not representative, the claimed gains may not generalize.
domain assumption The tree-sitter parse and contains/invokes edge construction faithfully capture the dependencies needed for localization.
Agent-directed DFS traverses these edges; if callbacks or dynamic dispatch are missed, root-cause paths can be skipped.
domain assumption LLMs (GPT-5.2, Claude-Sonnet-4.5) follow the tool interface and prompts reliably enough that two runs represent method performance.
No error bars or significance tests are provided; the framework's effectiveness depends on instruction following over long horizons.
domain assumption Ground-truth patch locations are accurate labels for localization.
Acc@k marks an instance correct only if all patch-modified locations appear in the top-k set, which may overstate or understate the true fault-relevant locations.

pith-pipeline@v1.3.0-alltime-deepseek · 18375 in / 9875 out tokens · 97199 ms · 2026-08-02T19:38:23.032690+00:00 · methodology

0 comments

read the original abstract

Large language models (LLMs) exhibit strong performance on self-contained programming tasks. However, they still struggle with repository-level software engineering (SWE), which demands (1) deep codebase navigation with effective context management for accurate localization, and (2) systematic approaches for iterative, test-driven code modification to resolve issues. To address these challenges, we propose SWE-Adept, an LLM-based two-agent framework where a localization agent identifies issue-relevant code locations and a resolution agent implements the corresponding fixes. For issue localization, we introduce agent-directed depth-first search that selectively traverses code dependencies. This minimizes issue-irrelevant content in the agent's context window and improves localization accuracy. For issue resolution, we employ adaptive planning and structured problem solving. We equip the agent with specialized tools for progress tracking and Git-based version control. These tools interface with a shared working memory that stores code-state checkpoints indexed by execution steps, facilitating precise checkpoint retrieval. This design enables reliable agent-driven version-control operations for systematic issue resolution, including branching to explore alternative solutions and reverting failed edits. Experiments on SWE-Bench Lite and SWE-Bench Pro demonstrate that SWE-Adept consistently outperforms prior approaches in both issue localization and resolution, improving the end-to-end resolve rate by up to 4.3%.

Figures

Figures reproduced from arXiv: 2603.01327 by Kang He, Kaushik Roy.

**Figure 1.** Figure 1: Overview of Issue Localization framework. (a) Codebase is indexed and represented as code-structure tree in the structured database. Based on this representation, (b) Issue Localization Agent performs search and pinpointing: (b-1) agent-directed depth-first traversal for selective, dependency-aware exploration, with search tools ( [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** Figure 2: Overview of Issue Resolution framework. (a) Issue Resolution Agent takes identified code locations as input and performs structured issue resolution. The agent is equipped with two CLI (command-line interface)-based tool families (§ 3.3), hypothesis_plan ( [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Figure 3: Venn diagram (left) of resolved-instance over [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗

**Figure 4.** Figure 4: Search behavior of Issue Localization Agent and localization accuracy by maximum search depth. (a) Search-tool invocation distribution of Issue Localization Agent. (b) Instance distribution (bars) and function-level localization accuracy (lines) by maximum search depth on SWE-Bench Lite with Claude-Sonnet-4.5. (a) Systematic problem-solving behaviors of Issue Resolution Agent 30% 40% 50% 60% 70% 80% 0 30 6… view at source ↗

**Figure 5.** Figure 5: Problem-solving behavior of Issue Resolution Agent and resolve rate by number of explored hypotheses. (a) Prevalence of systematic problem-solving behaviors: multi-hypothesis branching, dynamic to-do expansion, and checkpoint-based reversion (behaviors are non-mutually exclusive and may overlap). (b) Instance distribution (bars) and resolve rate (lines) by number of explored hypotheses on SWE-Bench Lite wi… view at source ↗

**Figure 6.** Figure 6: System prompt and stepwise instruction template for [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗

**Figure 7.** Figure 7: System prompt and stepwise instruction template for [PITH_FULL_IMAGE:figures/full_fig_p018_7.png] view at source ↗

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Know Before Fix: QA-Driven Repository Knowledge Acquisition for Software Issue Resolution
cs.SE 2026-07 conditional novelty 6.0

QA-driven pre-repair knowledge acquisition (Questioner + Answerer, then Resolver) raises SWE-bench Verified Pass@1 by up to 4.4 points over Mini-SWE-Agent and other pre-repair methods.

Reference graph

Works this paper leans on

5 extracted references · 2 linked inside Pith · cited by 1 Pith paper

[1]

Remove your generated reproduction/test script
[2]

w/o designed

Memory OS of AI agent. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 25961–25970, Suzhou, China. Association for Computational Linguistics. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paran- jape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language mod- els use long...

Pith/arXiv arXiv 2025
[3]

Figure 7: System prompt and stepwise instruction template forIssue Resolution Agent

Finally, run thesubmitcommand. Figure 7: System prompt and stepwise instruction template forIssue Resolution Agent. 18
[4]

If you have modified any original test files, restore them to the initial state
[2025]

Swe-bench pro: Can ai agents solve long- horizon software engineering tasks?arXiv preprint arXiv:2509.16941. Google. 2025. A new era of intelligence with gemini 3. Hiroaki Hayashi, Bo Pang, Wenting Zhao, Ye Liu, Akash Gokul, Srijan Bansal, Caiming Xiong, Semih Yavuz, and Yingbo Zhou. 2025. Self-abstraction from grounded experience for plan-guided policy r...

Pith/arXiv arXiv 2025

[1] [1]

Remove your generated reproduction/test script

[2] [2]

w/o designed

Memory OS of AI agent. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 25961–25970, Suzhou, China. Association for Computational Linguistics. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paran- jape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language mod- els use long...

Pith/arXiv arXiv 2025

[3] [3]

Figure 7: System prompt and stepwise instruction template forIssue Resolution Agent

Finally, run thesubmitcommand. Figure 7: System prompt and stepwise instruction template forIssue Resolution Agent. 18

[4] [4]

If you have modified any original test files, restore them to the initial state

[5] [2025]

Swe-bench pro: Can ai agents solve long- horizon software engineering tasks?arXiv preprint arXiv:2509.16941. Google. 2025. A new era of intelligence with gemini 3. Hiroaki Hayashi, Bo Pang, Wenting Zhao, Ye Liu, Akash Gokul, Srijan Bansal, Caiming Xiong, Semih Yavuz, and Yingbo Zhou. 2025. Self-abstraction from grounded experience for plan-guided policy r...

Pith/arXiv arXiv 2025