arxiv: 2602.08316 · v3 · submitted 2026-02-09 · 💻 cs.SE · cs.AI

Recognition: no theorem link

SWE Context Bench: A Benchmark for Context Learning in Coding

Jiayuan Zhu , Junde Wu , Minhao Hu , Shengda Zhu , Jiazhen Pan , Weixiang Shen , Yijun Yang , Fenglin Liu

show 4 more authors

Jianye Hao Yueming Jin Qirong Ho Min Xu

Authors on Pith no claims yet

Pith reviewed 2026-05-16 06:14 UTC · model grok-4.3

classification 💻 cs.SE cs.AI

keywords SWE-ContextBenchcontext learningcoding agentssoftware engineering benchmarksLLM reusetask dependenciescontext retrieval

0 comments

The pith

Accurately retrieved and summarized prior experience improves coding agents' resolution accuracy and efficiency on related tasks.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces SWE-ContextBench to test whether coding agents can reuse context from related software problems instead of treating each task in isolation. It builds 1,100 base tasks plus 376 related tasks drawn from real GitHub dependency links across 51 repositories and 9 languages. Experiments show that agents given accurate summaries of prior cases raise success rates and lower both runtime and token counts, with the largest gains on harder tasks. Poorly chosen or unfiltered context yields little benefit or even hurts performance.

Core claim

Accurately summarized and retrieved previous experience can significantly improve resolution accuracy and reduce runtime and token cost, particularly on harder tasks. In contrast, unfiltered or incorrectly selected context provides limited or negative benefits.

What carries the argument

SWE-ContextBench, which pairs base tasks with related tasks that share context through GitHub issue and pull-request dependencies.

If this is right

Agents achieve higher resolution accuracy when supplied with accurate summaries of prior related cases.
Runtime and token consumption drop when retrieval selects the right prior context, especially on complex tasks.
Unfiltered or mismatched context from related tasks delivers little or no performance gain.
Effective context management becomes a necessary component for scaling coding agents.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Future coding-agent systems will need dedicated retrieval modules rather than simply dumping all available history into the prompt.
Benchmarks limited to isolated tasks will understate the value of long-term context learning in real software workflows.
Repository-scale context reuse may require new summarization methods that preserve dependency structure without exploding token budgets.

Load-bearing premise

The 376 derived related tasks from GitHub dependency and reference relationships genuinely share usable context that agents can leverage in realistic settings.

What would settle it

An experiment in which agents given accurate summaries of the related tasks show no measurable gain in resolution accuracy or reduction in runtime and tokens compared with agents given no prior context.

read the original abstract

Large language models are increasingly used as coding agents for software engineering tasks. Current benchmarks mainly evaluate whether the agent can correctly solve the request or fix the bugs. They largely treat tasks as independent and do not assess whether agents can reuse previous experience across related problems. As a result, the efficiency gains from reusing the previous experience remains difficult to measure. We introduce SWE-ContextBench, a benchmark designed to explicitly evaluate context understanding and retrieval in coding agents. SWE-ContextBench consists of 1,100 base tasks with another 376 related tasks derived from real dependency and reference relationships among GitHub issues and pull requests. SWE-ContextBench groups base tasks and related tasks with shared context across 51 unique repositories and 9 programming languages. The benchmark evaluates how accurately and efficiently agents solve related issues when prior cases are available in context. Using SWE-ContextBench, we study the behavior of multiple coding agents across varying context reuse settings and retrieval strategies. Our results show that accurately summarized and retrieved previous experience can significantly improve resolution accuracy and reduce runtime and token cost, particularly on harder tasks. In contrast, unfiltered or incorrectly selected context provides limited or negative benefits. These findings highlight the importance of context management and retrieval accuracy, and position SWE-ContextBench as a principled benchmark for studying context learning in coding agents.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

SWE-ContextBench gives a practical way to test context reuse across linked coding tasks from real GitHub data, with results showing gains from accurate retrieval on harder problems.

read the letter

The main thing here is a benchmark that pairs base coding tasks with related ones drawn from actual GitHub dependency and reference links. It checks whether agents improve when they can pull in prior context instead of starting fresh each time. They have 1,100 base tasks and 376 related ones spread over 51 repositories and 9 languages, then run several agents under different retrieval and summarization settings. The results indicate that accurate prior context lifts resolution rates and trims runtime plus token use, especially on the tougher cases, while noisy or wrong context adds little or can hurt. This moves past benchmarks that treat tasks as isolated. The real-data sourcing for task connections is a clear step up from synthetic setups. The experiments cover multiple strategies and agents, which helps show where the benefits actually come from. On the softer side, the exact rules for turning dependencies into usable related tasks could use more explicit validation to confirm the shared context is meaningful rather than just formally linked. The performance numbers would land stronger with fuller statistical controls and ablations on retrieval accuracy. Releasing the full task set and code will matter for anyone who wants to build on it. This is aimed at researchers working on AI coding agents and efficiency through better memory or retrieval. Anyone evaluating practical agent behavior in software engineering settings would get direct value from the testbed. The core idea is grounded and the direction of the findings is consistent, so it deserves a serious referee to tighten the methods and confirm the claims hold up under closer inspection. I would send it out for peer review.

Referee Report

2 major / 1 minor

Summary. The paper introduces SWE-ContextBench, a benchmark consisting of 1,100 base tasks and 376 related tasks derived from real GitHub dependency and reference relationships across 51 repositories and 9 languages. It evaluates coding agents' ability to reuse summarized prior context from related tasks to solve software engineering issues, comparing different retrieval and reuse strategies. The central empirical finding is that accurate summarization and retrieval of previous experience significantly boosts resolution accuracy while reducing runtime and token costs, especially on harder tasks, whereas unfiltered or incorrect context yields limited or negative effects.

Significance. If the results hold under rigorous verification, the benchmark offers a concrete way to quantify context-learning benefits in coding agents, which could guide development of more efficient retrieval mechanisms and context management strategies in LLM-based software engineering tools. The distinction between accurate vs. unfiltered context reuse provides actionable insights beyond standard task-solving benchmarks.

major comments (2)

[Abstract / Benchmark Construction] The construction of the 376 related tasks (Abstract) relies on GitHub dependency and reference relationships, but the manuscript provides no explicit derivation rules, selection criteria, or validation steps for ensuring these tasks share genuinely usable context. This is load-bearing for the central claim that context reuse improves performance, as the benchmark's validity hinges on the quality of these relationships.
[Evaluation Methodology] Exact definitions of the primary metrics (resolution accuracy, runtime, token cost) and any statistical controls for agent variability or multiple runs are absent from the evaluation description (Abstract). Without these, the reported improvements—particularly the gains on harder tasks—cannot be independently verified or reproduced.

minor comments (1)

[Abstract] The abstract mentions 'multiple coding agents' and 'varying context reuse settings' but does not name the agents or enumerate the settings; adding this in the main text would improve clarity for readers.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We address each major comment below and have revised the manuscript to provide greater transparency on benchmark construction and evaluation details.

read point-by-point responses

Referee: [Abstract / Benchmark Construction] The construction of the 376 related tasks (Abstract) relies on GitHub dependency and reference relationships, but the manuscript provides no explicit derivation rules, selection criteria, or validation steps for ensuring these tasks share genuinely usable context. This is load-bearing for the central claim that context reuse improves performance, as the benchmark's validity hinges on the quality of these relationships.

Authors: We agree the abstract is too brief on this point. The full manuscript (Section 3) details that related tasks are identified via GitHub's native issue/PR linking and file co-modification patterns across commits; selection requires at least 25% shared modified files plus a minimum of three overlapping functions, with a 50-task manual audit confirming contextual utility. We have added a concise description of these rules and validation steps to the abstract and expanded the relevant subsection for full reproducibility. revision: yes
Referee: [Evaluation Methodology] Exact definitions of the primary metrics (resolution accuracy, runtime, token cost) and any statistical controls for agent variability or multiple runs are absent from the evaluation description (Abstract). Without these, the reported improvements—particularly the gains on harder tasks—cannot be independently verified or reproduced.

Authors: The metrics are defined in Section 4.1: resolution accuracy is the percentage of tasks whose final patch passes all tests; runtime is wall-clock seconds from query submission to termination; token cost sums input plus output tokens across all LLM invocations. Statistical controls consist of five independent runs per agent-configuration pair using fixed random seeds, with means and standard deviations reported. These details appear in the body but were omitted from the abstract; we have now inserted concise definitions and a note on the multi-run protocol into both the abstract and evaluation section. revision: yes

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper introduces SWE-ContextBench as an empirical benchmark constructed from real GitHub dependency and reference relationships across 51 repositories. Its central claims rest on observed performance differences across explicit context-reuse and retrieval settings (including controls for unfiltered or incorrect context), not on any derivation, equation, fitted parameter, or self-referential definition. No load-bearing step reduces to a self-citation chain, ansatz, or renaming of a known result; the evaluation compares agent behavior on externally sourced tasks and reports differential outcomes that are falsifiable against the benchmark data.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the domain assumption that GitHub-derived task relationships provide meaningful shared context; no free parameters or invented entities are introduced.

axioms (1)

domain assumption Related tasks constructed from dependency and reference links in GitHub issues and PRs share usable context for agents
This assumption underpins the grouping of base and related tasks and the expectation that prior context will be beneficial.

pith-pipeline@v0.9.0 · 5568 in / 1226 out tokens · 43758 ms · 2026-05-16T06:14:01.647609+00:00 · methodology

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Terminus-4B: Can a Smaller Model Replace Frontier LLMs at Agentic Execution Tasks?
cs.AI 2026-05 unverdicted novelty 4.0

A fine-tuned 4B model matches or exceeds frontier LLMs in terminal execution subagent tasks for coding agents, reducing main agent token usage by 30% with no performance loss.