Pith. sign in

REVIEW 4 major objections 5 minor 21 references

SWE-Dev: Building Software Engineering Agents with Training and Inference Scaling

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SWE-Dev shows that synthesizing fail-to-pass tests from GitHub issues, training on verified trajectories, and letting the agent take up to 75 rounds in one run lets a 32B open-weight model resolve 36.6% of SWE-bench-Verified issues.

desk verdict The headline 36.6% is credible but provisional until the authors rule out SWE-bench overlap in their training crawl. read the letter →

arxiv 2506.07636 v2 pith:VVRPXSVZ submitted 2025-06-09 cs.AI

Haoran Wang , Zhenyu Hou , Yao Wei , Jie Tang , Yuxiao Dong This is my paper
classification cs.AI
keywords softwareengineeringagentssynthetictestcasegenerationrejectionsamplingfine-tuningtrajectorydatascalinginference-timeiterationSWE-benchopen-sourcelanguagemodelscodeLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SWE-Dev is a recipe for turning an open-weight code model into a software-engineering agent without human-written test annotations. The paper's central claim is that LLM-synthesized fail-to-pass tests, generated from issue descriptions and golden patches, can serve as training labels: they pick out correct agent trajectories for rejection-sampling fine-tuning. On the inference side, the paper shows that letting the agent take more interaction rounds in a single episode (up to 75) raises resolve rates further. The resulting 32-billion-parameter model resolves 36.6% of SWE-bench-Verified issues, the best reported for an open-source agent and above GPT-4o's 33.2%, with a 7B model at 23.4%. A sympathetic reader would take this as evidence that data and inference scaling, not model size alone, drive SWE agent performance.

What carries the argument

The load-bearing mechanism is a two-stage test-synthesis pipeline: first a description model writes a Gherkin-style scenario capturing the issue, then a code model writes a pytest test that must fail on the unpatched repository and pass after the golden patch is applied. This turns each collected issue into an executable reward signal. The same signal filters the 17k sampled trajectories down to 2.3k correct ones for rejection-sampling fine-tuning, and the paper's iteration-scaling strategy extends a single evaluation episode to more interaction rounds so the agent can revise its own work.

What would settle it

Run the trained SWE-Dev-32B on SWE-bench-Verified instances, score its patches with the pipeline's synthetic tests, then score the same patches with the official developer-written tests; if many patches pass the synthetic tests while failing the official tests (or vice versa), the oracle assumption fails.

Watch

Extended reading notes

Core claim

The paper sets out to show that the bottleneck in training software-engineering agents is the missing test oracle, and that this bottleneck can be removed by synthesis. Its core discovery is a scaling relationship: as the number of sampled agent trajectories grows from 574 to 16,639, the 7B model's resolve rate rises from 13.0% to 22.8% along a near-log-linear curve, and when the interaction budget is extended from 30 to 75 rounds, the 32B model improves from 34.0% to 36.6%. Along the way, rejection-sampling fine-tuning beats the offline reinforcement-learning methods tried (KTO, OREO, and their hybrids), and an LLM-based patch filter keeps 65% of trajectories without hurting performance. These results together claim that a 32B open model can exceed prior open-source agents and match or beat a frontier proprietary model without relying on verifiers or repeated sampling.

Load-bearing premise

The whole training signal rests on LLM-generated fail-to-pass tests being faithful to the issue; if a test passes for reasons unrelated to the fix, the trajectory filters and rejection-sampling labels built on it are wrong.

Editorial extensions

If this is right

  • Training a SWE agent no longer requires pull requests to come with test cases: synthesized fail-to-pass tests provide the reward signal at scale.
  • Resolve rate grows roughly linearly with log training trajectories in the tested range, so collecting more trajectories should keep helping before saturation.
  • Extending a single run from 30 to 75 rounds gives a 2.6-point gain for the 32B model, a cheaper route than repeated pass@k sampling when test oracles are expensive.
  • Rejection-sampling fine-tuning outperforms the offline reinforcement-learning methods tried (KTO, OREO, and RFT+OREO hybrids), so high-quality positive trajectories matter more than the optimization objective.
  • A 32B open-weight agent can beat GPT-4o on SWE-bench-Verified, narrowing the gap between open and proprietary systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the synthetic oracle has a systematic blind spot, for example tests that encode the patch's behavior rather than the issue's actual requirement, the reported gains could overstate true repair ability; the paper's own validation benchmark is also built from synthetic tests, so it cannot fully exclude this risk.
  • Only 2,097 of 26k instances yielded fail-to-pass tests, so the trained models' behavior on issues without an available synthetic oracle may differ from the headline benchmark numbers.
  • The log-linear training-scaling trend implies that even larger trajectory pools should help, but only if oracle quality is preserved; with noisy labels, scaling would amplify the noise.
  • The iteration-scaling result invites an adaptive early-stopping or learned 'done' check that spends rounds only where they pay off, which the paper lists as future work rather than testing.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces SWE-Dev, an open-weight software-engineering agent built on Qwen2.5-Coder, Llama-3.1, and GLM-4 backbones. It constructs a large training set by crawling PyPI/GitHub repositories (38k instances from 4,413 repos), synthesizes test cases with an LLM pipeline, and uses DeepSeek-V3-generated trajectories under an OpenHands scaffold. Post-training comparisons cover RFT, KTO, OREO, and hybrid variants, and the paper studies scaling in training-trajectory count and in the number of interaction rounds. On SWE-bench-Verified, SWE-Dev-7B achieves 23.4% and SWE-Dev-32B achieves 36.6%, with the authors claiming state-of-the-art performance among open-source SWE agents.

Significance. If the headline numbers survive scrutiny, the paper makes a useful empirical contribution: it shows that a 32B open-weight model can exceed larger open models and match a proprietary model (GPT-4o) on SWE-bench-Verified, and it identifies RFT as the most effective post-training method among those tested. The public release of code, models, and data is a concrete strength that enables direct verification of the claimed results. However, the significance is currently conditional on two structural risks: possible overlap between the training crawl and SWE-bench-Verified source repositories, and the reliance on LLM-generated tests as training oracles without human validation. The scaling claims are interesting but confounded by the simultaneous change of context length and interaction budget in the inference-scaling experiment. The paper's tables and figures are internally consistent, and the dataset comparison against Nebius in Table 3 is a good external anchor.

major comments (4)
  1. [§3.1] The training-set construction follows the extraction methodology of Jimenez et al. (2024) from the same PyPI/GitHub repository pool that underlies SWE-bench and SWE-bench-Verified, yet the paper never states that SWE-bench or SWE-bench-Verified instances were excluded from the 38k-instance training set or from the 2.3k correct trajectories used for RFT. Since an SWE-bench instance is defined by a specific repo, PR, and issue, any overlap would directly inflate the 36.6% resolve rate reported in Table 4 and Figure 1. The authors should report a decontamination analysis (e.g., exact and near-duplicate matches by repo+PR number, issue text, and patch similarity) and re-evaluate the models after removing any overlapping instances.
  2. [§3.2 / Table 3] The synthesized fail-to-pass tests are generated from the golden patch and the issue description, which makes them a plausible training signal but also creates a self-referential oracle: the validation in Table 3 compares two datasets under the same synthetic oracle, so it cannot detect a systematic flaw such as tests that pass for reasons unrelated to the fix or tests that are too permissive. Because trajectory filtering (Figure 6) and RFT labels (Figures 4–5) depend entirely on these tests, a systematic oracle error would corrupt the training signal. The paper should add a human-evaluation study of a random sample of generated tests and/or a cross-check of a subset of synthesized F2P tests against developer-written tests on SWE-bench-Verified instances.
  3. [§4.2 / Table 4] The claim of state-of-the-art performance among open-source SWE agents is not fully supported by the comparison set. Table 4 omits several systems cited in the paper itself (e.g., SWE-Search from Antoniades et al., 2024, and open-weight baselines under scaffolds other than OpenHands/Agentless), and no error bars or significance tests are reported for the main resolve rates. The difference between 36.6% and the 32.8% of SWE-Fixer-72B may not be meaningful if the evaluation was run once. The authors should expand the baseline set, report the number of runs and variance, and ideally include a paired or bootstrap significance test.
  4. [§5.2 / Figure 7] The inference-scaling experiment changes two variables at once: the number of interaction rounds (30 to 75) and the context configuration (RoPE extended from 32k to 160k; max_tokens set to 160k for these runs). The text states that the RoPE change alone caused a drop for the 7B model, so the improvement in Figure 7 cannot be attributed solely to more interaction rounds. The authors should decouple these factors, for example by comparing 30 rounds with 160k context against 75 rounds with 160k context, and 75 rounds with 32k context if feasible.
minor comments (5)
  1. [§3.3] The sentence 'These experiments collectively highlight the strengths of our dataset in generating high-quality test cases and its utility for downstream tasks' appears twice in succession; one copy should be removed.
  2. [§4.2] The text says that SWE-Syninfer-72B and SWE-Fixer-72B 'have a resolve rate of 36.6%,' but Table 4 reports 30.2% and 32.8%, respectively; the sentence should be corrected to avoid an internal inconsistency.
  3. [Table 3] The caption refers to a '7B base model' with a 13.6% resolve rate, while Table 4 reports 13.6% for the 9B GLM-4 model; the model size should be stated consistently.
  4. [Table 1] The abbreviation 'F2F' is not defined; readers may confuse it with 'fail-to-fail' tests, so please spell it out or define it in the caption.
  5. [§3.1] The sentence 'Following the methodology outlined in Jimenez et al. (2024) with minor modifications' is too vague; the exact modifications and extraction filters should be specified so that the contamination risk can be assessed precisely.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark result is externally evaluated on developer-written SWE-bench tests, and the synthetic-test loop is training-data construction rather than a derivation-equivalent-to-input.

full rationale

The paper's central claim, a 36.6% resolve rate on SWE-bench-Verified, is measured against developer-written unit tests from an external benchmark, not against the synthetic tests generated from golden patches. The synthetic fail-to-pass tests are explicitly constructed from the golden patch and issue description (see Appendix A.1 prompts: 'Given a GitHub issue description and the golden patch, your task is to build test cases that reproduce the error according to the patch'), and they are used only to filter trajectories and provide RFT labels. This is a training-data construction loop, but the paper does not present the synthetic tests as independent predictions, and the headline evaluation does not reduce to them by construction. Table 3 additionally grounds the training signal by comparing SWE-Dev against the Nebius dataset on the external benchmark, so the result is not forced by the self-referential oracle alone. Self-citations to Hou et al. (2024, 2025) support general scaling observations but are not load-bearing for the main result. The most serious risk is potential train/test contamination: Section 3.1 crawls instances following the Jimenez et al. (2024) methodology from the same PyPI/GitHub pool used to build SWE-bench, and no deduplication against SWE-bench-Verified is reported. That is a leakage/correctness risk, not a circularity defect under the specified patterns, because the paper's equations or definitions do not make the benchmark outcome equivalent to its training inputs.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central pipeline relies on a chain of manually chosen hyperparameters and domain assumptions about the reliability of synthesized test cases and benchmark cleanliness. The trained model weights are the product of standard training under these assumptions, not independent scientific constants.

free parameters (4)
  • Interaction rounds (iteration budget) = 30 to 75
    Chosen by hand; the inference-scaling result (23.4% and 36.6%) depends on evaluating at 75 rounds, while default training/eval uses 30 rounds.
  • RoPE max_tokens context extension = 160k (from 32k)
    Set to enable longer interaction; initial context-extension drop for 7B (22.8% to 21.8%) is reported but not included in the headline gain.
  • Trajectory filtering retention threshold = ~65%
    Only trajectories rated 'identical' or 'mostly identical' by Llama-3.1-70B are kept; the threshold is a manual design choice.
  • RFT fine-tuning scales = 4 epochs for 7B, 8 epochs for 32B, lr 1e-5
    Selected by hand; final model quality depends on these settings.
assumptions (4)
  • domain assumption LLM-generated test cases, derived from golden patch and issue description, are valid fail-to-pass oracles for trajectory labeling.
    Invoked throughout Section 3.2 and 4.3; Table 3 validates relative dataset quality only under the same synthetic oracle.
  • domain assumption Trajectories labeled 'correct' because the generated patch passes the synthetic test actually solve the underlying issue.
    Used for RFT in Section 4.3; false-positive labels would bias training.
  • domain assumption SWE-bench-Verified is a fair, contamination-free evaluation of the trained agents.
    The paper never states that SWE-bench repositories (e.g., django, sympy) were excluded from the 4,413 training repos; Section 3.1 describes crawling PyPI repos without a decontamination step.
  • domain assumption The OpenHands ReAct scaffold and DeepSeek-V3 trajectories represent a useful training distribution for the final agents.
    Used to build the 17k trajectory pool in Section 4.1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SWE-Dev: Building Software Engineering Agents with Training and Inference Scaling." pith.science (2026). https://pith.science/paper/VVRPXSVZ

@misc{pith2026250607636,
  author       = {Pith},
  title        = {Pith review of: SWE-Dev: Building Software Engineering Agents with Training and Inference Scaling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VVRPXSVZ}},
  note         = {Machine review of arXiv:2506.07636}
}
read the original abstract

Large language models (LLMs) have advanced rapidly from conversational problem solving to addressing real-world tasks involving tool use, such as software engineering (SWE). Recent LLM-powered toolkits, such as OpenAI Codex and Cursor, have offered end-to-end automation of the software development process. However, building effective SWE agents remains challenging due to the lack of high-quality training data and effective test cases. To address this issue, we present SWE-Dev, an SWE agent built upon open-source LLMs. First, we develop a robust pipeline to synthesize test cases for patch evaluation. Second, we scale up agent trajectories to construct the training data for building SWE-Dev. Experiments on the SWE-bench-Verified benchmark show that the SWE-Dev models can achieve top performance among all open SWE agents. Specifically, the success rates of the SWE-Dev 7B and 32B parameter models reach 23.4% and 36.6%, respectively, outperforming state-of-the-art open-source models. All code, models, and datasets are publicly available at https://github.com/THUDM/SWE-Dev.

Figures

Figures reproduced from arXiv: 2506.07636 by the authors.

Figure 1
Figure 1. The SWE-DEV performance with training and inference scaling. Notably, SWE-Dev-32B achieves a resolve rate of 34.0%, matching the performance of GPT-4o even without inference scaling. SWE tasks usually require LLMs to interact with complex and fragile runtime environments, solve toolchain issues, execute scripts, and reason over large, interdependent codebases (Ma et al., 2024b). The SWE tasks are usually evaluated o… view at source ↗
Figure 2
Figure 2. Pipeline for test case generation, divided into description generation and code generation phases. The [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Distribution of instances per repository in the [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Model performance across different training [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Model performance with hybrid training. In [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Resolve rates across different training data sizes. Both 7B and 32B models exhibit performance [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Resolve rates across different numbers of interaction rounds (30–75). Larger models benefit more [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 18 canonical work pages

  1. [1]

    Differences in logging or documentation can be ignored

    When comparing patches, focus only on code changes that affect actual functionality. Differences in logging or documentation can be ignored

    work page

  2. [2]

    •mostly: Patch 2 implements most of the functionality correctly with minor differences

    The evaluation levels are: •identical: Patch 2’s functionality is exactly the same as Patch 1. •mostly: Patch 2 implements most of the functionality correctly with minor differences. •partially: Patch 2 only implements some of the required functionality. •different: Patch 2’s functionality is completely different or incorrect

    work page

  3. [3]

    If Patch 2 is rated aspartiallyormostly, please specify the functional differences in the explanation. Final Instructions: • Provide your analysis and judgment in the following format: [Explanation] Explanation of the reason why the patch is judged as ’identical’, ’mostly’, ’partially’, or ’different’. [Judgment] The judgment of the patch is ’identical’, ...

    work page

  4. [4]

    Instructions:- Wrap each Gherkin test description in triple backticks (“‘gherkin)

    Avoid including unimportant test cases, such as modifications to README files. Instructions:- Wrap each Gherkin test description in triple backticks (“‘gherkin). - Example format: ```gherkin {{YOUR DESCRIPTION}} ``` Prompt for Test Case Generation You are a test engineer. Given a GitHub issue description and the golden patch, your task is to build test ca...

    work page

  5. [7]

    Ensure the test triggers this behavior reliably and consistently

    Fail with the unpatched code: Demonstrate the specific bug, issue, or limitation that the patch is designed to address. Ensure the test triggers this behavior reliably and consistently

    work page

  6. [8]

    Focus Areas: • Exercise uncommon or edge-case code paths

    Pass with the patched code: Confirm that the patch resolves the issue without introducing new problems or regressions. Focus Areas: • Exercise uncommon or edge-case code paths. • Test for boundary conditions or unexpected input. • Mimic realistic usage scenarios where the original behavior fails. Information provided: •Repository name: {} •GitHub issue de...

    work page

  7. [9]

    Use theGiven-When-Thenstructure of Gherkin syntax

    work page

  8. [10]

    • Triggering events (When)

    Clearly describe: • Preconditions (Given). • Triggering events (When). • Expected outcomes (Then)

    work page

Show all 21 references
  1. [11]

    Ensure the test logic is clear, concise, and covers all relevant scenarios

  2. [13]

    Identify the required packages and the types of test cases to write

    Identify the incorrect code: Analyze the provided information to locate the error that the patch addresses. Identify the required packages and the types of test cases to write

  3. [14]

    Each test case must be enclosed within < testcase ></ testcase > tags

    Generate the test case: Write test cases that willfail without the correction patchandpass with the correction patch. Each test case must be enclosed within < testcase ></ testcase > tags

  4. [15]

    " " C a l c u l a t et h e sum o f a l l numbers i n a CSV f i l e

    Ensure that no additional execution beyond your test case is performed. Avoid unsafe commands or unnecessary changes to the project. Format Requirements: • Test Case: –Wrap each test case in < testcase ></ testcase > tags. –Use triple backticks (```) to enclose the test code w...

  5. [16]

    It tries to convertrow[0]to an integer usingint()

  6. [17]

    The goal is to write test cases that:

    Ifrow[0]is not a valid integer, it skips that row using atry...exceptblock. The goal is to write test cases that:

  7. [18]

    Test case with non-numeric data in the CSV (should raise an error in the original code)

  8. [19]

    " " F i x t u r e t o c r e a t e a t e m p o r a r yCSV f i l e f o r t e s t i n g

    Same test case should now correctly handle non-numeric rows and calculate the sum of valid numeric values. Example Test Case: < t e s t c a s e > ```python importos importp y t e s t froms r c . u t i l s . c s v _ u t i l simportread_csv_and_sum @pytest . f i x t u r e defc r...

  9. [20]

    For example: • Rewrite wrong test cases if errors occur on specific tests

    Analyze the error history carefully: Review the error history to understand why the previous test cases passed without the patch. For example: • Rewrite wrong test cases if errors occur on specific tests. • Considerimportdependencies when encounteringImportErroror similar errors

  10. [21]

    fix the issue,

    Preserve the original intent: Ensure the new test cases still target the original issues that the patch is designed to fix. 3.Format Requirements: Your test case should strictly follow the original format. Specifically: • Setup commands should be wrapped in <env></env> tags. T...

  11. [2022]

    Hao Liu, Matei Zaharia, and Pieter Abbeel

    Competition-level code generation with alpha- code.Science, 378(6624):1092–1097. Hao Liu, Matei Zaharia, and Pieter Abbeel. 2023. Ring attention with blockwise transformers for near- infinite context.Preprint, arXiv:2310.01889. Yizhou Liu, Pengfei Gao, Xinchen Wang, Jie Liu, Y...

    2023 arXiv

  12. [2024]

    Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar

    Scaling data collection for training software engineering agents.Nebius blog. Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar. 2024. Di- girl: Training in-the-wild device-control agents with autonomous reinforcement learning.Preprint, ar...

    2024 arXiv

  13. [2025]

    Advancing language model reasoning through reinforcement learning and inference scaling. In ICML. Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. 2024. Openrlhf: An easy- to-use, scalable and high-performance rlhf frame- work.Preprint, arXiv:2405.11...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.