Pith. sign in

REVIEW 4 major objections 4 minor 3 cited by

Skywork-SWE: Unveiling Data Scaling Laws for Software Engineering in LLMs

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that LLM agents' software-engineering skill follows a data scaling law, with pass@1 on SWE-bench Verified reaching 38.0% at 8,209 validated trajectories and still not saturating.

desk verdict A useful dataset-scale SWE model result and a plausible but under-supported scaling-law claim; the dataset deserves release and the curve needs a controlled rerun before 'no saturation' is credible. read the letter →

arxiv 2506.19290 v1 pith:CHWUPHRD submitted 2025-06-24 cs.AI cs.CL

Liang Zeng , Yongcong Li , Yuzhen Xiao , Changshi Li , Chris Yuhao Liu , Rui Yan , Tianwen Wei , Jujie He
show 3 more authors
Xuchen Song Yang Liu Yahui Zhou
This is my paper
classification cs.AIcs.CL
keywords datascalinglawssoftwareengineeringagentsSWE-benchVerifiedexecution-basedvalidationagenttrajectorygenerationsupervisedfine-tuningcodeLLMstest-time
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that software-engineering ability in large language model (LLM) agents is governed by a data scaling law: performance on real GitHub issue-fixing keeps climbing as the number of validated training trajectories grows, and the curve had not flattened at 8,209 trajectories. To test this, it builds an automated pipeline that converts GitHub pull requests into 10,169 execution-validated Python task instances across 2,531 repositories, each with its own container runtime for unit-test validation, and then collects 8,209 multi-turn agent trajectories whose final patches pass all tests. Fine-tuning a 32-billion-parameter open model on these trajectories reaches 38.0% pass@1 on the SWE-bench Verified benchmark without verifiers or multiple rollouts, and 47.0% with test-time scaling, which the paper reports as new best results among similarly sized open software-engineering agents. A reader should care because the finding suggests that the main bottleneck for coding agents is scalable, executable, validated training data rather than model size or agent architecture.

What carries the argument

The load-bearing object is the Skywork-SWE dataset: 10,169 real-world Python issue-fix instances from 2,531 GitHub repositories, each with a natural-language task description, a gold patch, a test patch, and a container runtime image built for automated unit-test execution, plus 8,209 runtime-validated agent trajectories. The machinery that produces it is a three-stage pipeline: repository and pull-request pre-filtering, execution-based validation that keeps only instances with a non-empty FAIL_TO_PASS test set, and agent trajectory generation by proprietary LLMs with final-patch validation. What carries the scaling argument is the trajectory-count versus resolve-rate curve, a log-linear fit over nine training sizes that continues upward and gives the paper its empirical data-scaling-law claim.

What would settle it

Train the same 32B model on random subsets of the 8,209 trajectories that are matched for repository mix, issue difficulty, and unit-test counts, and plot resolve rate against subset size; if the curve flattens or loses monotonicity, the claimed scaling law is an artifact of chronological curation. A simpler observable is whether the last-added trajectories are systematically different in difficulty or repository coverage from the first ones.

Watch

Extended reading notes

Core claim

The central claim, stated in the paper's own terms, is that a log-linear data scaling law holds for software engineering: as the number of training trajectories $N_{\mathrm{traj}}$ increases from 125 to 8,209, the resolve rate on SWE-bench Verified rises monotonically from 6.4% to 38.0%, with no signs of saturation. The paper attributes the trend to the quality and diversity of the curated trajectories, each ending in a patch that passes all unit tests within a per-instance runtime image. It further claims that the same model, given Best-of-8 test-time scaling, reaches 47.0%, surpassing prior state-of-the-art results for sub-32-billion-parameter models. The supporting comparison shows the model outperforming three published 32B agents at the 2,000, 6,000, and 8,000 trajectory marks, which the paper reads as evidence that scaling validated data can match or exceed gains from more complex agent designs.

Load-bearing premise

The scaling curve assumes that the chronological batches of trajectories are comparable samples of one stable distribution of software tasks, even though the paper acknowledges that later subsets may differ in composition, so the steady gains could reflect dataset evolution rather than a general law that adding data alone improves performance.

Editorial extensions

If this is right

  • At a fixed 32B model size, performance rises with each addition of validated trajectories; the paper reports the model overtaking SWE-Dev-32B at 2,000 trajectories, OpenHands-LM-32B-v0.1 at 6,000, and SWE-Agent-LM-32B at 8,000.
  • The dataset doubles as an evaluation resource because every instance brings an executable runtime and validated tests, so organizations can generate and check new training trajectories without manual environment setup.
  • Test-time scaling compounds the data gain: raising independent rollouts from 1 to 8 lifts resolve rate from 38.0% to 47.0%, and extending the maximum interaction rounds from 10 to 100 lifts it from 28.2% to 38.0%.
  • The absence of saturation implies that enlarging the dataset further, rather than enlarging the model or redesigning the agent scaffold, is a viable route to better open-source software-engineering agents.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference, not the paper's: the monotone curve may partly reflect the dataset's chronological construction; Table 4 notes that some trajectories came from earlier subsets, so later additions could be easier or closer to the evaluation distribution, and volume alone may not produce the same gains.
  • A direct test would be to train on random difficulty- and repository-matched subsets of the 8,209 trajectories; if the log-linear curve persists, the scaling law is about data volume, and if it flattens, it is about curation order.
  • The pipeline's yields, roughly 16% of instances install successfully and under 7% pass execution validation, suggest the practical cost of scaling is high; a cost-aware accounting of validated instances per unit of compute and storage would be the next useful quantity.
  • The paper evaluates only Python on one benchmark, so an open question is whether the scaling law transfers to other languages or to closed-source repositories.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper presents Skywork-SWE, an automated pipeline for building a 10,169-instance, 2,531-repository Python software-engineering dataset with runtime-validated unit tests, and uses 8,209 validated agent trajectories to fine-tune Qwen-2.5-Coder-32B-Instruct, reporting 38.0% pass@1 on SWE-bench Verified without verifiers or multiple rollouts and 47.0% with test-time scaling. The headline contribution is an empirical 'data scaling law' for SWE: a log-linear improvement on SWE-bench Verified with increasing trajectory count, with 'no signs of saturation' at 8,209 trajectories. The paper also releases the model checkpoint and discusses pragmatic data-curation lessons.

Significance. If the scaling-law claim held, it would be a valuable result: it would provide evidence that execution-grounded, test-validated trajectory data behaves like standard pretraining data for SWE agents, and it would justify large data-curation investments. The dataset itself is a useful community resource, the explicit exclusion of SWE-bench Verified repositories from collection is commendable, and the evaluation is external and execution-validated, so there is no training/evaluation circularity. However, the scaling-law claim is currently an uncontrolled observational fit rather than a demonstrated law, so the paper's significance is conditional on the additional controlled experiments described below.

major comments (4)
  1. [§4.2, Fig. 1 (top)] The central scaling claim rests on eight single-run data points with no seeds, no variance estimates, and no error bars. On the 500-instance SWE-bench Verified set, the binomial standard error at 38% pass@1 is about 2.2 percentage points, which is comparable to several adjacent increments in the curve (e.g., 18.4 to 20.8 or 25.2 to 28.8). Without repeated runs or confidence intervals, the 'clear log-linear trend' and the monotonic improvement are not statistically established.
  2. [§4.3, Table 4] The scaling curve confounds data quantity with dataset composition. The authors note that 'the dataset was constructed chronologically, some trajectories were collected from earlier subsets,' and Table 4 shows large variation in teacher-model resolve rates (from 3.63% for Gemini-2.0-Flash at temperature 1.0 to 20.23% for Gemini-2.5-Pro). Later prefixes therefore differ not only in N but also in repository mix, task difficulty, and teacher quality. Controlled random-subset retraining from the final 8,209-trajectory corpus at matched N, ideally with multiple seeds, is needed before the gains can be attributed to data quantity alone.
  3. [Abstract, §4.2, Fig. 1 caption] The claim of 'no signs of saturation' extrapolates beyond the observed data. The largest training size is 8,209 trajectories, the fit is an unconstrained log-linear curve, and no saturation model or confidence band is provided. The defensible claim is 'no saturation observed up to 8,209 trajectories'; the current wording overstates what the experiment can support.
  4. [§4.2, Table 3, Fig. 1] The comparative claims contain internal inconsistencies that need correction. SWE-Dev-32B is listed as 36.6 in Table 3 but plotted as 25.2 in Fig. 1, and the text says Skywork-SWE 'outperforms SWE-Dev-32B at N_traj=2000' when both plotted values at that point are 25.2. Similarly, the text states that Skywork-SWE 'exceeds SWE-smith-LM-32B by absolute 6.8% points,' yet Table 3 lists SWE-smith-LM-32B at 40.2, which is higher than the reported 38.0. These discrepancies must be resolved and a consistent evaluation protocol stated before the SOTA comparisons can be audited.
minor comments (4)
  1. [Table 4] The table header contains a typo ('up tp') and the Temperature column entries are inconsistent: reasoning models such as Doubao-1.5-Thinking-Pro and o3-mini are marked '–' but the footnote does not specify the temperature used for the other proprietary models that do support it.
  2. [References] Several reference entries are malformed, including the Hoffmann et al. 2022a entry, which contains a garbled author list ('Lei Chinchilla'), and the Haosheng Zou et al. entry, which begins with 'Shousheng Jia Haosheng Zou, Xiaowei Lv'. These should be cleaned up before publication.
  3. [Fig. 1] The top panel includes baselines such as OpenHands-LM-32B-v0.1 and SWE-Agent-LM-32B that are not defined in Table 3 or in the experimental setup; please add a pointer to their source and state their evaluation protocol so readers can compare them with the reported numbers.
  4. [Fig. 6] The word-cloud figure is hard to read in grayscale and the font sizes are not quantified; a simple bar chart of the top repositories would be more informative.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: evaluation is external, dataset excludes the eval benchmark, and the scaling-law claim is an empirical fit, not a prediction forced by construction.

full rationale

The paper's central derivation chain is: build a new SWE dataset via an automated pipeline, validate trajectories by executing unit tests, fine-tune Qwen2.5-Coder-32B-Instruct, and evaluate on the external SWE-bench Verified benchmark. The evaluation is genuinely held out: the paper states it excludes repositories already in SWE-bench Verified to avoid leakage, and the benchmark is an independent, externally maintained test set. The 38.0% pass@1 result is therefore not circular with respect to the training data. The 'data scaling law' claim is an empirical observation over nine training-set sizes, presented as a fit line; it is not derived from the training data by construction, nor is any fitted parameter renamed as a prediction. The noted chronological composition of the dataset (Table 4: 'the dataset was constructed chronologically, some trajectories were collected from earlier subsets') is a validity or attribution concern about confounding data quantity with data evolution, not a circularity: it does not make the performance curve equal to its inputs by definition. No load-bearing self-citation is used to force the central claim; citations to Kaplan, Hoffmann, SWE-Gym, SWE-Dev, OpenHands, and the OpenHands critic are external references, and no uniqueness theorem from the authors is invoked. Thus the paper is self-contained against an external benchmark, and the appropriate finding is no significant circularity.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper's contributions rest on the validity of SWE-bench Verified as an external benchmark, the use of unit-test passage as a correctness proxy, and the quality of teacher-generated trajectories. No new theoretical entities are introduced; the main free parameter is the slope of the empirically fitted scaling curve.

free parameters (1)
  • slope of log-linear scaling fit = not reported
    The paper fits a log-linear line to resolve rate vs. number of training trajectories (Fig. 1, top) and uses its continued trend to claim no saturation. The slope and intercept are free parameters fitted to the model's own performance data.
assumptions (3)
  • domain assumption SWE-bench Verified is a valid external measure of software engineering capability
    The scaling-law and SOTA claims are evaluated on this 500-instance benchmark; the paper treats resolve rate on it as the target metric.
  • domain assumption Passing unit tests (FAIL_TO_PASS and PASS_TO_PASS) is sufficient validation of a patch's correctness
    Used both in dataset validation (Sec. 3.2.3) and trajectory validation (Sec. 3.3); this defines what counts as a 'successful' trajectory.
  • domain assumption Trajectories generated by proprietary LLMs are high-quality and representative of good SWE problem-solving
    Training data comes entirely from rollouts of GPT-4.1, Gemini, DeepSeek, o3-mini, and others (Table 4); any systematic bias of these teachers is inherited by the fine-tuned model.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Skywork-SWE: Unveiling Data Scaling Laws for Software Engineering in LLMs." pith.science (2026). https://pith.science/paper/CHWUPHRD

@misc{pith2026250619290,
  author       = {Pith},
  title        = {Pith review of: Skywork-SWE: Unveiling Data Scaling Laws for Software Engineering in LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CHWUPHRD}},
  note         = {Machine review of arXiv:2506.19290}
}
read the original abstract

Software engineering (SWE) has recently emerged as a crucial testbed for next-generation LLM agents, demanding inherent capabilities in two critical dimensions: sustained iterative problem-solving (e.g., >50 interaction rounds) and long-context dependency resolution (e.g., >32k tokens). However, the data curation process in SWE remains notoriously time-consuming, as it heavily relies on manual annotation for code file filtering and the setup of dedicated runtime environments to execute and validate unit tests. Consequently, most existing datasets are limited to only a few thousand GitHub-sourced instances. To this end, we propose an incremental, automated data-curation pipeline that systematically scales both the volume and diversity of SWE datasets. Our dataset comprises 10,169 real-world Python task instances from 2,531 distinct GitHub repositories, each accompanied by a task specified in natural language and a dedicated runtime-environment image for automated unit-test validation. We have carefully curated over 8,000 successfully runtime-validated training trajectories from our proposed SWE dataset. When fine-tuning the Skywork-SWE model on these trajectories, we uncover a striking data scaling phenomenon: the trained model's performance for software engineering capabilities in LLMs continues to improve as the data size increases, showing no signs of saturation. Notably, our Skywork-SWE model achieves 38.0% pass@1 accuracy on the SWE-bench Verified benchmark without using verifiers or multiple rollouts, establishing a new state-of-the-art (SOTA) among the Qwen2.5-Coder-32B-based LLMs built on the OpenHands agent framework. Furthermore, with the incorporation of test-time scaling techniques, the performance further improves to 47.0% accuracy, surpassing the previous SOTA results for sub-32B parameter models. We release the Skywork-SWE-32B model checkpoint to accelerate future research.

Figures

Figures reproduced from arXiv: 2506.19290 by the authors.

Figure 1
Figure 1. (Top) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of our three-stage Skywork-SWE data collection pipeline. (1) Stage A. Data Collection and Pre-filtering: Step A.1 scrapes GitHub repo metadata. Step A.2 collects and filters relevant pull requests (PRs). Step A.3 validates PRs via installation checks. (2) Stage B. Execution-based Validation: Step B.1 configures unified execution commands. Step B.2 builds Docker-based runtime environments. Step B.3 validates… view at source ↗
Figure 3
Figure 3. Visualization of data flow across four key hierarchical filtering steps in our data collection pipeline. The first three steps belong to the Data Collection & Pre-filtering stage in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: The default configuration snippet specifying environment setup, dependency installation (lines 2-6), and test execution commands (line 7) for all task instances. 3.2.1. Command Configuration To standardize execution-based validation and runtime environment setup across…
Figure 5
Figure 5. Figure 5: Dockerfile snippets illustrating the construction of a three-level Docker image structure, comprising a base-level image, an environment-level image, and an instance-level image. • The environment-level image builds upon the base image by executing the setup_env.sh scr…
Figure 6
Figure 6. Figure 6: Word cloud of repository names in the Skywork￾SWE dataset. Font size is proportional to the number of instances in each repository. The Skywork-SWE dataset demonstrates high diversity across the collected GitHub repositories. The final Skywork-SWE dataset comprises 10,…
Figure 7
Figure 7. Figure 7: Repository- and year-wise histograms on the Skywork-SWE dataset. (a) The x-axis denotes the number of instances per repository, with every 150 repositories grouped into a single bin for clarity. (b) The x-axis indicates the year in which each issue was created, reflect…
Figure 8
Figure 8. Figure 8: Statistical analysis of instance-level edits on the Skywork-SWE dataset. (a) Histogram of the number of edited files per instance. (b) Histogram of the number of edited functions per instance. (c) Histogram of the number of edited code hunks per instance. (d) Histogram…
Figure 9
Figure 9. Figure 9: Resolve rate (%) of Skywork-SWE-32B under different test-time scaling strategies. (a) Effect of Best-of-N Sampling with 𝑁 ranging from 1 to 8. (b) Effect of maximum rollout turns varying from 10 to 100. 10 to 25 turns yields a 4.6 percentage point gain—while later incr…

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DCAS: Decoupling CLI Agent Scaffolding to Internalize Planning across Scaffolds

    cs.SE 2026-08 conditional novelty 7.0 of 10

    Fine-tuning a 30B coding agent on 576 planning-aware trajectories collected under Claude Code improves its SWE-bench score on unseen harnesses (OpenCode +3.4, mini-swe-agent +7.0 with self-plans).

  2. Function-Aware Fill-in-the-Middle as Mid-Training for Coding Agent Foundation Models

    cs.AI 2026-07 unverdicted novelty 6.0 of 10

    Mid-training on function-aware fill-in-the-middle data improves SWE-Bench by 2.8-5.4 points and reduces post-training capability loss.

  3. SERA: Soft-Verified Efficient Repository Agents

    cs.CL 2026-01 conditional novelty 6.0 of 10

    SERA trains competitive coding agents via supervised finetuning on synthetic trajectories verified only by patch-overlap recall, cutting data-generation cost by roughly two orders of magnitude.

Reference graph

Works this paper leans on

34 extracted references · 10 canonical work pages · cited by 3 Pith papers

  1. [1]

    Claude 3.5 sonnet announcement.https://www.anthropic.com/news/claude-3-5-sonnet, 2024a

    Anthropic. Claude 3.5 sonnet announcement.https://www.anthropic.com/news/claude-3-5-sonnet, 2024a. Accessed: 2025-06-04. Anthropic. Claude haiku 3.5.https://www.anthropic.com/claude/haiku, 2024b. Accessed: 2025-06-

    work page 2025

  2. [4]

    Accessed: 2025-06-04. Shiyi Cao, Sumanth Hegde, Dacheng Li, Tyler Griggs, Shu Liu, Eric Tang, Jiayi Pan, Xingyao Wang, Akshay Malik, Graham Neubig, Kourosh Hakhamaneshi, Richard Liaw, Philipp Moritz, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. Skyrl-v0: Train real-world long-horizon agents via reinforcement learning,

    work page 2025

  3. [5]

    A systematic literature review of the opportunities and advantages for aigc (openai chatgpt, copilot, codex) in programming course

    Chi In Chang, Wan Chong Choi, and Iek Chong Choi. A systematic literature review of the opportunities and advantages for aigc (openai chatgpt, copilot, codex) in programming course. InProceedings of the 2024 7th International Conference on Big Data and Education, pages 29–35,

    work page 2024

  4. [6]

    Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374,

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374,

    arXiv

  5. [7]

    DeepSeek-AI

    Accessed: 2025-05-13. DeepSeek-AI. Deepseek-v3 technical report,

    work page 2025

  6. [8]

    Google DeepMind

    URLhttps://arxiv.org/abs/2412.19437. Google DeepMind. Gemini 2.0 flash.https://deepmind.google/technologies/gemini,

    arXiv

  7. [9]

    Google DeepMind

    Accessed: 2025-06-04. Google DeepMind. Gemini 2.5 pro.https://deepmind.google/models/gemini/pro/,

    work page 2025

  8. [10]

    Accessed: 2025-06-04. 20 Skywork-SWE: Unveiling Data Scaling Laws for Software Engineering in LLMs Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    arXiv 2025

Show all 34 references
  1. [11]

    Jordan Hoffmann, Sebastian Biette, Jacopo de Aiken, James Lei, Zi Lin Chen, Jeff Clune, Li Zhang, Jose Suarez, FrankM.Hutter, GregBrockman, JackW.Rae, LeiChinchilla, etal

    URLhttps: //github.com/Qihoo360/360-LLaMA-Factory. Jordan Hoffmann, Sebastian Biette, Jacopo de Aiken, James Lei, Zi Lin Chen, Jeff Clune, Li Zhang, Jose Suarez, FrankM.Hutter, GregBrockman, JackW.Rae, LeiChinchilla, etal. Trainingcompute-optimallarge language models.arXiv pre...

    arXiv

  2. [12]

    Openai o1 system card.arXiv preprint arXiv:2412.16720,

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Alek- sander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720,

    arXiv

  3. [13]

    Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974,

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974,

    arXiv

  4. [14]

    A survey on large language models for code generation.arXiv preprint arXiv:2406.00515,

    Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation.arXiv preprint arXiv:2406.00515,

    arXiv

  5. [15]

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    URLhttps://openreview.net/forum?id=VTF8yNQM66. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020a. Jared Ka...

    2001 arXiv

  6. [17]

    Sequence parallelism: Long sequence training from system perspective.arXiv preprint arXiv:2105.13120,

    Shenggui Li, Fuzhao Xue, Chaitanya Baranwal, Yongbin Li, and Yang You. Sequence parallelism: Long sequence training from system perspective.arXiv preprint arXiv:2105.13120,

    arXiv

  7. [19]

    Thinking longer, not larger: Enhancing software engineering agents via scaling test-time compute.arXiv preprint arXiv:2503.23803,

    Yingwei Ma, Yongbin Li, Yihong Dong, Xue Jiang, Rongyu Cao, Jue Chen, Fei Huang, and Binhua Li. Thinking longer, not larger: Enhancing software engineering agents via scaling test-time compute.arXiv preprint arXiv:2503.23803,

    arXiv

  8. [20]

    Accessed: 2025-06-03. OpenAI. Introducing gpt-4.1 in the api.https://openai.com/index/gpt-4-1/,

    2025

  9. [21]

    Accessed: 2025- 06-04. OpenAI. Hello gpt-4o, 2024a. URLhttps://openai.com/index/hello-gpt-4o/. Accessed: 2025-05-30. OpenAI. Introducing openai o1, 2024b. URLhttps://openai.com/o1/. Accessed: 2025-05-30. OpenAI. Introducing openai o1 preview, 2024c. URL https://openai.com/inde...

    2025

  10. [22]

    Accessed: 2025-06-04. OpenAI. Openai o3 and o4-mini system card. https://cdn.openai.com/pdf/ 2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf , April

    2025

  11. [23]

    Arvid Orwall

    Accessed: 2025-05-13. Arvid Orwall. Moatless tools: Practical large-codebase editing with language models.https://github. com/aorwall/moatless-tools,

    2025

  12. [24]

    Training software engineering agents and verifiers with swe-gym.arXiv preprint arXiv:2412.21139,

    Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with swe-gym.arXiv preprint arXiv:2412.21139,

    arXiv

  13. [25]

    URLhttps://pypi.org/. Qwen. Qwen2.5-coder series: Powerful, diverse, practical. https://qwenlm.github.io/blog/qwen2. 5-max/, 2024a. Accessed: 2025-06-04. Qwen. Qwen2.5-max: Maximizing reasoning and tool use abilities.https://qwenlm.github.io/blog/ qwen2.5-max/, 2024b. Accessed...

    2025 arXiv

  14. [26]

    Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256,

    22 Skywork-SWE: Unveiling Data Scaling Laws for Software Engineering in LLMs Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256,

    arXiv

  15. [27]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805,

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805,

    arXiv

  16. [28]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599,

    arXiv

  17. [29]

    Xingyao Wang

    Accessed: 2025-05-13. Xingyao Wang. Sota on swe-bench verified with inference-time scal- ing and critic model,

    2025

  18. [30]

    github.io/posts/2023-06-23-agent/

    URLhttps://lilianweng. github.io/posts/2023-06-23-agent/. Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying llm-based software engineering agents.arXiv preprint arXiv:2407.01489,

    2023 arXiv

  19. [31]

    Swe-fixer: Training open-source llms for effective and efficient github issue resolution.arXiv preprint arXiv:2501.05040,

    Chengxing Xie, Bowen Li, Chang Gao, He Du, Wai Lam, Difan Zou, and Kai Chen. Swe-fixer: Training open-source llms for effective and efficient github issue resolution.arXiv preprint arXiv:2501.05040,

    arXiv

  20. [32]

    Swe-smith: Scaling data for software engineering agents

    John Yang, Kilian Leret, Carlos E Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. Swe-smith: Scaling data for software engineering agents. arXiv preprint arXiv:2504.21798,

    arXiv

  21. [33]

    Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury

    URL https://arxiv.org/abs/2504.02605. Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. Autocoderover: Autonomous program improvement. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2024, page 1592–1604, New ...

    2024 arXiv

  22. [34]

    ISBN 9798400706127

    Association for Computing Machinery. ISBN 9798400706127. doi: 10.1145/3650212.3680384. URLhttps://doi.org/10.1145/3650212. 3680384. Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et...

  23. [2021]

    Wizardcoder: Empowering code large language models with evol-instruct.arXiv preprint arXiv:2306.08568,

    21 Skywork-SWE: Unveiling Data Scaling Laws for Software Engineering in LLMs Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct.arXiv pr...

    arXiv

  24. [2023]

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al

    Accessed: 2025-01-14. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you!arXiv preprint arXiv:2305.06161,

    2025 arXiv

  25. [2024]

    Large language monkeys: Scaling inference compute with repeated sampling.arXiv preprint arXiv:2407.21787,

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling.arXiv preprint arXiv:2407.21787,

    arXiv

  26. [2025]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al

    Accessed: 2025-06-04. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models.arXiv preprint arXiv:2108.07732,

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.