Pith. sign in

REVIEW 5 major objections 7 minor 57 references

PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks

T0 review · 5 major / 7 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper establishes that 13.6% of SWE-bench Verified instances pair a pull request with a misaligned issue, and that a three-phase multi-agent checker can detect such misalignment with up to 92.12% binary accuracy.

desk verdict A genuinely useful taxonomy and a well-built checker, but the headline accuracy and prevalence numbers rest on the authors' own ground truth and need independent re-annotation before I'd trust them. read the letter →

arxiv 2607.28587 v2 pith:MULX42CS submitted 2026-07-30 cs.SE cs.AI

Manyi Wang , Junjielong Xu , Pinjia He This is my paper · ORCID
classification cs.SEcs.AI
keywords PR-IssuemisalignmentSWE-benchbenchmarkqualitymulti-agentLLMsystemssoftwareengineeringevaluationpullrequestissueresolutiontaxonomy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SWE-bench-style benchmarks assume each pull request cleanly fixes its linked issue. The authors manually audit all 500 SWE-bench Verified instances and find 68 (13.6%) violate that assumption, in five patterns from scope creep and defective fixes to underspecified issues. To make such audits scalable, they build PAIChecker, a three-phase multi-agent system that first detects specific patterns, then synthesizes cross-agent evidence, then validates against code. On two other SWE-bench-like datasets it reaches 92.12% and 91.67% binary accuracy, beating single-prompt and single-agent baselines on every tested LLM. If the finding holds, widely used leaderboards are partly measuring misalignment noise, and benchmark builders can filter it out.

What carries the argument

PAIChecker is a three-phase multi-agent framework operating on a text-driven, code-validation principle. Phase I runs three specialized subagents—an Issue Analyzer for incomplete or redefined specifications, a PR Scope Analyzer for scope creep and unspecified literals, and a PR Connection Analyzer for defective and follow-up PRs. Phase II's coordinator agent synthesizes suspicious clues from all three, assigns an Others label for beyond-taxonomy misalignment, and drops labels whose textual evidence is weak. Phase III's code validator inspects the actual code diff and surrounding files and vetoes labels unsupported by implementation. The separation of authority—only Phase I adds predefined la

What would settle it

Independently re-annotate all 500 SWE-bench Verified instances with a pre-registered coding scheme and a different annotation team. If the misaligned fraction falls below roughly 10% or inter-rater agreement on the five-pattern taxonomy is low, the prevalence claim and downstream leaderboard re-ranking numbers would not reproduce.

Watch

Extended reading notes

Core claim

The paper's central claim is that PR-Issue misalignment is a systematic construction-level defect in SWE-bench-like benchmarks, not a rare exception. Manual open coding of all 500 SWE-bench Verified instances yields five misalignment patterns—PR Scope Creep, Defective PR, Incomplete Specification, Follow-up PR, and Unspecified Literal—covering 13.6% of instances, with 41.2% of never-resolved instances misaligned. The paper further claims PAIChecker detects and categorizes these misalignments better than existing prompting and agent baselines, because it decomposes the task into focused subagents and gives a coordinator and a code validator veto power over predefined labels, including an Othe

Load-bearing premise

The manually produced ground-truth labels are correct and unbiased: the 13.6% prevalence figure and all accuracy numbers are measured against the authors' own open-coding annotation, with only a small live-GitHub check and partial comparison against another labeling pipeline as external confirmation.

Editorial extensions

If this is right

  • Excluding the 68 misaligned instances changes the rank of 64.1% of 131 leaderboard agents, so published SWE-bench leaderboards overstate some models and understate others.
  • Benchmark builders can run PAIChecker before release to filter or relabel misaligned PR-issue pairs.
  • Training datasets built by the same pipeline inherit the same misalignment, so models may learn defective or unrelated patches; auditing them should improve training signal.
  • Because PAIChecker beats baselines across four LLM backbones, alignment checking can be automated cheaply enough for routine benchmark curation.
  • The monotonic drop in misalignment rate with resolution count implies unsolved hard instances are more likely to be broken tasks, not merely hard tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the 13.6% figure from human-curated SWE-bench Verified likely underestimates misalignment in less curated SWE-bench-like benchmarks; the authors' own numbers (44.3% on SWE-Gym, 24.3% on Multilingual) point that way.
  • Editorial inference: the taxonomy's boundary between misaligned and aligned depends on treating maintainer clarification requests as authoritative, so applying PAIChecker to projects with weak maintainer response would require a different grounding standard.
  • Editorial inference: a testable extension is to use PAIChecker as a filtering step for training data and measure whether models trained on filtered data resolve held-out issues more often.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper investigates PR-Issue misalignment in SWE-bench-like benchmarks. It manually analyzes all 500 SWE-bench Verified instances, identifies five misalignment patterns (SC, DP, IS, FP, UL) with 11 scenarios, and reports a 13.6% misalignment prevalence. It also correlates misalignment with agent resolution rates and shows that excluding misaligned instances changes 64.1% of agent rankings. The authors then propose PAIChecker, a three-phase multi-agent framework (three specialized subagents, a coordinator, and a code validator) following a text-driven, code-validation principle. Evaluation on SWE-Gym and SWE-bench Multilingual with four LLM backbones reports up to 92.12% and 91.67% binary accuracy and 84.66% and 90.67% exact match, outperforming prompting and agent baselines. Ablations and label-change analyses support the contribution of each phase.

Significance. If the results hold, the paper makes a meaningful contribution: it provides a systematic taxonomy of a subtle benchmark-quality issue, quantifies its prevalence in a widely used benchmark, and offers a practical automated tool. The strengths are notable: the manual study is grounded in open-coding practices, the evaluation spans two benchmark families and four LLMs, the data are released, and the authors include a live GitHub validation exercise (though with a small response rate) and a direct comparison with SPICE. These elements raise the bar relative to purely prompt-based analyses. However, the central quantitative claims—the 13.6% prevalence and the detection accuracies—rest on ground-truth labels produced by the authors themselves using the taxonomy that also drives PAIChecker. Until that measurement pipeline is independently validated, the headline numbers must be treated as conditional.

major comments (5)
  1. [Section 2.1.2 / Finding in Section 2.2.1] The manual coding of all 500 SWE-bench Verified instances is the foundation for the 13.6% prevalence claim and the RQ2 analyses. Yet Section 2.1.2 reports no inter-rater reliability statistic and describes only a review by the second author with disagreements resolved by a third. By contrast, Section 4.1 reports Cohen's Kappa of 0.91/0.86 for the evaluation datasets. Given that boundary decisions (e.g., IS triggering on a maintainer clarification request, SC on whether PR claims exceed issue scope) are judgment-heavy, the absence of a dual-coding or independent re-annotation protocol for the Section 2 study is a load-bearing gap. Please add IRR or an independent validation subset, and report agreement separately for the five patterns.
  2. [Section 4.1 / Table 2; circularity] The ground-truth labels for SWE-Gym and SWE-bench Multilingual were produced by the same authors using the same taxonomy that defines PAIChecker's prompts. This creates a circularity risk: the reported accuracies partly measure agreement with the authors' own annotation criteria rather than an external truth. The paper acknowledges annotation subjectivity in Section 7 but does not provide an independent gold standard for the 2,738 evaluation instances. The SPICE overlap (644 shared instances, 281 vs 143 flags, 78 in common) and the 17 live GitHub confirmations are suggestive but limited and potentially response-biased. Please provide an external validation, e.g., re-annotation by independent annotators on a random subset, or a maintainer-feedback study on a random sample rather than only self-selected reports.
  3. [Tables 3 and 4; small class sizes] Several pattern counts are extremely small: Table 1 lists FP-2 n=1 and UL-1 n=1; Table 4 omits per-label F1 for FP (n=2) and Others (n=1) on Multilingual, correctly noting they are insufficient for meaningful comparison. However, the headline binary accuracy and EM on Multilingual are still reported without confidence intervals, and the accuracy metric is dominated by the aligned class (75.7% of instances). The paper should report confidence intervals or bootstrap estimates for the central numbers, and explicitly caution against interpreting per-pattern F1s for low-prevalence categories. A balanced accuracy or AUROC would additionally clarify performance on the minority class.
  4. [Section 5.2 / Table 5] The ablation study is run only on Gemini-3.1-Pro-Preview and only on SWE-Gym (Table 5), while the phase-wise progression analysis covers all backbones but only on SWE-Gym (Figure 6). The paper claims general effectiveness across four LLM backbones in RQ3, but the component-contribution claims are not tested across backbones or on SWE-bench Multilingual. Given that the relative contributions of the three subagents likely depend on the model's text and code reasoning strengths, a single-backbone ablation is insufficient to support the generality of the component analysis. Please either extend the ablations or temper the corresponding claims.
  5. [Section 5.1.1 / Table 3; significance testing] The footnote to Table 3 states that all differences are significant at p<0.001, but no significance test is described in Section 4.3 or elsewhere. It is unclear whether a McNemar test, bootstrap, or other procedure was used, whether the test was applied to all pairwise comparisons, and how multiple comparisons were handled. Please specify the statistical methodology; otherwise the significance claim is unverifiable.
minor comments (7)
  1. [Abstract / Section 1] The name 'PAIChecker' is capitalized inconsistently ('PAIchecker' appears in the abstract). Also, 'PR–Issue' and 'PR-Issue' are used interchangeably; please standardize.
  2. [Figure 1 and Section 2.2.1] Figure 1 is difficult to read: the example boxes are repeated across subfigures, and the labels (a)-(d) do not always match the citations in the text (e.g., the text refers to 'django__django12125' without an underscore in Section 2.2.1). Please clean up the figure and verify all instance identifiers.
  3. [Section 3.1.1] The IS definition says a well-formed bug report comprises reproduction code, actual behavior, and expected behavior, but IS-2 'Addresses new problem from later discussion' is more about scope change than missing components. Consider clarifying how the three-component rule maps to IS-2, since the current wording conflates incompleteness with redefinition.
  4. [Section 4.2] The baselines Claude Code and Codex are described as 'with their officially supported models,' but the exact model versions are not stated. For reproducibility, please list the specific model identifiers used for these baselines.
  5. [Section 6] The live GitHub validation is a nice external check, but the response rate (17/200) is low and possibly biased. Please state the response rate explicitly and discuss how non-response might affect the 94% confirmation figure.
  6. [References] Reference [10] is labeled as the Sphinx issue template in the URL but appears to point to the Sphinx issue template rather than the SymPy one, and the caption text in Section 2.2.1 mentions 'python__mypy-11632' but the citation style is inconsistent. Please check references for accuracy.
  7. [Table 2] The 'Count' column for Multilingual pattern rows does not sum to the total misaligned count because instances can carry multiple labels; this is stated in the table caption, but it may still confuse readers. Consider adding a footnote that the sum of pattern counts exceeds the misaligned total.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the detection pipeline is evaluated on held-out annotations and is not fitted to its own outputs; the main validity concern is annotation subjectivity, which the paper explicitly acknowledges.

full rationale

PAIChecker's claimed derivation chain is not circular in the sense defined here. The taxonomy (Section 2.2.1) is produced by open coding of SWE-bench Verified, and PAIChecker's prompts operationalize that taxonomy, but the evaluation on SWE-Gym and SWE-bench Multilingual (Section 4.1) uses held-out instances labeled by two annotators following the taxonomy, with Cohen's kappa of 0.91/0.86 and third-author arbitration. PAIChecker is not trained or fitted on these labels; its accuracy and exact-match numbers are genuine, falsifiable measurements of whether the prompted multi-agent pipeline reproduces the annotation protocol on new data. The 13.6% prevalence figure is the output of the manual coding process, not a quantity derived from PAIChecker itself, and thus cannot be circular with respect to the detector. The paper explicitly identifies the main threat: 'Internal Validity. The primary threat is annotation subjectivity' (Section 7). This is a measurement-validity limitation, not a derivation-level circularity: if the annotators were systematically biased, the reported numbers would be wrong, but they are not forced by construction. The paper also offers partial external grounding: 16 of 17 live GitHub maintainer confirmations and a partial overlap with SPICE on 644 shared instances (78 in common). No load-bearing self-citation chain, imported uniqueness theorem, or fitted-parameter-renamed-as-prediction appears; references to the authors' other work are contextual only. Accordingly, the central claims are self-contained as an empirical study and an applied LLM-agent evaluation, with the caveat that the gold standard is the authors' own annotation criteria.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claims rest on the authors' taxonomy and annotations, not on fitted parameters or invented entities. The main axioms are domain assumptions about textual artifacts and maintainer behavior.

assumptions (3)
  • domain assumption PR-side textual artifacts summarize changes at a level of abstraction similar to issue descriptions, making textual alignment comparison more reliable than direct code comparison.
    Section 1 justifies the text-driven, code-validation design on this asymmetry; if false, the entire pipeline's emphasis on textual evidence is unjustified.
  • domain assumption A maintainer's explicit request for missing information is a valid and sufficient signal that an issue is incompletely specified.
    Section 2.2.1 defines Incomplete Specification using maintainer requests; this is a design choice, not an empirically established rule.
  • domain assumption The five-pattern taxonomy plus 'Others' is sufficient to describe meaningful PR-Issue misalignments.
    Section 2.2.1 derives the taxonomy by open coding; the completeness of the taxonomy is assumed when computing prevalence and when evaluating the checker's label accuracy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks." pith.science (2026). https://pith.science/paper/MULX42CS

@misc{pith2026260728587,
  author       = {Pith},
  title        = {Pith review of: PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MULX42CS}},
  note         = {Machine review of arXiv:2607.28587}
}
read the original abstract

SWE-bench-like benchmarks are widely used for evaluating LLM's issue resolution capability. They typically follow a common construction pipeline: each PR (Pull Request) is paired with its linked issue by extracting issue references from the PR description; the issue description is used as the problem statement, and the PR patch serves as the test oracle. However, due to the inherent complexity of developing and maintaining large repositories, such PR-Issue pairings are often misaligned in practice. In this work, we systematically study SWE-bench Verified instances, finding that 13.6% exhibit misalignment across five patterns in eleven fine-grained scenarios. To enable reliable and scalable construction of those benchmarks in the future, we propose PAIChecker, a multi-agent system for checking PR-Issue misalignment in SWE-bench-like benchmarks. Specifically, PAIChecker adopts a three-phase design that combines specific pattern identification, cross-agent label synthesis, and code-level validation, thereby enabling more accurate, generalizable, and progressively verified detection. Experiments on SWE-Gym and SWE-bench Multilingual show that PAIchecker achieves the best performance across all four LLM backbones, reaching up to 92.12% and 91.67% binary accuracy, respectively.

Figures

Figures reproduced from arXiv: 2607.28587 by the authors.

Figure 1
Figure 1. Examples of four misalignment patterns [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Per-agent rank change after excluding 68 mis [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 2
Figure 2. Misalignment rate decreases monotonically as in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Motivating Example iterative__dvc-4872: PR #4872 is accidentally linked to issue #4796 but actually fixes issue #4504. Backbone: Gemini-3.1-Pro Preview. Mini-SWE-Agent and CoT force-classify as SC; PaiChecker’s coordinator correctly drops SC and assigns Others by synth…
Figure 5
Figure 5. Figure 5: Workflow of PaiChecker. Phase I performs specific misalignment identification through three specialized subagents. Phase II synthesizes their outputs into a preliminary label and rationale. Phase III validates the preliminary result against code-level evidence and supp…
Figure 6
Figure 6. Figure 6: Binary Accuracy and Exact Match across pipeline [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 6
Figure 6. Figure 6: Binary Accuracy and Exact Match across pipeline [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 27 canonical work pages

  1. [1]

    SWE-bench Annotation Instructions

    2024. SWE-bench Annotation Instructions. https://cdn.openai.com/introducing- swe-bench-verified/swe-b-annotation-instructions.pdf accessed:2026-03-06

    work page 2024

  2. [2]

    SWE-bench Verified Official Experiments Result

    2024. SWE-bench Verified Official Experiments Result. https://github.com/SWE- bench/experiments accessed:2026-03-06

    work page 2024

  3. [3]

    SWE-bench Verified Official Leaderboards

    2024. SWE-bench Verified Official Leaderboards. https://www.swebench.com/ index.html accessed:2026-03-06

    work page 2024

  4. [4]

    Issue Template of Astropy

    2025. Issue Template of Astropy. https://github.com/astropy/astropy/tree/main/ .github/ISSUE_TEMPLATE Accessed: 2026-03-25

    work page 2025

  5. [5]

    Issue Template of Pytest

    2025. Issue Template of Pytest. https://github.com/pytest-dev/pytest/tree/main/ .github/ISSUE_TEMPLATE Accessed: 2026-03-25

    work page 2025

  6. [6]

    Issue Template of Sphinx

    2025. Issue Template of Sphinx. https://github.com/sphinx-doc/sphinx/tree/ master/.github/ISSUE_TEMPLATE Accessed: 2026-03-25

    work page 2025

  7. [7]

    Issue Template of Xarray

    2025. Issue Template of Xarray. https://github.com/pydata/xarray/tree/main/ .github/ISSUE_TEMPLATE Accessed: 2026-03-25

    work page 2025

  8. [8]

    Pull Request Template of Astropy

    2025. Pull Request Template of Astropy. https://github.com/astropy/astropy/ blob/main/.github/PULL_REQUEST_TEMPLATE.md Accessed: 2026-03-25

    work page 2025

Show all 57 references
  1. [9]

    Pull Request Template of Scikit-learn

    2025. Pull Request Template of Scikit-learn. https://github.com/scikit-learn/ scikit-learn/blob/main/.github/PULL_REQUEST_TEMPLATE.md Accessed: 2026- 03-25

    2025

  2. [10]

    Pull Request Template of Sympy

    2025. Pull Request Template of Sympy. https://github.com/sphinx-doc/sphinx/ tree/master/.github/ISSUE_TEMPLATE Accessed: 2026-03-25

    2025

  3. [11]

    Claude Code

    2026. Claude Code. https://claude.com/product/claude-code accessed:2026-03- 06

    2026

  4. [12]

    2026. Codex. https://openai.com/codex/ accessed:2026-03-06

    2026

  5. [13]

    https://github.com/manyiResearch/PAIChecker

    2026.PAIChecker Github. https://github.com/manyiResearch/PAIChecker

    2026

  6. [14]

    Why SWE-bench Verified no longer measures frontier coding capa- bilities

    2026. Why SWE-bench Verified no longer measures frontier coding capa- bilities. https://openai.com/index/why-we-no-longer-evaluate-swe-bench- verified/ accessed:2026-03-06

    2026

  7. [15]

    Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, and Song Wang. 2024. SWE-Bench+: Enhanced Coding Benchmark for LLMs. doi:10.48550/arXiv.2410.06992

    2024 doi arXiv

  8. [16]

    Anthropic. 2025. Claude Sonnet 4.6. https://www.anthropic.com/news/claude- sonnet-4-6 Accessed: 2026-03-25

    2025

  9. [17]

    Ibragim Badertdinov, Alexander Golubev, Maksim Nekrashevich, Anton Shevtsov, Simon Karasik, Andrei Andriushchenko, Maria Trofimova, Daria Litvintseva, and Boris Yangel. 2025. Swe-rebench: An automated pipeline for task collection and decontaminated evaluation of software engin...

    2025

  10. [18]

    Shraddha Barke, Michael B James, and Nadia Polikarpova. 2023. Grounded copilot: How programmers interact with code-generating models.Proceedings of the ACM on Programming Languages7, OOPSLA1 (2023), 85–111

    2023

  11. [19]

    Nicolas Bettenburg, Sascha Just, Adrian Schröter, Cathrin Weiss, Rahul Premraj, and Thomas Zimmermann. 2008. What makes a good bug report?. InProceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engineering (SIGSOFT ’08/FSE-16). Association fo...

    2008

  12. [20]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffr...

    2020

  13. [21]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott G...

    2021 arXiv

  14. [22]

    Mouxiang Chen, Lei Zhang, Yunlong Feng, Xuwu Wang, Wenting Zhao, Ruisheng Cao, Jiaxi Yang, Jiawei Chen, Mingze Li, Zeyao Ma, Hao Ge, Zongmeng Zhang, Zeyu Cui, Dayiheng Liu, Jingren Zhou, Jianling Sun, Junyang Lin, and Binyuan Hui. 2026. SWE-Universe: Scale Real-World Verifiabl...

    2026 doi

  15. [23]

    Jimenez, John Yang, Leyton Ho, Tejal Patwardhan, Kevin Liu, and Aleksander Madry

    Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn, Giulio Starace, Evan Mays, Rachel Dias, Marwan Aljubeh, Mia Glaese, Carlos E. Jimenez, John Yang, Leyton Ho, Tejal Patwardhan, Kevin Liu, and Aleksander Madry. 2024. Introducing SWE-bench Verified. https:...

    2024

  16. [24]

    Mariana Coutinho, Lorena Marques, Anderson Santos, Marcio Dahia, Cesar França, and Ronnie de Souza Santos. 2024. The role of generative ai in software development productivity: A pilot case study. InProceedings of the 1st ACM International Conference on AI-Powered Software. 131–138

    2024

  17. [25]

    Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, et al. 2025. Swe-bench pro: Can ai agents solve long-horizon software engineering tasks?arXiv preprint arXiv:2509.16941(2025)

    2025 arXiv

  18. [26]

    Google. 2025. Gemini 3.1 Pro. https://blog.google/innovation-and-ai/models- and-research/gemini-models/gemini-3-1-pro/ Accessed: 2026-03-25

    2025

  19. [27]

    Xinyi He, Qian Liu, Mingzhe Du, Lin Yan, Zhijie Fan, Yiming Huang, Zejian Yuan, and Zejun Ma. 2025. Swe-perf: Can language models optimize code performance on real-world repositories?arXiv preprint arXiv:2507.12415(2025)

    2025 arXiv

  20. [28]

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Live- CodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. arXiv:2403.07974 [cs.SE] https://arxiv.org/...

    2024 arXiv

  21. [29]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world Github Issues?. InThe Twelfth International Conference on Learning Representations. https://openreview.net/forum?...

    2024

  22. [31]

    OpenAI. 2025. Introducing GPT-5.3 Codex. https://openai.com/index/ introducing-gpt-5-3-codex/ Accessed: 2026-03-25

    2025

  23. [32]

    Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. 2024. Training software engineering agents and verifiers with swe-gym.arXiv preprint arXiv:2412.21139(2024)

    2024 arXiv

  24. [33]

    Jorge Pérez, Jessica Díaz, Javier Garcia-Martin, and Bernardo Tabuenca. 2020. Systematic literature reviews in software engineering—enhancement of the study selection process using Cohen’s Kappa statistic.Journal of Systems and Software 168 (2020), 110657

    2020

  25. [34]

    Qwen Team. 2025. Qwen 3.5. https://qwen.ai/blog?id=qwen3.5 Accessed: 2026-03-25

    2025

  26. [35]

    Muhammad Shihab Rashid, Christian Bock, Yuan Zhuang, Alexander Buch- holz, Tim Esler, Simon Valentin, Luca Franceschi, Martin Wistuba, Prabhu Teja Sivaprasad, Woo Jung Kim, et al . 2025. Swe-polybench: A multi-language benchmark for repository level evaluation of coding agents...

    2025 arXiv

  27. [36]

    Klaas-Jan Stol, Paul Ralph, and Brian Fitzgerald. 2016. Grounded theory in software engineering research: a critical review and guidelines. InProceedings of the 38th International conference on software engineering. 120–131

    2016

  28. [37]

    Chaofan Tao, Jierun Chen, Yuxin Jiang, Kaiqi Kou, Shaowei Wang, Ruoyu Wang, Xiaohui Li, Sidi Yang, Yiming Du, Jianbo Dai, Zhiming Mao, Xinyu Wang, Lifeng Shang, and Haoli Bai. 2026. SWE-Lego: Pushing the Limits of Super- vised Fine-tuning for Software Issue Resolving. doi:10.4...

    2026 doi

  29. [38]

    Trae Research Team, Pengfei Gao, Zhao Tian, Xiangxin Meng, Xinchen Wang, Ruida Hu, Yuanan Xiao, Yizhou Liu, Zhao Zhang, Junjie Chen, Cuiyun Gao, Yun Lin, Yingfei Xiong, Chao Peng, and Xia Liu. 2025. Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scaling...

    2025 doi

  30. [39]

    Minh V. T. Thai, Tue Le, Dung Nguyen Manh, Huy Phan Nhat, and Nghi D. Q. Bui. 2025. SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios. doi:10.48550/arXiv.2512.18470

    2025 doi arXiv

  31. [40]

    Lilin Wang, Lucas Ramalho, Alan Celestino, Phuc Anthony Pham, Yu Liu, Umang Kumar Sinha, Andres Portillo, Onassis Osunwa, and Gabriel Maduekwe

  32. [41]

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. 2025. Openhands: An open platform for ai software developers as generalist agents. InInternational Conference on Learning Representations, Vol...

    2025

  33. [42]

    Solved Issues

    You Wang, Michael Pradel, and Zhongxin Liu. 2025. Are "Solved Issues" in SWE- bench Really Solved Correctly? An Empirical Study. icse 2026. http://arxiv.org/ abs/2503.15223 PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks ASE ’26, October ...

    2025 arXiv

  34. [43]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv:2201.11903 [cs.CL] https: //arxiv.org/abs/2201.11903

    2023 arXiv

  35. [44]

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2024. Agent- less: Demystifying LLM-based Software Engineering Agents. doi:10.48550/ARXIV. 2407.01489

    2024 doi arXiv

  36. [45]

    Chengxing Xie, Bowen Li, Chang Gao, He Du, Wai Lam, Difan Zou, and Kai Chen

  37. [46]

    Junjielong Xu, Ying Fu, Shin Hwei Tan, and Pinjia He. 2025. Aligning the Objective of LLM-based Program Repair. Inicse 2025. icse 2025. doi:10.48550/arXiv.2404. 08877 arXiv:2404.08877 [cs]

    2025 doi arXiv

  38. [47]

    InFindings of the Association for Computational Linguistics: ACL 2025, Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.)

    SWE-Fixer: Training Open-Source LLMs for Effective and Efficient GitHub Issue Resolution. InFindings of the Association for Computational Linguistics: ACL 2025, Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Li...

    2025

  39. [48]

    John Yang, Kilian Lieret, Carlos E Jimenez, Alexander Wettig, Kabir Khand- pur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang

  40. [49]

    Zonghan Yang, Shengjie Wang, Kelin Fu, Wenyang He, Weimin Xiong, Yibo Liu, Yibo Miao, Bofei Gao, Yejie Wang, Yingwei Ma, Yanhao Li, Yue Liu, Zhenxing Hu, Kaitai Zhang, Shuyi Wang, Huarong Chen, Flood Sung, Yang Liu, Yang Gao, Zhilin Yang, and Tianyu Liu. 2025. Kimi-Dev: Agentl...

    2025 doi

  41. [50]

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. 2024. SWE-agent: Agent-Computer Inter- faces Enable Automated Software Engineering. InThe Thirty-eighth Annual Con- ference on Neural Information Processing Systems....

    2024 arXiv

  42. [51]

    Boxi Yu, Yuxuan Zhu, Pinjia He, and Daniel Kang. 2025. UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench. arXiv:2506.09289 [cs.SE] https: //arxiv.org/abs/2506.09289

    2025 arXiv

  43. [52]

    Swe-smith: Scaling data for software engineering agents.arXiv preprint arXiv:2504.21798(2025)

    2025 arXiv

  44. [53]

    Lei Zhang, Jiaxi Yang, Min Yang, Jian Yang, Mouxiang Chen, Jiajun Zhang, Zeyu Cui, Binyuan Hui, and Junyang Lin. [n. d.]. SWE-Flow: Synthesizing Software Engineering Data in a Test-Driven Manner. ([n. d.])

  45. [54]

    Boxi Yu, Yang Cao, Yuzhong Zhang, Liting Lin, Junjielong Xu, Zhiqing Zhong, Qinghua Xu, Guancheng Wang, Jialun Cao, Shing-Chi Cheung, Pinjia He, and Lionel Briand. 2026. SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmark. arXiv:...

    2026

  46. [56]

    Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, et al. 2025. Swe-bench goes live!arXiv preprint arXiv:2505.23419(2025)

    2025 arXiv

  47. [58]

    Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. Au- toCodeRover: Autonomous Program Improvement. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, Vienna Austria, 1592–1604. doi:10.1145/3650212.3680384 Re...

    2024

  48. [1139]

    doi:10.18653/v1/2025.findings-acl.62

    2025 doi

  49. [2025]

    doi:10.48550/arXiv

    SWE-Bench++: A Framework for the Scalable Generation of Software Engineering Benchmarks from Open-Source Repositories. doi:10.48550/arXiv. 2512.17419

    doi

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.