Pith. sign in

REVIEW 3 major objections 8 minor 40 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

Bug reports written for humans fail AI repair agents

2026-07-09 05:42 UTC pith:2MTRMF3Y

load-bearing objection Solid two-study design comparing bug-report feature importance for AI agents vs. humans; one internal inconsistency the authors don't reconcile. the 3 major comments →

arxiv 2607.07593 v1 pith:2MTRMF3Y submitted 2026-07-08 cs.SE

What Makes a Good Bug Report for an AI Agent?

Lara Khatib , Noble Saji Mathews , Meiyappan Nagappan , Pengyu Nie , Thomas Zimmermann This is my paper
classification cs.SE
keywords automated program repairbug report qualityLLM agentsfault localizationSWE-benchsoftware engineeringmixed-effects regressioncontrolled ablation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the bug-report conventions optimized for human developers do not transfer cleanly to LLM-based automated program repair agents. Through an observational study of 87 agents across 433 real GitHub issues and a controlled ablation across 2 models and 17 report mutations, the authors find that agents benefit most from concrete, executable, and well-localized information: fix suggestions, reproduction scripts, repository source code, and file-level fault localization cues. Conversely, qualities long emphasized for human readers — natural-language steps to reproduce, readability, and shorter technical prose — show no independent benefit or even correlate with lower success rates. Longer reports are associated with roughly half the odds of resolution. Structural formatting (section headers, numbered lists) matters independently of content: removing headers or flattening lists into prose reduces solve rates even when no information is deleted, because agents confuse observed and expected behavior or implement only part of the specification. The two tested models diverge in how they handle missing information: Qwen searches more broadly and can exhaust its turn budget, while Gemma commits to a plausible interpretation early and patches on it. The central mechanism the paper identifies is that agents, unlike human developers, cannot ask follow-up questions; whatever the report omits, the agent must infer or do without, and any inference it acts on becomes a committed guess that shapes the rest of the repair.

Core claim

The paper's central discovery is that the features helping AI repair agents succeed are not the features long identified as helping human developers. Fix suggestions (odds ratio 3.61), repository source code (2.82), executable reproduction scripts (2.52), and file-level localization (2.33) are the strongest positive predictors of resolution, while report length carries a negative association (odds ratio 0.49 per standard deviation increase). Controlled ablation confirms causality for several of these: removing file references drops solve rates by 29–40 percentage points, removing expected behavior drops them by 35–42 percentage points, and purely structural changes — flattening lists orstriw

What carries the argument

The argument is carried by two complementary mechanisms. The first is a mixed-effects logistic regression model with crossed random intercepts for instance and agent, which isolates the association between 27 bug-report features and resolution probability while controlling for bug difficulty and agent capability. The second is a controlled ablation design that holds the underlying task fixed while mutating the problem statement — removing components, deleting localization cues, stripping sections, and reformatting structure — so that each change in solve rate can be attributed to the specific information or formatting removed. Together, the observational model supplies ecological validity (t

Load-bearing premise

The ablation results rest on two open-weight models running through a minimal bash-loop harness, and the paper acknowledges that findings may differ for proprietary models or more complex agent scaffolds. A 10-instance check with one proprietary model is too small to establish that the structural and content-mutation findings generalize across model families, scaffolding architectures, or the proprietary systems that dominate real-world deployments.

What would settle it

If a proprietary model with a sophisticated agent harness (tool-calling, multi-turn planning, clarification-seeking) showed no solve-rate drop when file references, section headers, or expected behavior were removed from bug reports, the paper's central claim that agents need concrete, well-localized, well-structured information would not generalize beyond the minimal-harness, open-weight setting tested.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Bug-tracking templates and reporting guidelines may need to be redesigned with a separate 'agent-readable' section emphasizing file paths, executable reproduction scripts, and fix suggestions rather than prose-oriented steps to reproduce.
  • Report length acting as a negative predictor suggests that verbose natural-language context may actively mislead agents by injecting noise into the context window, which has implications for how much context to include in agent prompts generally.
  • The finding that structural formatting matters independently of content implies that LLM-based agents are sensitive to document layout as a parsing signal, not just to the text itself — a property that could be exploited or broken by report-generation tools.
  • The divergence between Qwen and Gemma in handling missing information suggests that optimal report content may be model-dependent, complicating the goal of a single universal bug-report standard for agents.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If structural formatting matters independently of content, then auto-generated or templated reports that standardize headers and lists could outperform free-form human-written reports for agent consumption, even when the content is identical.
  • The negative association between report length and success, combined with the positive association for harder-to-read prose (higher SMOG), suggests that agents benefit from dense technical content but are harmed by verbose natural-language explanation — implying a trade-off between human-friendly and agent-friendly report styles that may not be bridgeable by a single document.
  • The model-specific divergence in handling missing information raises the possibility that report-completeness requirements could serve as a diagnostic for agent robustness: models that fail more gracefully when information is removed may be better suited for real-world deployment where reports are inevitably incomplete.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. This paper investigates which bug-report features are associated with successful resolution by AI repair agents, combining (1) an observational mixed-effects regression over 433 SWE-bench Verified issues attempted by 87 agents and (2) a controlled ablation over 2 open-weight models and 17 problem-statement mutations on SWE-bench Pro. The central claim is that a good bug report for an agent overlaps with but is not identical to a good report for a human: agents benefit most from concrete, executable, and well-localized information (fix suggestions, reproduction scripts, repository code, file-level localization), whereas some qualities emphasized for human readers (natural-language steps to reproduce, readability, shorter technical prose) contribute little or even correlate with lower success. The mixed-effects model is well-specified with crossed random intercepts, VIF-based collinearity checks, and prevalence-based feature exclusion. The ablation design is careful, with eligible-set construction, 3-run repetition, and matched baselines. Replication materials are publicly available.

Significance. The paper addresses a timely and practically important question: as AI repair agents enter developer workflows, bug-report templates optimized for human readers may not serve agents equally well. The two-study design (observational + causal ablation) is a genuine strength, providing both ecological validity and controlled causal evidence. The finding that structural formatting changes alone (flattening lists, removing headers) can reduce solve rates without any content removal is novel and actionable. The comparison with prior human-centric findings (Table IV) is well-constructed and frames the contribution clearly. The public replication package and pre-registered-style ablation protocol add credibility. The paper ships falsifiable predictions (specific feature effects that can be tested on other models/benchmarks) and a parameter-free observational model (no fitted constants carried into the ablation design).

major comments (3)
  1. §VI.A and Table IV: The headline claim that natural-language steps to reproduce 'contribute little' for agents (OR=0.83, n.s. in Table II) is directly contradicted by the Study 2 ablation: removing steps to reproduce from report content causes drops of 38.5 pp (Qwen, n=13) and 25.0 pp (Gemma, n=12) in Table III — among the largest content-mutation drops. Table IV reports both findings side by side but the paper never reconciles them. The most likely explanation is residual confounding in the observational model (reports with NL steps may also be longer, OR=0.49***, or lack executable scripts), but the paper should explicitly acknowledge this tension rather than presenting the observational null as the characterization and the ablation result as a separate finding. Since the NL-steps-vs-scripts contrast is the paper's sharpest example of agent-human divergence, this unreconciled internal
  2. §V.F–V.H, Table III: The content-mutation conditions (RC−Steps to Reproduce, RC−Observed Behavior, etc.) are evaluated on very small eligible sets (n=8–13 for Qwen, n=12–26 for Gemma). With only 3 runs per instance, the solve@3 metric on 8–13 instances has substantial variance. The paper reports point estimates (e.g., 61.5% for Qwen RC−Steps to Reproduce) without any uncertainty quantification. Given that these small-n results are used to support the claim that 'no single part of [report content] is safely removable' (§V.H), confidence intervals or at least an explicit acknowledgment of the statistical power limitation would strengthen the claim. The structural and spec-composition mutations (n=79/83) are more robustly powered.
  3. §VII, External Validity: The generalizability of the ablation results rests on two open-weight models running through a minimal bash-loop harness (mini-SWE-agent). The 10-instance Claude Sonnet 4.6 check is too small to establish that findings generalize across model families, scaffolding architectures, or proprietary systems. The paper acknowledges this, but the Discussion (§VI.B) presents prescriptive recommendations ('Point the agent to the fix location,' 'Give the agent an executable reproduction script,' 'Keep the report structured') without hedging. These recommendations should be framed as hypothesis-generating for broader model/harness configurations rather than general guidelines, or the validation set should be expanded.
minor comments (8)
  1. §IV.E: The VIF threshold of 5 and prevalence range of [10%, 90%] are stated without justification. A brief citation or rationale for these specific cutoffs would help readers assess sensitivity to them.
  2. §IV.F: The readability finding (higher SMOG → higher success, OR=1.55**) is counterintuitive and the paper attributes it to 'more technical detail.' This is plausible but speculative. A brief check of whether high-SMOG reports correlate with presence of technical features (stack traces, code snippets) would support the interpretation.
  3. Table I: 'has itemization' has prevalence 10.6%, which is just outside the [10%, 90%] exclusion range, yet it is retained in the model. The boundary case should be noted.
  4. §V.C: The decomposition of SWE-bench Pro instances into five components is clear, but a brief example showing how a real instance maps to these components would aid reproducibility.
  5. §V.H: The Jaccard similarity (0.26–0.34) for unsolved bugs across spec-composition deletions is mentioned in prose but not tabulated. A small table or figure would strengthen the point that different deletions lose different instances.
  6. Table III: The 'Only Interface' row has n=35/36, notably smaller than the other spec-composition conditions (n=79/83). This is because the interface is optional. The paper notes this in §V.F, but a footnote on the table itself would improve readability.
  7. §VI.B, 'Give the agent an executable reproduction script': This recommendation is supported by Study 1 (OR=2.52**) but not directly tested in Study 2 ('Not tested' in Table IV). The recommendation should note that the causal evidence is from the observational study only.
  8. The abstract states '17 problem-statement mutations' but Table III lists more conditions when counting both models. Clarify whether 17 refers to distinct mutation types or model-mutation pairs.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive review. The referee raises three major points: (1) an unreconciled tension between the observational null for NL steps-to-reproduce and the ablation drops when steps are removed, (2) absence of uncertainty quantification for small-n content-mutation conditions, and (3) insufficient hedging on prescriptive recommendations given limited external validation. We agree with all three points and will revise accordingly.

read point-by-point responses

  1. Referee: §VI.A and Table IV: The headline claim that NL steps to reproduce 'contribute little' for agents (OR=0.83, n.s.) is contradicted by the Study 2 ablation showing 25–38.5 pp drops when steps are removed. Table IV presents both side by side without reconciling them. The paper should explicitly acknowledge this tension.

    Authors: The referee is correct that this tension exists and that we failed to reconcile it in the manuscript. We will revise to address it directly. The most likely explanation is indeed residual confounding in the observational model: reports with NL steps-to-reproduce may also be longer (OR=0.49*** for length) or may lack executable reproduction scripts, and the observational model cannot fully separate these correlated features. The ablation, by contrast, holds everything else fixed and removes only the steps, so its drop reflects the causal effect of that content in isolation. We will add an explicit paragraph in §VI.A reconciling the two findings: the observational null means that, after adjusting for co-occurring features, the marginal association of NL steps with success is not detectable; the ablation drop means that when steps are the only thing removed, agents do lose something. These are not contradictory—they answer different questions. We will also soften the headline framing in the abstract and Table IV to make clear that the agent-human divergence on this dimension is about the form of reproduction information (executable vs. prose), not about whether steps carry useful content. The claim that NL steps 'contribute little' will be revised to 'show no independent association in the observational model, though their causal removal in the ablation does reduce solve rates,' with the confounding explanation stated explicitly. revision: yes

  2. Referee: §V.F–V.H, Table III: Content-mutation conditions are evaluated on very small eligible sets (n=8–13 for Qwen, n=12–26 for Gemma) with only 3 runs per instance. Point estimates are reported without uncertainty quantification. Given these small-n results support the claim that 'no single part of [report content] is safely removable,' confidence intervals or explicit acknowledgment of statistical power limitations would strengthen the claim.

    Authors: We agree. The content-mutation conditions are underpowered for strong inferential claims, and we should not present point estimates without acknowledging the uncertainty. We will add Wilson score confidence intervals for each content-mutation solve@3 proportion in Table III, and we will add an explicit caveat in §V.H noting that the content-mutation conditions have small eligible sets (n=8–26) and that the point estimates carry wide uncertainty. We will also revise the claim 'no single part of [report content] is safely removable' to note that this is supported most robustly by the Qwen results (where all four content deletions produced drops of 35–40 pp) and that the Gemma results, while directionally consistent, are based on small samples and should be interpreted with caution. The structural and spec-composition mutations (n=79/83) are more robustly powered and we will note this distinction explicitly. revision: yes

  3. Referee: §VII, External Validity: The ablation results rest on two open-weight models through a minimal bash-loop harness. The 10-instance Claude Sonnet check is too small to establish generalization. The Discussion (§VI.B) presents prescriptive recommendations without hedging. These should be framed as hypothesis-generating rather than general guidelines, or the validation set should be expanded.

    Authors: We agree that the prescriptive recommendations in §VI.B are stated more strongly than the evidence warrants given the limited model and harness coverage. We will revise §VI.B to frame each recommendation as a hypothesis supported by our two-model, single-harness evaluation, explicitly noting that generalization to proprietary models, alternative scaffolding architectures, and other harness configurations has not been established. We will also strengthen the external validity discussion in §VII to state plainly that the 10-instance Claude Sonnet check is a sanity check, not a validation, and that broader evaluation across model families and scaffolding types is needed before these recommendations can be treated as general guidelines. We considered expanding the validation set but cannot do so within the revision timeframe due to API cost constraints; we will note this as future work. revision: yes

Circularity Check

0 steps flagged

No circularity: both studies use external data and models; no result reduces to its inputs by construction

full rationale

The paper presents two studies that are methodologically independent of each other and do not derive their conclusions from self-referential definitions or fitted constants. Study 1 fits a mixed-effects logistic regression (Eq. 1) to externally generated outcomes (87 agents on 433 SWE-bench Verified instances from the public leaderboard), annotating 27 bug-report features based on prior human-centric work (Zimmermann et al. [4], not a self-citation). The odds ratios in Table II are statistical associations estimated from this external data, not predictions that reduce to fitted inputs. Study 2 runs controlled ablations on a separate external benchmark (SWE-bench Pro [3], cited to external authors) using two external open-weight models (Qwen3.6-35B-A3B [34], Gemma-4-31B-IT [35]) through an external harness (mini-SWE-agent [32]). The eligible-set construction (retaining only solvable instances, yielding a 100% baseline by design) is a measurement design choice that the paper transparently discloses; it does not make the reported drops circular, because the drops measure how mutations affect solvability relative to that baseline on the same instances. The paper's central claim—that agents benefit from concrete, executable, well-localized information while NL steps to reproduce show no independent observational association—is supported by the regression (external data) and partially tested by the ablation (external benchmark). The apparent tension between the observational null for NL steps (OR=0.83, Table II) and the ablation drops for removing steps to reproduce (25-38.5 pp, Table III) is an issue of internal consistency and confounding, not circularity: neither result is defined in terms of the other. No self-citation chain is load-bearing for the central premise; the key prior work cited for bug-report quality (Zimmermann et al. [4]) is by a co-author (Zimmermann), but it provides background context and comparison data, not a uniqueness theorem or ansatz that forces the present paper's conclusions. The derivation is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The paper introduces no new entities, particles, forces, or postulated objects. The free parameters are experimental design choices (turn budget, cost cap, VIF threshold, prevalence range, repetition count, eligible-set threshold) rather than theory-fitted constants. The axioms are domain assumptions about benchmark validity, harness representativeness, and feature-annotation adequacy, all of which are standard for empirical SE but carry epistemic weight.

free parameters (6)
  • VIF threshold = 5
    Variance Inflation Factor cutoff for excluding collinear predictors (Section IV-E). Standard but chosen by the authors.
  • Prevalence range = [10%, 90%]
    Range for excluding near-constant binary features (Section IV-E). Features outside this range were dropped. Standard but chosen by the authors.
  • Turn budget = 50
    Maximum agent turns per run in Study 2 (Section V-E). Directly affects results, especially Qwen's budget-exhaustion failures.
  • Cost cap = $2
    Maximum cost per run in Study 2 (Section V-E). Affects which runs complete.
  • Number of repetitions = 3
    Number of runs per condition for solve@3 (Section V-E). Affects statistical power and reliability.
  • Eligible-set threshold = solve@3 >= 1
    Instances retained only if solved in at least one of three baseline runs (Section V-F). Creates 100% baseline by construction.
axioms (5)
  • domain assumption SWE-bench Verified leaderboard outcomes accurately reflect agent capability on the underlying issues
    Study 1 treats leaderboard pass/fail as ground truth. Training contamination for newer models is acknowledged but not corrected (Section VII).
  • domain assumption SWE-bench Pro is contamination-resistant due to GPL licensing
    Section V-B states all repositories are under strong copyleft licenses, making inclusion in commercial training corpora 'legally and practically unlikely.' This is an assumption, not a verified fact.
  • domain assumption mini-SWE-agent is a representative harness for evaluating report-content effects
    Section V-E justifies the choice by noting fewer tool-layer failures and competitive leaderboard performance. The harness choice affects how the agent processes report content.
  • domain assumption Binary feature presence adequately captures bug-report quality for the modeled features
    Section VII acknowledges this limitation: 'A one-line fix suggestion and a detailed root-cause analysis are treated identically under this scheme.'
  • domain assumption Bug difficulty is adequately captured by the four-level time-to-resolve rating from SWE-bench Verified
    Section IV-D uses this as a fixed-effect predictor to control for difficulty. If the rating is noisy or incomplete, residual confounding remains.

pith-pipeline@v1.1.0-glm · 21306 in / 3212 out tokens · 554849 ms · 2026-07-09T05:42:44.909550+00:00 · methodology

0 comments
read the original abstract

Automated program repair (APR) agents are transitioning from research benchmarks to developer workflows, yet they still begin with bug reports written for human developers. While decades of research have established what makes a good bug report for humans (e.g., steps to reproduce, stack traces), it remains unclear whether these features transfer to LLM-based agents. We study this question in two analyses. First, we use statistical modeling to examine associations between 27 bug-report features and repair success across 433 SWE-bench Verified issues attempted by 87 repair agents. We find that fix suggestions, reproduction scripts, repository source code, and localization info are associated with higher resolution likelihood, while longer reports are associated with lower odds. Second, we conduct controlled ablations across 2 models and 17 problem-statement mutations on SWE-bench Pro, varying the information available to an agent while holding the underlying task fixed. We remove or isolate selected bug-report content, delete fault-localization cues, and test structural changes that flatten lists or remove section headers. We find that both models depend on localization cues and expected behavior, and that structural changes alone can reduce solve rates, even without removing any content. The two models diverge in how they handle missing information: Qwen searches more widely and can exhaust its turn budget, while Gemma commits to a plausible interpretation early and patches on it. Our findings indicate that a good bug report for an agent overlaps with, but is not identical to, a good report for a human: agents benefit most from concrete, executable, and well-localized information, whereas some qualities long emphasized for human readers, such as natural language steps to reproduce and readable descriptions, contribute little or even correlate with lower success.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 40 canonical work pages · 5 internal anchors

  1. [1]

    Improving bug tracking systems,

    T. Zimmermann, R. Premraj, J. Sillito, and S. Breu, “Improving bug tracking systems,” in2009 31st International Conference on Software Engineering-Companion Volume. IEEE, 2009, pp. 247–250

    work page 2009

  2. [2]

    SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. R. Narasimhan, and O. Press, “SWE-agent: Agent-computer interfaces enable automated software engineering,” inThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. [Online]. Available: https://arxiv.org/abs/2405.15793

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  3. [3]

    SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?

    X. Deng, J. Da, E. Pan, Y . Y . He, C. Ide, K. Garg, N. Lauffer, A. Park, N. Pasari, C. Raneet al., “Swe-bench pro: Can ai agents solve long- horizon software engineering tasks?”arXiv preprint arXiv:2509.16941, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  4. [4]

    What makes a good bug report?

    T. Zimmermann, R. Premraj, N. Bettenburg, S. Just, A. Schroter, and C. Weiss, “What makes a good bug report?”IEEE Transactions on Software Engineering, vol. 36, no. 5, pp. 618–643, 2010

    work page 2010

  5. [5]

    The significance of bug report elements,

    M. Soltani, F. Hermans, and T. B ¨ack, “The significance of bug report elements,”Empirical Software Engineering, vol. 25, no. 6, pp. 5255– 5294, 2020

    work page 2020

  6. [6]

    What’s in a bug report?

    S. Davies and M. Roper, “What’s in a bug report?” inProceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, 2014, pp. 1–10

    work page 2014

  7. [7]

    Do stack traces help developers fix bugs?

    A. Schroter, A. Schr ¨oter, N. Bettenburg, and R. Premraj, “Do stack traces help developers fix bugs?” in2010 7th IEEE working conference on mining software repositories (MSR 2010). IEEE, 2010, pp. 118–121

    work page 2010

  8. [8]

    Assessing the quality of the steps to reproduce in bug reports,

    O. Chaparro, C. Bernal-C ´ardenas, J. Lu, K. Moran, A. Marcus, M. Di Penta, D. Poshyvanyk, and V . Ng, “Assessing the quality of the steps to reproduce in bug reports,” inProceedings of the 2019 27th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering, 2019, pp. 86–96

    work page 2019

  9. [9]

    Frequently asked questions in bug reports,

    S. Breu, R. Premraj, J. Sillito, and T. Zimmermann, “Frequently asked questions in bug reports,” 2009

    work page 2009

  10. [10]

    Introducing swe-bench verified,

    OpenAI, “Introducing swe-bench verified,” https://openai.com/index/ introducing-swe-bench-verified/, Aug 2024, accessed: 2026-06-9

    work page 2024

  11. [11]

    What makes a satisficing bug report?

    T. Dal Sasso, A. Mocci, and M. Lanza, “What makes a satisficing bug report?” in2016 IEEE International Conference on Software Quality, Reliability and Security (QRS). IEEE, 2016, pp. 164–174

    work page 2016

  12. [12]

    Detecting missing information in bug descrip- tions,

    O. Chaparro, J. Lu, F. Zampetti, L. Moreno, M. Di Penta, A. Marcus, G. Bavota, and V . Ng, “Detecting missing information in bug descrip- tions,” inProceedings of the 2017 11th joint meeting on foundations of software engineering, 2017, pp. 396–407

    work page 2017

  13. [13]

    Chatbr: Automated assessment and improvement of bug report quality using chatgpt,

    L. Bo, W. Ji, X. Sun, T. Zhang, X. Wu, and Y . Wei, “Chatbr: Automated assessment and improvement of bug report quality using chatgpt,” inProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, 2024, pp. 1472–1483

    work page 2024

  14. [14]

    Past, present, and future of bug tracking in the generative ai era,

    U. B. Torun, M. T. Demircan, M. F. G ¨on, and E. T ¨uz¨un, “Past, present, and future of bug tracking in the generative ai era,”ACM Transactions on Software Engineering and Methodology, 2025

    work page 2025

  15. [15]

    The Rise of AI Teammates in Software Engineering (SE) 3.0: How Autonomous Coding Agents Are Reshaping Software Engineering

    H. Li, H. Zhang, and A. E. Hassan, “The rise of ai teammates in software engineering (se) 3.0: How autonomous coding agents are reshaping software engineering,”arXiv preprint arXiv:2507.15003, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  16. [16]

    Large language model-based agents for software engineering: A sur- vey,

    J. Liu, K. Wang, Y . Chen, X. Peng, Z. Chen, L. Zhang, and Y . Lou, “Large language model-based agents for software engineering: A sur- vey,”ACM Transactions on Software Engineering and Methodology, 2024

    work page 2024

  17. [17]

    Information needs in bug reports: improving cooperation between developers and users,

    S. Breu, R. Premraj, J. Sillito, and T. Zimmermann, “Information needs in bug reports: improving cooperation between developers and users,” inProceedings of the 2010 ACM conference on Computer supported cooperative work, 2010, pp. 301–310

    work page 2010

  18. [18]

    Humanevalcomm: Benchmarking the commu- nication competence of code generation for llms and llm agents,

    J. J. Wu and F. H. Fard, “Humanevalcomm: Benchmarking the commu- nication competence of code generation for llms and llm agents,”ACM Transactions on Software Engineering and Methodology, vol. 34, no. 7, pp. 1–42, 2025

    work page 2025

  19. [19]

    Evaluating agent-based program repair at google,

    P. Rondon, R. Wei, J. Cambronero, J. Cito, A. Sun, S. Sanyam, M. Tufano, and S. Chandra, “Evaluating agent-based program repair at google,” in2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 2025, pp. 365–376

    work page 2025

  20. [20]

    Agentic program repair from test failures at scale: A neuro-symbolic approach with static analysis and test execution feedback,

    C. Maddila, A. Tait, C. Chang, D. Cheng, N. Ahmad, V . Murali, M. Roch, A. Avondet, A. Meltzer, V . Montalvaoet al., “Agentic program repair from test failures at scale: A neuro-symbolic approach with static analysis and test execution feedback,”IEEE Transactions on Software Engineering, 2026

    work page 2026

  21. [21]

    Autocoderover: Autonomous program improvement,

    Y . Zhang, H. Ruan, Z. Fan, and A. Roychoudhury, “Autocoderover: Autonomous program improvement,” inProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2024, pp. 1592–1604

    work page 2024

  22. [22]

    Swe-bench: Can language models resolve real-world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “Swe-bench: Can language models resolve real-world github issues?” inInternational Conference on Learning Representa- tions, vol. 2024, 2024, pp. 54 107–54 157

    work page 2024

  23. [23]

    Swe-bench verified leaderboard,

    SWE-bench, “Swe-bench verified leaderboard,” 2026, accessed: 2026- 06-10. [Online]. Available: https://www.swebench.com/

    work page 2026

  24. [24]

    SWE-Bench+: Enhanced Coding Benchmark for LLMs

    R. Aleithan, H. Xue, M. M. Mohajer, E. Nnorom, G. Uddin, and S. Wang, “Swe-bench+: Enhanced coding benchmark for llms,”arXiv preprint arXiv:2410.06992, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  25. [25]

    Chaff from the wheat: Characterizing and determining valid bug reports,

    Y . Fan, X. Xia, D. Lo, and A. E. Hassan, “Chaff from the wheat: Characterizing and determining valid bug reports,”IEEE transactions on software engineering, vol. 46, no. 5, pp. 495–525, 2018

    work page 2018

  26. [26]

    readability: Python implementation of readability formulas,

    A. van Cranenburgh, “readability: Python implementation of readability formulas,” 2019, python package implementing Flesch, Fog, SMOG, ARI, and related metrics. [Online]. Available: https://pypi.org/project/ readability/

    work page 2019

  27. [27]

    Mixed-effects modeling with crossed random effects for subjects and items,

    R. H. Baayen, D. J. Davidson, and D. M. Bates, “Mixed-effects modeling with crossed random effects for subjects and items,”Journal of memory and language, vol. 59, no. 4, pp. 390–412, 2008

    work page 2008

  28. [28]

    Fitting linear mixed- effects models using lme4,

    D. Bates, M. M ¨achler, B. Bolker, and S. Walker, “Fitting linear mixed- effects models using lme4,”Journal of statistical software, vol. 67, pp. 1–48, 2015

    work page 2015

  29. [29]

    A general and simple method for obtaining r2 from generalized linear mixed-effects models,

    S. Nakagawa and H. Schielzeth, “A general and simple method for obtaining r2 from generalized linear mixed-effects models,”Methods in ecology and evolution, vol. 4, no. 2, pp. 133–142, 2013

    work page 2013

  30. [30]

    Generalized collinearity diagnostics,

    J. Fox and G. Monette, “Generalized collinearity diagnostics,”Journal of the American Statistical Association, vol. 87, no. 417, pp. 178–183, 1992

    work page 1992

  31. [31]

    The SWE-Bench illusion: When state-of-the-art LLMs remember instead of reason,

    S. Liang, S. Garg, and R. Zilouchian Moghaddam, “The SWE-Bench illusion: When state-of-the-art LLMs remember instead of reason,” in 2026 IEEE/ACM 48th International Conference on Software Engineer- ing: Software Engineering in Practice (ICSE-SEIP), 2026

    work page 2026

  32. [32]

    mini-SWE-agent: The 100-line AI agent that solves github issues,

    SWE-agent team, “mini-SWE-agent: The 100-line AI agent that solves github issues,” https://github.com/SWE-agent/mini-swe-agent, 2025

    work page 2025

  33. [33]

    Swe-bench pro leaderboard,

    Scale AI Research Team, “Swe-bench pro leaderboard,” 2025, accessed: 2026-06-10. [Online]. Available: https://scale.com/blog/swe-bench-pro

    work page 2025

  34. [34]

    Qwen3 Technical Report

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lvet al., “Qwen3 technical report,”arXiv preprint arXiv:2505.09388, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  35. [35]

    Gemma 4 31b it,

    Google DeepMind, “Gemma 4 31b it,” https://huggingface.co/google/ gemma-4-31B-it, 2026, accessed: 2026-06-14

    work page 2026

  36. [36]

    What makes a good bug report?

    N. Bettenburg, S. Just, A. Schr ¨oter, C. Weiss, R. Premraj, and T. Zim- mermann, “What makes a good bug report?” inProceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engineering, 2008, pp. 308–318

    work page 2008

  37. [37]

    Why are some bugs non-reproducible?:–an empirical investigation using data fusion– ,

    M. M. Rahman, F. Khomh, and M. Castelluccio, “Why are some bugs non-reproducible?:–an empirical investigation using data fusion– ,” in2020 IEEE international conference on software maintenance and evolution (ICSME). IEEE, 2020, pp. 605–616

    work page 2020

  38. [38]

    Modeling bug report quality,

    P. Hooimeijer and W. Weimer, “Modeling bug report quality,” inPro- ceedings of the 22nd IEEE/ACM international conference on Automated software engineering, 2007, pp. 34–43

    work page 2007

  39. [39]

    Extracting structural information from bug reports,

    N. Bettenburg, R. Premraj, T. Zimmermann, and S. Kim, “Extracting structural information from bug reports,” inProceedings of the 2008 international working conference on Mining software repositories, 2008, pp. 27–30

    work page 2008

  40. [40]

    System card: Claude sonnet 4.6,

    Anthropic, “System card: Claude sonnet 4.6,” Anthropic, Tech. Rep., Feb. 2026. [Online]. Available: https://www.anthropic.com/ claude-sonnet-4-6-system-card

    work page 2026