Pith. sign in

REVIEW 2 major objections 5 minor 1 cited by

EvoOtter generates state-of-the-art bug reproduction tests at a fraction of prior inference-scaling cost by combining evolutionary programming with LLMs.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 06:34 UTC pith:SGXTMIW2

load-bearing objection Practical cost-controlled evolutionary search for BRTs that actually delivers higher F2P rates than prior inference-scaling work. the 2 major comments →

arxiv 2607.02854 v1 pith:SGXTMIW2 submitted 2026-07-03 cs.SE cs.LG

EvoOtter: Evolutionary Reproduction Test Generator

Toufique Ahmed , Jatin Ganhotra , Avraham Shinnar , Martin Hirzel This is my paper
classification cs.SE cs.LG
keywords bug reproduction testsevolutionary programminglarge language modelsmutation testingsuccessive halvingfail-to-pass testsinference scalingsoftware engineering
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

A bug reproduction test must fail on the current buggy code for the reason stated in an informal issue report, then pass after the fix; without the future fixed code there is no reliable oracle for whether a candidate test is good. Earlier methods generate many LLM candidates and patches and use execution feedback to select and repair them, but the feedback is noisy and the cost is high. This paper shows that an evolutionary loop—scoring candidate tests by how many rule-based mutants of the localized focal functions they kill, then performing selection and crossover—sharpens that signal while successive-halving keeps the number of test executions constant and a single batched LLM call rewrites an entire generation. The resulting system reaches 75.3 % fail-to-pass on TDD-Bench-Verified and 66.3 % on SWE-Rebench with Claude-Opus-4.7 at roughly half the cost or less of previous approaches. The same design pattern points toward cheaper evolutionary programming for other software-engineering tasks that need LLMs under noisy feedback.

Core claim

Evolutionary programming tailored for bug reproduction tests—fitness defined by killing rule-based mutants via normalized execution-log differences, successive-halving of the test population against a doubling mutant set, and one-shot batched LLM crossover—produces higher-quality fail-to-pass tests than pure selection or pure repair baselines while using far fewer LLM calls and a constant execution budget per generation.

What carries the argument

Successive-halving co-evolution of a shrinking predator population of candidate tests against a growing prey population of rule-based code mutants, scored by a BRT-specific mutation-killing fitness (log differences after normalization), plus single-call batched crossover that rewrites the entire next generation of tests in one LLM prompt.

Load-bearing premise

A test’s ability to change the execution log on rule-based mutants of the suspected buggy functions is a reliable stand-in for whether that test truly fails for the reason described in the informal issue report.

What would settle it

On a held-out set of issues, rank candidate tests solely by the number of rule-based mutants they kill via log difference, then check whether that ranking predicts which tests actually go from fail to pass once the developer’s real patch is applied; near-zero rank correlation would falsify the fitness signal.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • BRT generation can reach 75 %+ fail-to-pass rates without surrogate patches or dozens of per-candidate repair calls.
  • Test-execution cost per generation stays flat even as selection pressure increases, because tests are halved while mutants are doubled.
  • Rule-based (non-LLM) mutants supply a cheap, unbiased fitness signal sufficient for competitive BRT quality.
  • Single-prompt initial populations plus evolution can outperform heterogeneous multi-prompt sampling when stronger reasoning models are used.
  • The predator-prey successive-halving pattern can be reused for other software-engineering tasks that require selection under noisy execution feedback.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same log-difference fitness may transfer to other oracle-less generation tasks such as security tests or performance-regression tests.
  • Successive-halving of candidates against a growing set of cheap synthetic environments could reduce cost in multi-agent software-engineering workflows more generally.
  • If the force-failure mutant alone already works well, even simpler mutation operators may suffice for many repositories.
  • Modern frontier models appear strong enough at joint rewriting that classic multi-parent crossover can be collapsed into one prompt without collapsing diversity.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces EvoOtter, an evolutionary-programming workflow for repository-level bug reproduction test (BRT) generation. Starting from an initial population of candidate tests (via heterogeneous or single-prompt BaseBRT), it maintains a separate population of rule-based mutants of localized focal functions, scores each test by a BRT-specific mutation fitness (normalized log differences between runs on c_old and on mutants), halves the test population each generation while doubling the mutant population (successive halving), and produces the next generation via a single batched LLM crossover call that also consumes execution logs on c_old. Empirically, with Claude-Opus-4.7 single-prompt it reports 75.3% F2P on TDD-Bench-Verified, 66.3% on SWE-Rebench, and 77.8% on SWT-Bench-Verified, at substantially lower LLM/execution cost than prior inference-scaling BRT systems; ablations compare mutation vs. LLM vs. random selection, with/without crossover, and heterogeneous vs. single-prompt initialization.

Significance. BRT generation is a practically important and still under-solved step for automated issue resolution. The paper’s main contribution is a concrete, cost-controlled way to combine evolutionary programming with modern LLMs for this task: successive halving keeps test-execution cost constant per generation, batched crossover collapses an entire generation into one LLM call, and rule-based mutants avoid spending LLM budget on the prey population. The evaluation is comparatively thorough for the area (two primary benchmarks plus SWT-Bench-Verified, McNemar tests, pre-/post-cutoff split, cost tables, and multiple ablations). If the reported F2P/cost trade-off holds under independent reimplementation, the design patterns (batched crossover, BRT-oriented mutation fitness, successive-halving co-population) are likely to be reused beyond BRT generation.

major comments (2)
  1. Section III-C defines “killing” a mutant solely via a change in normalized execution logs relative to the run on c_old. Fig. 5 and Table III show that this score correlates with true F2P (measured against hidden golden patches) and that mutation+crossover beats LLM or random selection. However, the proxy never checks whether the changed failure reason is the one described in d_issue; it only detects that the log changed. When localization is imprecise (explicitly acknowledged) or residual non-determinism survives whitespace/constant normalization, the score can reward brittleness to any syntactic edit rather than issue-faithfulness. Because the evolutionary narrative (“sharper feedback”) rests on this proxy, the paper should either (a) report how often the log-diff failure mode actually matches the issue semantics (e.g., via manual or LLM-as-judge audit on a sample of killed mutants), or
  2. Table VI and the cost discussion in §IV-A/IV-E claim “fraction of the cost of prior inference-scaling approaches,” citing e-Otter’s earlier $1.80–$2.75 figures that included surrogate-patch generation. The fairer comparison is against systems that also do only BRT generation/selection (Otter, Aegis-style ranking, single-skill e-Otter). The paper already shows EvoOtter beats those on F2P at ~$0.48–$1.12, but the abstract and Finding 5 still frame the result against the more expensive joint test+patch pipelines. Please restate the cost claim against the closest BRT-only baselines and, if possible, report wall-clock or total token counts for the full pipeline (including BaseBRT localization) so readers can reproduce the “fraction” claim.
minor comments (5)
  1. Capitalization of the system name is inconsistent (EvoOtter / EVOOTTER / EVOOtter) across abstract, figures, and body; pick one form.
  2. Fig. 2 timeline and the n=8 / m=20 example are helpful; adding the exact default m used in the main experiments would make the successive-halving cost argument fully concrete.
  3. Table IX (single-mutant-category ablation on 100 instances) is useful but small; a short note on variance or confidence intervals would help readers weigh the “force-failure alone can match all-mutants” observation.
  4. Related-work placement of Adamopoulos et al. and EvoSuite is appropriate; a one-sentence clarification that those systems evolve traditional (pass-to-pass) tests, not F2P BRTs, would further reduce reader confusion.
  5. Threats (§VI) correctly flag Python/pytest-only evaluation and limited hyperparameter search; mentioning whether the rule-based mutant operators are language-specific would help assess portability.

Circularity Check

0 steps flagged

No circularity: empirical SE method paper whose F2P claims are measured against hidden golden patches independent of the mutant-killing fitness used for selection.

full rationale

EvoOtter is an engineering contribution that defines a workflow (initial heterogeneous or single-prompt generation, rule-based mutants, successive-halving selection by log-difference mutation score, batched LLM crossover) and evaluates it on external benchmarks (TDD-Bench-Verified, SWE-Rebench, SWT-Bench-Verified) via the objective F2P metric that requires the hidden c_new. The fitness definition in §III-C (a test kills a mutant when its normalized execution log differs from the log on c_old) is a design choice for ranking candidates; it is never claimed to be mathematically equivalent to F2P, nor is any free parameter fitted to the reported F2P rates. Ablations (Tables II–V, Figs. 4–6) treat mutation score, LLM selection, and random selection as alternatives and measure their effect on the same external metric. Self-citations to the authors’ prior Otter/e-Otter/TDD-Bench work appear as baselines and components, but the novel evolutionary loop, successive-halving cost control, and batched crossover are self-contained and do not reduce the SOTA F2P numbers to those citations by construction. No uniqueness theorem, ansatz smuggled via self-citation, or fitted-input-as-prediction pattern is present. The paper is therefore free of the enumerated circularity patterns.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 2 invented entities

The central empirical claim rests on a small set of design hyperparameters, standard mutation-testing assumptions, and the new fitness definition; no new physical entities are postulated.

free parameters (3)
  • initial population size n = 8
    Fixed at 8 (and log2(n) generations) by design choice; not fitted to the evaluation metric but controls cost/quality trade-off.
  • initial mutant count m = 20 (example)
    Chosen so that 2m executions stay constant under successive doubling; example value 20 appears in the text.
  • set of nine rule-based mutant operators
    Hand-selected operators (arithmetic, relational, logical, constant, return, assignment, force-failure); composition of the mutant pool is therefore a free design choice.
axioms (3)
  • ad hoc to paper A change in normalized execution logs between a test run on original code and on a mutant constitutes “killing” the mutant and is a useful fitness signal for BRTs.
    Defined in Section III-C; traditional mutation score assumes the original program is correct, which is false for BRTs, so the authors invent a log-difference proxy.
  • domain assumption Localized focal functions obtained by the three-stage LLM localizer are sufficiently accurate that mutants of those functions provide a meaningful selection signal.
    Localization is imperfect; the paper notes that mutation score can be zero or uniform when localization fails, falling back to an LLM critic.
  • domain assumption Successive-halving (halve tests, double mutants each generation) keeps total execution cost constant while improving selection quality.
    Imported from multi-armed-bandit / AutoML literature (Jamieson & Talwalkar, DAUB) and applied without re-derivation.
invented entities (2)
  • BRT-specific mutation-score fitness (log-difference killing) no independent evidence
    purpose: Provide a cheap, execution-based ranking signal that does not require golden patches or surrogate patches.
    Traditional mutation score is inapplicable because the original code is already buggy; the log-difference definition is new to this paper.
  • Batched crossover (one LLM call for an entire generation of offspring) no independent evidence
    purpose: Keep LLM cost linear in the number of generations rather than linear in population size.
    Standard evolutionary crossover is per-individual; the single-call formulation is an engineering invention of the paper.

pith-pipeline@v1.1.0-grok45 · 22212 in / 2486 out tokens · 30233 ms · 2026-07-12T06:34:55.967515+00:00 · methodology

0 comments
read the original abstract

Before fixing an issue, it is useful to first reproduce it by generating a bug reproduction test (BRT). However, generating a BRT is itself a challenging task, because issue descriptions tend to be informal, making it difficult to determine whether a candidate BRT indeed fails for the reason in the issue. Prior work has attempted to tackle this problem via inference scaling, using large language models to generate many BRTs and patches, then using execution feedback to select and improve them. Unfortunately, this is expensive and the feedback is unreliable. This paper explores evolutionary programming for BRT generation to sharpen the feedback, while enhancing evolutionary programming to keep costs in check. Our new approach, EvoOtter, controls test execution costs via successive halving. Furthermore, it controls LLM costs via batched crossover for an entire generation in a single LLM call, as well as via rule-based code mutations, with a new fitness score tailored for BRTs. As a result, EvoOtter generates state-of-the-art quality BRTs at the fraction of the cost of prior inference-scaling approaches to this problem. More broadly, this paper points at how to efficiently and effectively combine evolutionary programming with large language models for software engineering.

Figures

Figures reproduced from arXiv: 2607.02854 by Avraham Shinnar, Jatin Ganhotra, Martin Hirzel, Toufique Ahmed.

Figure 1
Figure 1. Figure 1: Overview of EVOOTTER workflow. code for context in the next round of BRT generation [28]. To the best of our knowledge, none of the prior work on BRT generation uses evolutionary programming. III. METHODOLOGY This section describes the overall workflow of EVOOTTER and its main components (initial test generator, the selection process, and crossover). A. Overall Workflow [PITH_FULL_IMAGE:figures/full_fig_p… view at source ↗
Figure 2
Figure 2. Figure 2: Timeline. the doubling/halving regime, the number of test executions per iteration is constant at 2m; in the example, with n = 8 and m = 20, each iteration runs 2m = 40 test executions. Prior work performed either selection [29], [6], [7], [28], repair [23], [17], [6], [7], or both, but in separate phases. In contrast, EVOOTTER combines selection (based on mutant￾killing fitness) with repair (crossover) un… view at source ↗
Figure 4
Figure 4. Figure 4: Accuracy of selection strategies at different stages [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Mutation killing ability of F2P tests (0.51) and random selection (0.49). Nevertheless, mutation￾based selection achieves the best overall performance. This is because the final performance depends on both selection and crossover. Although LLM-based selection produces a slightly higher F2P ratio after the second crossover, mutation-based selection retains better F2P tests throughout the evolutionary proces… view at source ↗
Figure 6
Figure 6. Figure 6: Impact of crossover at different stages 0 1 2 3 4 5 6 7 8 Number of Tests Passed Sonnet (Heterogeneous) Sonnet (Single Prompt) Opus (Single Prompt) Sonnet (Heterogeneous) Sonnet (Single Prompt) Opus (Single Prompt) Sonnet (Heterogeneous) Sonnet (Single Prompt) Opus (Single Prompt) 2nd Crossover 1st Crossover Init Tests 29.8% 9.2% 61.0% 33.5% 11.3% 55.2% 21.7% 7.1% 71.2% 28.9% 4.3% 6.5% 15.5% 44.8% 33.9% 3.… view at source ↗
Figure 7
Figure 7. Figure 7: Distribution of F2P tests for heterogeneous and single prompting on TDD-Bench-Verified test candidates in a single call instead of generating one test per call. Table V shows that single prompting results in a slight performance drop on both benchmarks (from 65.9% to 62.4% on TDD-Bench-Verified and from 51.9% to 49.9% on SWE-Rebench). This is expected because all candidate tests are generated from the same… view at source ↗
Figure 8
Figure 8. Figure 8: Evolution of coverage in generated tests [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Validation Evidence in LLM Repair Agents: How Much of What Passes Actually Tests the Bug?

    cs.SE 2026-07 conditional novelty 7.0

    By replaying each validation command on buggy, candidate, and gold-fix code, this study finds 46% of positive checks do not discriminate the bug, and a reminder alone explains part of the feedback benefit.

Reference graph

Works this paper leans on

33 extracted references · 1 canonical work pages · cited by 1 Pith paper

  1. [1]

    [Online]

    2025. [Online]. Available: https://www.anthropic.com/news/ claude-sonnet-4-5

    2025

  2. [2]

    [Online]

    2025. [Online]. Available: https://swtbench.com/?results=verified

    2025

  3. [3]

    [Online]

    2026. [Online]. Available: https://openai.com/index/ why-we-no-longer-evaluate-swe-bench-verified/

    2026

  4. [4]

    How to overcome the equivalent mutant problem and achieve tailored selective mutation using co-evolution,

    K. Adamopoulos, M. Harman, and R. M. Hierons, “How to overcome the equivalent mutant problem and achieve tailored selective mutation using co-evolution,” inGenetic and Evolutionary Computation (GECCO), 2004, pp. 1338–1349. [Online]. Available: https://doi.org/10.1007/978-3-540-24855-2 155

    work page doi:10.1007/978-3-540-24855-2 2004

  5. [5]

    GEPA: Reflective prompt evolution can outperform reinforcement learning,

    L. A. Agrawal, S. Tan, D. Soylu, N. Ziems, R. Khare, K. Opsahl-Ong, A. Singhvi, H. Shandilya, M. J. Ryan, M. Jiang, C. Potts, K. Sen, A. G. Dimakis, I. Stoica, D. Klein, M. Zaharia, and O. Khattab, “GEPA: Reflective prompt evolution can outperform reinforcement learning,” Jul. 2025. [Online]. Available: https://arxiv.org/abs/2507.19457

    Pith/arXiv arXiv 2025

  6. [6]

    Otter: Generating tests from issues to validate SWE patches,

    T. Ahmed, J. Ganhotra, R. Pan, A. Shinnar, S. Sinha, and M. Hirzel, “Otter: Generating tests from issues to validate SWE patches,” in International Conference on Machine Learning (ICML), Jul. 2025. [Online]. Available: https://proceedings.mlr.press/v267/ahmed25b.html

    2025

  7. [7]

    Heterogeneous prompting and execution feedback for SWE issue test generation and selection,

    T. Ahmed, J. Ganhotra, A. Shinnar, and M. Hirzel, “Heterogeneous prompting and execution feedback for SWE issue test generation and selection,” inInternational Conference on Software Engineering (ICSE), Apr. 2026

    2026

  8. [8]

    TDD-Bench Verified: Can LLMs generate tests for issues before they get resolved?

    T. Ahmed, M. Hirzel, R. Pan, A. Shinnar, and S. Sinha, “TDD-Bench Verified: Can LLMs generate tests for issues before they get resolved?” Dec. 2024. [Online]. Available: https://arxiv.org/abs/2412.02883

    Pith/arXiv arXiv 2024

  9. [9]

    Swe- rebench: An automated pipeline for task collection and decontaminated evaluation of software engineering agents,

    I. Badertdinov, A. Golubev, M. Nekrashevich, A. Shevtsov, S. Karasik, A. Andriushchenko, M. Trofimova, D. Litvintseva, and B. Yangel, “Swe- rebench: An automated pipeline for task collection and decontaminated evaluation of software engineering agents,” May 2025. [Online]. Available: https://arxiv.org/abs/2505.20411

    arXiv 2025

  10. [10]

    Agentic bug reproduction for effective automated program repair at Google,

    R. Cheng, M. Tufano, J. Cito, J. Cambronero, P. Rondon, R. Wei, A. Sun, and S. Chandra, “Agentic bug reproduction for effective automated program repair at Google,” Feb. 2025. [Online]. Available: https://arxiv.org/abs/2502.01821

    Pith/arXiv arXiv 2025

  11. [11]

    CodeMonkeys: Scaling test-time compute for software engineering,

    R. Ehrlich, B. Brown, J. Juravsky, R. Clark, C. Re, and A. Mirhoseini, “CodeMonkeys: Scaling test-time compute for software engineering,” Jan. 2025. [Online]. Available: https://arxiv.org/abs/2501.14723

    Pith/arXiv arXiv 2025

  12. [12]

    An introduction to evolutionary programming,

    D. B. Fogel and L. J. Fogel, “An introduction to evolutionary programming,” inEuropean Conference on Artificial Evolution (AE), Sep. 1995, pp. 21–33. [Online]. Available: https://doi.org/10.1007/ 3-540-61108-8 28

    1995

  13. [13]

    EvoSuite: automatic test suite generation for object-oriented software,

    G. Fraser and A. Arcuri, “EvoSuite: automatic test suite generation for object-oriented software,” inSymposium and the European Conference on Foundations of Software Engineering (ESEC), 2011, pp. 416–419. [Online]. Available: https://doi.org/10.1145/2025113.2025179

    doi:10.1145/2025113.2025179 2011

  14. [14]

    Non-stochastic best arm identification and hyperparameter optimization,

    K. Jamieson and A. Talwalkar, “Non-stochastic best arm identification and hyperparameter optimization,” inConference on Artificial Intelligence and Statistics (AISTATS), May 2016, pp. 240–248. [Online]. Available: https://proceedings.mlr.press/v51/jamieson16.html

    2016

  15. [15]

    An analysis and survey of the development of mutation testing,

    Y . Jia and M. Harman, “An analysis and survey of the development of mutation testing,”IEEE Transactions on Software Engineering (TSE), vol. 37, no. 5, pp. 649–678, 2011. [Online]. Available: https://doi.org/10.1109/TSE.2010.62

    doi:10.1109/tse.2010.62 2011

  16. [16]

    SWE-bench: Can language models resolve real-world GitHub issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “SWE-bench: Can language models resolve real-world GitHub issues?” inInternational Conference on Learning Representa- tions (ICLR), May 2024

    2024

  17. [17]

    AssertFlip: Reproducing bugs via inversion of LLM-generated passing tests,

    L. Khatib, N. S. Mathews, and M. Nagappan, “AssertFlip: Reproducing bugs via inversion of LLM-generated passing tests,” inInternational Conference on Software Engineering (ICSE), Apr. 2026

    2026

  18. [18]

    GenProg: A generic method for automatic software repair,

    C. Le Goues, T. Nguyen, S. Forrest, and W. Weimer, “GenProg: A generic method for automatic software repair,”IEEE Transactions on Software Engineering (TSE), vol. 38, no. 1, pp. 54–72, 2012. [Online]. Available: https://doi.org/10.1109/TSE.2011.104

    doi:10.1109/tse.2011.104 2012

  19. [19]

    The SWE-bench illusion: When state-of-the-art LLMs remember instead of reason,

    S. Liang, S. Garg, and R. Z. Moghaddam, “The SWE-bench illusion: When state-of-the-art LLMs remember instead of reason,” Jun. 2025. [Online]. Available: https://arxiv.org/abs/2506.12286

    arXiv 2025

  20. [20]

    Dissecting the swe-bench leaderboards: Profiling submitters and architectures of llm-and agent-based repair systems,

    M. Martinez and X. Franch, “Dissecting the swe-bench leaderboards: Profiling submitters and architectures of llm-and agent-based repair systems,”arXiv preprint arXiv:2506.17208, 2025

    arXiv 2025

  21. [21]

    Note on the sampling error of the difference between correlated proportions or percentages,

    Q. McNemar, “Note on the sampling error of the difference between correlated proportions or percentages,”Psychometrika, vol. 12, no. 2, pp. 153–157, 1947

    1947

  22. [22]

    SWT-Bench: Testing and validating real-world bug-fixes with code agents,

    N. M ¨undler, M. N. M ¨uller, J. He, and M. Vechev, “SWT-Bench: Testing and validating real-world bug-fixes with code agents,” inConference on Neural Information Processing Systems (NeurIPS), Dec. 2024. [Online]. Available: https://openreview.net/forum?id=9Y8zUO11EQ

    2024

  23. [23]

    Issue2Test: Generating reproducing test cases from issue reports,

    N. Nashid, I. Bouzenia, M. Pradel, and A. Mesbah, “Issue2Test: Generating reproducing test cases from issue reports,” Mar. 2025. [Online]. Available: https://arxiv.org/abs/2503.16320

    arXiv 2025

  24. [24]

    Refine: En- hancing program repair agents through context-aware patch refinement,

    A. Pabba, S. Chen, A. Mathai, A. Chakraborty, and B. Ray, “Refine: En- hancing program repair agents through context-aware patch refinement,” arXiv preprint arXiv:2510.03588, 2025

    arXiv 2025

  25. [25]

    Mathematical discoveries from program search with large language models,

    B. Romera-Paredes, M. Barekatain, A. Novikov, M. Balog, M. P. Kumar, E. Dupont, F. J. R. Ruiz, J. S. Ellenberg, P. Wang, O. Fawzi, P. Kohli, and A. Fawzi, “Mathematical discoveries from program search with large language models,”Nature, vol. 625, pp. 468–475, Jan. 2024. [Online]. Available: https://doi.org/10.1038/s41586-023-06924-6

    doi:10.1038/s41586-023-06924-6 2024

  26. [26]

    Selecting near- optimal learners via incremental data allocation,

    A. Sabharwal, H. Samulowitz, and G. Tesauro, “Selecting near- optimal learners via incremental data allocation,” inConference on Artificial Intelligence (AAAI), Feb. 2016, pp. 2007–2015. [Online]. Available: https://www.aaai.org/ocs/index.php/AAAI/AAAI16/ paper/viewPaper/12524

    2016

  27. [27]

    Is the cure worse than the disease? overfitting in automated program repair,

    E. K. Smith, E. T. Barr, C. Le Goues, and Y . Brun, “Is the cure worse than the disease? overfitting in automated program repair,” in Symposium on the Foundations of Software Engineering (FSE), Aug

  28. [28]

    Available: https://doi.org/10.1145/2786805.2786825

    [Online]. Available: https://doi.org/10.1145/2786805.2786825

    doi:10.1145/2786805.2786825

  29. [29]

    iCoRe: An iterative correlation-aware retriever for bug reproduction test generation,

    J. Wang, J. Cao, and Z. Liu, “iCoRe: An iterative correlation-aware retriever for bug reproduction test generation,” Apr. 2026. [Online]. Available: https://arxiv.org/abs/2604.19224

    Pith/arXiv arXiv 2026

  30. [30]

    AEGIS: An agent-based framework for general bug reproduction from issue descriptions,

    X. Wang, P. Gao, X. Meng, C. Peng, R. Hu, Y . Lin, and C. Gao, “AEGIS: An agent-based framework for general bug reproduction from issue descriptions,” Nov. 2024. [Online]. Available: https://arxiv.org/abs/2411.18015

    Pith/arXiv arXiv 2024

  31. [31]

    Co-evolving LLM coder and unit tester via reinforcement learning,

    Y . Wang, L. Yang, Y . Tian, K. Shen, and M. Wang, “Co-evolving LLM coder and unit tester via reinforcement learning,” inAdvances in Neural Information Processing Systems, vol. 38, Dec. 2025, pp. 143 630–143 664. [On- line]. Available: https://proceedings.neurips.cc/paper files/paper/2025/ file/d38653cdaa8e992549e1e9e1621610d7-Paper-Conference.pdf

    2025

  32. [32]

    Demystifying LLM-based software engineering agents,

    C. S. Xia, Y . Deng, S. Dunn, and L. Zhang, “Demystifying LLM-based software engineering agents,” inSymposium on the Foundations of Software Engineering (FSE), Jun. 2025, pp. 801–824. [Online]. Available: https://doi.org/10.1145/3715754

    doi:10.1145/3715754 2025

  33. [33]

    SWE-Agent: Agent-computer interfaces enable automated software engineering,

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “SWE-Agent: Agent-computer interfaces enable automated software engineering,” inConference on Neural Information Processing Systems (NeurIPS), Dec. 2024. [Online]. Available: https://proceedings.neurips.cc/paper files/paper/2024/hash/ 5a7c947568c1b1328ccc5230172e1e7c-Abstra...

    2024