Pith. sign in

REVIEW 2 major objections 8 minor 1 cited by

42% bug-reproducing tests from issue descriptions alone

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-05 08:58 UTC pith:4MJVZM4R

load-bearing objection iCoRe combines differentiated code/test retrieval, function-call-structure similarity, and generation-to-retrieval feedback for bug reproduction test generation; the approach is well-engineered with consistent gains, but headline margins rest on a self-re-implemented baseline and lack significance testing. the 2 major comments →

arxiv 2604.19224 v2 pith:4MJVZM4R submitted 2026-04-21 cs.SE

iCoRe: An Iterative Correlation-Aware Retriever for Bug Reproduction Test Generation

Junyi Wang , Jialun Cao , Zhongxin Liu This is my paper
classification cs.SE
keywords bug reproduction testcode retrievalfunction call treetree edit distanceLLM-based test generationiterative retrievalsoftware maintenance
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes iCoRe, a retrieval method that improves LLM-based bug reproduction test generation by treating source code and test cases as distinct retrieval targets, using function-call-structure similarity (not just text) to judge behavioral relevance, and feeding generated sketch tests back into retrieval for iterative refinement. The core mechanism is a loop: retrieve context, generate a rough test, compare that test's call tree against existing tests using weighted tree edit distance, rerank, and repeat. On two benchmarks, this yields Fail-to-Pass rates of 42.0% and 52.8%, representing 19.7%–31.7% relative improvements over prior retrievers, while using roughly 2,500 tokens of context at about $0.28 per instance.

Core claim

The central discovery is that closing the loop between retrieval and generation — using each generated sketch test as a new query to find behaviorally similar existing tests via function-call-tree comparison — produces substantially better context than any one-pass retrieval strategy. The paper shows that differentiating production-code retrieval (keyword-driven, heuristic-filtered) from test-code retrieval (iterative, similarity-based) and combining textual and structural similarity signals yields concise, high-quality context that lets even a basic LLM generator outperform more complex end-to-end systems at a fraction of the cost.

What carries the argument

Pruned Function Call Tree (a tree rooted at each test function, expanded only until it enters production code); Weighted Tree Edit Distance (Zhang-Shasha algorithm with IDF-based node weights, keyword functions weighted at 1.0); two-stage differentiated retrieval (keyword extraction + heuristic selection for production code, iterative LLM-guided search + BM25 + call-tree similarity + LLM reranking for test code); sketch BRT feedback loop (generated test serves as query for next retrieval round, up to 3 iterations).

Load-bearing premise

The pruned Function Call Tree stops expanding call paths once they enter production code, which means bugs whose root causes depend on deep interactions within production code — such as a constraint defined in a parent class — may be missed because the relevant call chain is truncated.

What would settle it

If one replaced the function-call-tree similarity with pure textual (BM25) similarity and saw no change in retrieval accuracy or final Fail-to-Pass rates, the central structural-similarity contribution would be unfounded. The ablation removing function-call similarity shows a 6–12% drop, providing partial evidence.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 8 minor

Summary. The paper proposes iCoRe, an iterative correlation-aware retrieval approach for bug reproduction test generation. iCoRe has three main components: (1) differentiated retrieval for production code (keyword-based with heuristic selection) and test code (LLM-guided exploration with iterative refinement), (2) function-call-structure similarity measured via weighted tree edit distance on pruned function call trees, and (3) a feedback loop from a sketch BRT generator to the test retrieval module. The approach is evaluated on SWT-bench Lite (276 instances) and TDD-bench Verified (449 instances) using three LLMs (GPT-4o, DeepSeek-V3, Qwen3-32B), with five research questions covering end-to-end performance, comparison with SOTA systems, ablation, retrieval accuracy, and hyperparameter sensitivity.

Significance. The paper addresses a well-motivated problem: the quality of retrieved context is a known bottleneck for LLM-based bug reproduction test generation. The three identified limitations of existing retrievers (undifferentiated code/test retrieval, neglect of call-structure relationships, lack of generation-to-retrieval feedback) are concrete and well-illustrated through the motivating example. The differentiated retrieval strategy and the weighted tree edit distance on function call trees are technically novel contributions to this problem space. The experimental design is thorough: two benchmarks, three LLMs, four ablation variants, a dedicated retrieval accuracy evaluation (RQ4), and hyperparameter sensitivity analysis (RQ5). The ablation study (Table 4) confirms that each component contributes positively, and the retrieval accuracy evaluation (Table 5) provides an independent line of evidence beyond end-to-end F→P rates. The replication package is publicly available.

major comments (2)
  1. §5.2, Table 3: The end-to-end comparison against e-Otter++ (whose numbers come from the original paper, not a re-implementation) shows differences of only 1.8 pp on SWT-bench Lite (42.0% vs 40.2%) and 1.4 pp on TDD-bench Verified (52.8% vs 51.4%). With 276 and 449 instances, these correspond to approximately 5 and 6 instance-level differences. No statistical significance test (McNemar, bootstrap, or otherwise) is reported for any comparison in the paper. The abstract and §5.2 describe these as 'significant' improvements. Given the small margins in the only comparison that does not depend on a self-re-implemented baseline, significance testing is needed to substantiate the claim of superiority over e-Otter++. At minimum, the word 'significant' should be qualified or removed until such tests are provided.
  2. §4.3.1: The headline relative improvement of 19.7%–31.7% (RQ1, Table 2) is computed against Otter's retriever, which the authors re-implemented themselves because official retrieval results are unavailable. Otter is the strongest baseline in Table 2, so the headline numbers are sensitive to re-implementation fidelity. The paper states 'we faithfully re-implemented this method based on the paper's description and prompts' but provides no quantitative validation of the re-implementation's quality (e.g., comparison against any available partial artifacts, or reproduction of Otter's reported end-to-end numbers). The retrieval accuracy evaluation in RQ4 (Table 5) does show iCoRe outperforming the re-implemented Otter retriever by substantial margins in MAP/MRR/Hit@k, which provides some independent evidence. However, the paper should explicitly acknowledge that the RQ1 headline numbers depend
minor comments (8)
  1. §3.2.2, Eq. (1): The weight function W(f) caps non-keyword functions at 0.9 and keyword functions at 1.0. The rationale for the specific cap of 0.9 (rather than, say, 0.95 or 0.85) is not discussed. A brief justification would help.
  2. §3.2.2: The textual similarity formula 'Similarity_textual = sim_name + sim_code' lacks normalization. Are sim_name and sim_code on the same scale? If BM25 scores are used for both, the raw sum may be dominated by whichever component has larger magnitude. Clarify whether any normalization is applied.
  3. Table 2: The note states that AEGIS's and AssertFlip's retrieval artifacts are GPT-4o-based and used as fixed inputs for DeepSeek-V3 and Qwen3-32B experiments. This means the cross-model comparison for these two baselines is not fully controlled (the retrieval was optimized for GPT-4o). This should be noted more prominently in the table caption or the text discussing cross-model results.
  4. §5.5, Figure 4: The x-axis starts at 0 iterations, but the text in §3.2.3 and §4.4 states max iterations is set to 3. It is unclear whether '0 iterations' corresponds to the 'w/o Iteration' ablation in Table 4. Clarify the relationship between Figure 4's x-axis and the ablation variants.
  5. §7.2, Category D (Context Deficiency): The django__django-11797 example shows that the pruned Function Call Tree (§3.2.2) misses the parent class AbstractUser because the NOT NULL constraint is defined in a parent class not captured by retrieval. The paper acknowledges this as a limitation but does not discuss whether the pruning strategy (stopping at production module boundaries) is the root cause. The connection between the pruning design decision and this failure mode should be made explicit, as it suggests a structural limitation of the approach.
  6. §4.4: The basic BRT generator is described as 'inspired by LIBRO' but adapted for Python. The temperature is set to 0.7 for generation and 0 for retrieval steps. The justification for 0.7 (rather than 0 or another value) is not provided. Since the generator is a key variable in the end-to-end results, a brief rationale would be helpful.
  7. Table 3: The Patch Coverage (ΔC) for e-Otter++ on SWT-bench Lite is listed as '–' (unknown). The paper should note whether this metric was simply not reported in the e-Otter++ paper or could not be computed.
  8. §1 and Abstract: The phrase '19.7%–31.7% relative improvements over existing retrieval methods' could be misread as applying to all existing methods. It specifically refers to the comparison with Otter's retriever using GPT-4o. Consider clarifying that this range refers to the strongest single baseline.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the careful and constructive review. The two major comments both concern statistical rigor and re-implementation fidelity. We address each below and commit to concrete revisions.

read point-by-point responses

  1. Referee: §5.2, Table 3: Differences of only 1.8 pp and 1.4 pp over e-Otter++ (numbers from original paper, not re-implementation). No significance test reported. Abstract and §5.2 describe these as 'significant' improvements. Significance testing needed, or the word 'significant' should be qualified or removed.

    Authors: The referee is correct that we did not report statistical significance tests for the end-to-end comparisons in Table 3, and that the margins over e-Otter++ are small in absolute terms (approximately 5 and 6 instances). We will address this in two ways in the revision. First, we will run McNemar's test on the per-instance pass/fail outcomes for iCoRe+Basic Generator vs. e-Otter++ on both benchmarks (using GPT-4o) and report the p-values in Table 3. Second, regardless of the outcome, we will remove the word 'significant' from the abstract and from the RQ2 answer where it is used to describe the e-Otter++ comparison, replacing it with a neutral description of the observed differences. We agree that without a significance test, the word 'significant' is not substantiated for this particular comparison. We note that the RQ1 headline numbers (19.7%–31.7% relative improvement over Otter's retriever) are based on a self-re-implemented baseline and are a separate concern addressed in the second comment. revision: yes

  2. Referee: §4.3.1: The headline relative improvement of 19.7%–31.7% (RQ1, Table 2) is computed against Otter's retriever, which was re-implemented by the authors. No quantitative validation of re-implementation fidelity is provided. The paper should explicitly acknowledge that the RQ1 headline numbers depend on re-implementation fidelity.

    Authors: We agree that this dependency should be stated more explicitly. In the revision, we will add a clear caveat in §4.3.1 and in the RQ1 discussion acknowledging that the Otter retriever baseline is our own re-implementation (since official retrieval artifacts are unavailable) and that the headline RQ1 numbers are therefore contingent on re-implementation fidelity. We will also note the existing partial mitigation: the RQ4 retrieval accuracy evaluation (Table 5) shows iCoRe outperforming the re-implemented Otter retriever by substantial margins in MAP, MRR, and Hit@k at both file and function levels, which provides an independent line of evidence that does not depend on end-to-end generation. However, we acknowledge that RQ4 still uses the same re-implemented Otter retriever as the baseline, so it does not fully eliminate the fidelity concern. We will make this limitation explicit in the revised Threats to Validity section (§6, Internal Validity), where we already briefly mention the replication issue but will now expand it to directly flag the RQ1 headline numbers as dependent on re-implementation quality. We will also add a note that our re-implementation code is included in the replication package, enabling independent verification. revision: yes

Circularity Check

0 steps flagged

No significant circularity found; derivation chain is self-contained against external benchmarks.

full rationale

The paper's central claims (42.0% and 52.8% F→P rates) are evaluated on externally defined benchmarks (SWT-bench Lite, TDD-bench Verified) whose ground-truth test patches are independent of the authors. The derivation chain proceeds as: issue description → keyword extraction (Section 3.1) → production-code retrieval + initial test retrieval → sketch BRT generation → similarity calculation (Eq. 1, using keywords from the issue description and IDF from the test suite) → LLM reranking → refined retrieval → iterate → final context → BRT generation → external evaluation. No step reduces to its own inputs by construction. The keyword set S_keyword in Eq. 1 is derived from the issue description (an input), not from the predicted output. The iterative feedback loop uses a generated sketch BRT to refine retrieval, but the final output is evaluated against external ground truth, so the loop is refinement, not circularity. The RQ4 retrieval accuracy evaluation uses developer-modified test functions as ground truth, which are externally defined. The authors do cite their own prior work (refs [9], [21], [36], [39]), but none of these citations are load-bearing for the central claims—the claims rest on experimental results, not on imported theorems or ansätze from self-citations. The skeptic's concerns about the re-implemented Otter baseline and lack of significance testing are validity/fairness concerns, not circularity. Score 1 reflects the presence of non-load-bearing self-citations with no circular derivation.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

iCoRe introduces no new postulated entities, particles, forces, or dimensions. All components (BM25, IDF, Zhang-Shasha TED, LLM reranking, hierarchical code structure) are established techniques. The weighted TED formulation (Eq. 1) is a new combination of existing components (IDF + keyword set + tree edit distance), not a new entity. The free parameters (α, max iterations, top-k) are hyperparameters tuned on the evaluation benchmarks, not fundamental constants. The axioms are domain assumptions about the sufficiency of the chosen representations, all stated explicitly in the paper.

free parameters (5)
  • α (base weight in function similarity) = 0.1
    Set as default; sensitivity analysis (Table 7) shows performance is robust to α in [0.1, 0.4] with std=0.45%. Selected for optimal F→P(@1) on SWT-bench Lite.
  • max iterations = 3
    Number of sketch-generation-and-reranking cycles. Figure 4 shows performance peaks at 3 iterations and stabilizes. Selected for performance-efficiency balance.
  • top-k tests in reranking = 5
    Maximum number of tests selected by LLM reranker as final context. Table 6 shows k=5 yields best F→P(@1); k=10 is slightly worse for @1 but marginally better for @5.
  • weight cap for keyword functions = 1.0
    Fixed maximum weight for functions whose names appear in the issue description (Eq. 1). Not tuned; set by design choice to distinguish from non-keyword functions (capped at 0.9).
  • weight cap for non-keyword functions = 0.9
    Upper bound for IDF-derived weights of non-keyword functions (Eq. 1). Set by design to be below keyword weight (1.0). Not tuned.
axioms (5)
  • domain assumption Pruned Function Call Trees (stopping at production module boundaries) are a sufficient behavioral representation for matching test relevance.
    Section 3.2.2: 'We model these relationships for each test as a pruned Function Call Tree, rooted at the test function itself, and stop expanding the call path once it enters a production module.' This is load-bearing for the function-call similarity computation.
  • domain assumption BM25 textual similarity on test code and test names captures meaningful textual relevance for test retrieval.
    Section 3.2.2: textual similarity is computed as sim_name + sim_code using BM25. This is standard but assumes token overlap correlates with test relevance.
  • domain assumption LLM-based reranking can reliably select relevant tests from candidate sets without systematic bias.
    Section 3.2.3: 'While LLMs may make biased judgments, we employ them in the loop for their automation and powerful reasoning capabilities.' The paper reports only 4% of target tests are filtered out (Section 5.4), but this is measured on the same benchmarks used for evaluation.
  • domain assumption SWT-bench Lite and TDD-bench Verified are representative benchmarks for evaluating BRT generation.
    Section 4.1: both benchmarks contain Python projects from 12 open-source repositories. The paper acknowledges this limitation in Section 6 (External Validity).
  • domain assumption GPT-4o, DeepSeek-V3, and Qwen3-32B are representative models for evaluating BRT generation.
    Section 4.4: these three models are selected as representatives of closed-source, large open-source, and medium-scale open-source models respectively.

pith-pipeline@v1.1.0-glm · 29005 in / 3634 out tokens · 232501 ms · 2026-07-05T08:58:11.731582+00:00 · methodology

0 comments
read the original abstract

Automatically generating bug reproduction tests (BRT) from issue descriptions is crucial for software maintenance. LLM-based approaches have shown great potential for this task. Their effectiveness heavily relies on retrieving high-quality context from the codebase. The retrieval phase of existing approaches relies on either traditional methods like BM25 or LLM-driven strategies. LLM-based retrieval strategies typically equip an LLM with tools to autonomously explore the repository or select the most relevant files and code snippets from a provided list as context. However, these retrieval methods suffer from three key limitations: 1) They often employ a unified strategy for retrieving both source code and test cases, overlooking their distinct retrieval requirements. 2) They focus solely on semantic similarity while ignoring function call relationships, leading to irrelevant context. 3) The retrieval lacks a feedback loop from the generation phase, preventing it from refining the context based on execution results. These limitations collectively result in low-quality context, thereby hindering the accuracy of bug reproduction. To address these challenges, we propose iCoRe, an iterative, correlation-aware context retrieval approach explicitly aware of three key correlations: 1) between source code and test cases, which requires differentiated retrieval, 2) between textual semantics and function call structures for accurate relevance assessment, and 3) between the retrieval and generation phases, which enables iterative feedback and refinement. To evaluate iCoRe, we integrate it with an LLM-based BRT generator and conduct a comprehensive evaluation on the SWT-bench Lite and TDD-bench Verified benchmarks. Experimental results show that our method achieves a Fail-to-Pass rate of 42.0% and 52.8% respectively, representing 19.7%-31.7% relative improvements over existing retrieval methods.

Figures

Figures reproduced from arXiv: 2604.19224 by Jialun Cao, Junyi Wang, Zhongxin Liu.

Figure 1
Figure 1. Figure 1: Motivating Example To address the above limitations, we propose a novel iterative Correlation-aware Retrieval approach, iCoRe, for automated BRT generation. For the limitation of the unified retrieval strategy, Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE186. Publication date: July 2026 [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Motivating Example (Continued from Figure [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of iCoRe. relevance; and 3) the powerful feedback correlation between the generation and retrieval phases, which enables iterative refinement. By being aware of these correlations, iCoRe systematically alleviates the failures of prior approaches. 3 Approach In this section, we introduce the framework of iCoRe. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Impact of Iteration Rounds. We conducted a sensitivity analysis on three key hyperparameters using the SWT-bench Lite dataset: the number of retrieval iterations, the number of rel￾evant tests provided as context, and the base weight used in function similarity calculation. Impact of Iteration Rounds. We define an itera￾tion as a cycle of generating a sketch BRT and using its feedback to refine the retriev… view at source ↗
Figure 5
Figure 5. Figure 5: Examples of Failure Cases The majority of failures occurred when the retrieved context was sufficient, but the generator failed to output a correct BRT. These failures stem from the limited reasoning capabilities of the model or the simple one-pass generation process. For example, for sympy__sympy-23413 ( Figure 5A), the generated test failed to reproduce the issue simply due to a missing numpy import. Wit… view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. EvoOtter: Evolutionary Reproduction Test Generator

    cs.SE 2026-07 conditional novelty 7.0

    EvoOtter combines evolutionary programming, rule-based mutants, successive halving, and batched LLM crossover to generate high-quality fail-to-pass bug reproduction tests cheaply.

Reference graph

Works this paper leans on

53 extracted references · 53 canonical work pages · cited by 1 Pith paper · 3 internal anchors

  1. [1]

    Toufique Ahmed, Jatin Ganhotra, Rangeet Pan, Avraham Shinnar, Saurabh Sinha, and Martin Hirzel. 2025. Otter: Generating Tests from Issues to Validate SWE Patches. InProceedings of the 42nd International Conference on Machine Learning. 752–771

    work page 2025

  2. [2]

    Toufique Ahmed, Jatin Ganhotra, Avraham Shinnar, and Martin Hirzel. 2026. Heterogeneous Prompting and Execution Feedback for SWE Issue Test Generation and Selection. InProceedings of the 48th International Conference on Software Engineering. 1262–1273

    work page 2026

  3. [3]

    Toufique Ahmed, Martin Hirzel, Rangeet Pan, Avraham Shinnar, and Saurabh Sinha. 2024. TDD-Bench Verified: Can LLMs Generate Tests for Issues Before They Get Resolved?CoRRabs/2412.02883 (2024). arXiv:2412.02883

    work page arXiv 2024

  4. [4]

    Amazon Web Services. 2024. Amazon Q. https://aws.amazon.com/q/. Accessed: September 1, 2025

    work page 2024

  5. [5]

    Moritz Beller, Niels Spruit, Diomidis Spinellis, and Andy Zaidman. 2018. On the Dichotomy of Debugging Behavior Among Programmers. InProceedings of the 40th International Conference on Software Engineering. 572–583

    work page 2018

  6. [6]

    Nicolas Bettenburg, Sascha Just, Adrian Schröter, Cathrin Weiss, Rahul Premraj, and Thomas Zimmermann. 2008. What makes a good bug report?. InProceedings of the 16th International Symposium on Foundations of Software Engineering. 308–318

    work page 2008

  7. [7]

    Jialun Cao, Wuqi Zhang, and Shing-Chi Cheung. 2024. Concerned with Data Contamination? Assessing Countermea- sures in Code Language Model.CoRRabs/2403.16898 (2024). arXiv:2403.16898

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  8. [8]

    Dong Chen, Shaoxin Lin, Muhan Zeng, Daoguang Zan, Jian-Gang Wang, Anton Cheshkov, Jun Sun, Hao Yu, Guoliang Dong, Artem Aliev, Jie Wang, Xiao Cheng, Guangtai Liang, Yuchi Ma, Pan Bian, Tao Xie, and Qianxiang Wang. 2024. CodeR: Issue Resolving with Multi-Agent and Task Graphs.CoRRabs/2406.01304 (2024). arXiv:2406.01304

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  9. [9]

    Mouxiang Chen, Zhongxin Liu, He Tao, Yusu Hong, David Lo, Xin Xia, and Jianling Sun. 2024. B4: Towards optimal assessment of plausible code solutions with plausible tests. InProceedings of the 39th International Conference on Automated Software Engineering. 1693–1705

    work page 2024

  10. [10]

    Ning Chen and Sunghun Kim. 2015. STAR: Stack Trace Based Automatic Crash Reproduction via Symbolic Execution. IEEE Transactions on Software Engineering41, 2 (2015), 198–220

    work page 2015

  11. [11]

    Runxiang Cheng, Michele Tufano, Jürgen Cito, José Cambronero, Pat Rondon, Renyao Wei, Aaron Sun, and Satish Chandra. 2025. Agentic Bug Reproduction for Effective Automated Program Repair at Google.CoRRabs/2502.01821 (2025). arXiv:2502.01821

    work page arXiv 2025

  12. [12]

    Ermira Daka and Gordon Fraser. 2014. A Survey on Unit Testing Practices and Problems. InProceedings of the 25th International Symposium on Software Reliability Engineering. 201–211

    work page 2014

  13. [13]

    Pouria Derakhshanfar, Xavier Devroey, Annibale Panichella, Andy Zaidman, and Arie van Deursen. 2020. Botsing, a Search-based Crash Reproduction Framework for Java. InProceedings of the 35th International Conference on Automated Software Engineering. 1278–1282

    work page 2020

  14. [14]

    Shihan Dou, Haoxiang Jia, Shenxi Wu, Huiyuan Zheng, Muling Wu, Yunbo Tao, Ming Zhang, Mingxu Chai, Jessica Fan, Zhiheng Xi, et al. 2026. What is wrong with your code generated by large language models? An extensive study. Science China Information Sciences69, 1 (2026), 112107

    work page 2026

  15. [15]

    Shahriar Golchin and Mihai Surdeanu. 2023. Time Travel in LLMs: Tracing Data Contamination in Large Language Models.CoRRabs/2308.08493 (2023). arXiv:2308.08493

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  16. [16]

    Roman Haas, Daniel Elsner, Elmar Juergens, Alexander Pretschner, and Sven Apel. 2021. How can manual testing processes be optimized? developer survey, optimization guidelines, and case studies. InProceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1281–1291

    work page 2021

  17. [17]

    Sungmin Kang, Juyeon Yoon, and Shin Yoo. 2023. Large Language Models are Few-Shot Testers: Exploring LLM-Based General Bug Reproduction. InProceedings of the 45th International Conference on Software Engineering. 2312–2323

    work page 2023

  18. [18]

    Lara Khatib, Noble Saji Mathews, and Meiyappan Nagappan. 2026. AssertFlip: Reproducing Bugs via Inversion of LLM-Generated Passing Tests. InProceedings of the 48th International Conference on Software Engineering. 3838–3847

    work page 2026

  19. [19]

    Pavneet Singh Kochhar, Xin Xia, and David Lo. 2019. Practitioners’ Views on Good Software Testing Practices. In Proceedings of the 41st International Conference on Software Engineering: Software Engineering in Practice. 61–70

    work page 2019

  20. [20]

    Fang Liu, Yang Liu, Lin Shi, Houkun Huang, Ruifeng Wang, Zhen Yang, and Li Zhang. 2024. Exploring and Evaluating Hallucinations in LLM-Powered Code Generation.CoRRabs/2404.00971 (2024). arXiv:2404.00971

    work page arXiv 2024

  21. [21]

    Zhongxin Liu, Kui Liu, Xin Xia, and Xiaohu Yang. 2023. Towards more realistic evaluation for neural test oracle generation. InProceedings of the 32nd International Symposium on Software Testing and Analysis. 589–600

    work page 2023

  22. [22]

    Niels Mündler, Mark Niklas Mueller, Jingxuan He, and Martin Vechev. 2024. SWT-Bench Lite Leaderboard. https: //swtbench.com/?results=lite. Accessed: 2025-09-12

    work page 2024

  23. [23]

    Niels Mündler, Mark Niklas Müller, Jingxuan He, and Martin Vechev. 2024. SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents. InProceedings of the 38th International Conference on Neural Information Processing Systems, Vol. 37. 81857–81887. Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE186. Publication date: July 2026. FSE186:22 W...

    work page 2024

  24. [24]

    Mathieu Nayrolles, Abdelwahab Hamou-Lhadj, Sofiène Tahar, and Alf Larsson. 2015. JCHARMING: A bug reproduction approach using crash traces and directed model checking. InProceedings of the 22nd International Conference on Software Analysis, Evolution, and Reengineering. 101–110

    work page 2015

  25. [25]

    Mohammad Masudur Rahman, Foutse Khomh, and Marco Castelluccio. 2020. Why are Some Bugs Non-Reproducible? : –An Empirical Investigation using Data Fusion–. InProceedings of the 36th International Conference on Software Maintenance and Evolution. 605–616

    work page 2020

  26. [26]

    Mozhan Soltani, Pouria Derakhshanfar, Annibale Panichella, Xavier Devroey, Andy Zaidman, and Arie van Deursen

    work page

  27. [27]

    InProceedings of the 10th International Symposium on Search-Based Software Engineering

    Single-objective versus multi-objectivized optimization for evolutionary crash reproduction. InProceedings of the 10th International Symposium on Search-Based Software Engineering. 325–340

    work page

  28. [28]

    Karen Sparck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval.Journal of documentation28, 1 (1972), 11–21

    work page 1972

  29. [29]

    Kuo-Chung Tai. 1979. The tree-to-tree correction problem.J. ACM26, 3 (1979), 422–433

    work page 1979

  30. [30]

    Dhaval Vyas, Thomas Fritz, and David Shepherd. 2014. Bug Reproduction: A Collaborative Practice Within Software Maintenance Activities. InProceedings of the 11th International Conference on the Design of Cooperative Systems. 189–207

    work page 2014

  31. [31]

    Nalin Wadhwa, Atharv Sonwane, Daman Arora, Abhav Mehrotra, Saiteja Utpala, Ramakrishna B Bairi, Aditya Kanade, and Nagarajan Natarajan. 2024. MASAI: Modular Architecture for Software-engineering AI Agents. InNeurIPS 2024 Workshop on Open-World Agents

    work page 2024

  32. [32]

    Xinchen Wang, Pengfei Gao, Xiangxin Meng, Chao Peng, Ruida Hu, Yun Lin, and Cuiyun Gao. 2025. AEGIS: An Agent- based Framework for Bug Reproduction from Issue Descriptions. InProceedings of the 33rd International Conference on the Foundations of Software Engineering. 331–342

    work page 2025

  33. [33]

    Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. 2025. OpenHands: An Open Platform for A...

    work page 2025

  34. [34]

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2025. Demystifying llm-based software engineer- ing agents.Proceedings of the ACM on Software Engineering2, FSE (2025), 801–824

    work page 2025

  35. [35]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

    work page

  36. [36]

    InProceedings of the 38th International Conference on Neural Information Processing Systems, Vol

    SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. InProceedings of the 38th International Conference on Neural Information Processing Systems, Vol. 37. 50528–50652

    work page

  37. [37]

    Zhiqiang Yuan, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, Xin Peng, and Yiling Lou. 2024. Evaluating and Improving ChatGPT for Unit Test Generation.Proc. ACM Softw. Eng.1, FSE, Article 76 (2024), 24 pages

    work page 2024

  38. [38]

    Chengming Zhang, Haoye Wang, Chuyang Xu, Jiakun Liu, Kui Liu, and Zhongxin Liu. 2026. Can test cases generated by large language models facilitate automated program repair?Empirical Software Engineering31, 3 (2026), 68

    work page 2026

  39. [39]

    Kaizhong Zhang and Dennis Shasha. 1989. Simple Fast Algorithms for the Editing Distance between Trees and Related Problems.SIAM J. Comput.18, 6 (1989), 1245–1262

    work page 1989

  40. [40]

    Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. AutoCodeRover: Autonomous Program Improvement. InProceedings of the 33rd International Symposium on Software Testing and Analysis. 1592–1604

    work page 2024

  41. [41]

    Yuanhe Zhang, Zhiquan Yang, Shengyi Pan, and Zhongxin Liu. 2025. Unit Test Update through LLM-Driven Context Collection and Error-Type-Aware Refinement. InProceedings of the 40th International Conference on Automated Software Engineering. 2504–2515

    work page 2025

  42. [42]

    Ziyao Zhang, Chong Wang, Yanlin Wang, Ensheng Shi, Yuchi Ma, Wanjun Zhong, Jiachi Chen, Mingzhi Mao, and Zibin Zheng. 2025. LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigation.Proc. ACM Softw. Eng.2, ISSTA, Article ISSTA022 (2025), 23 pages. Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE186. Publication date: July 20...

    work page 2025

  43. [43]

    These may include function names, class names, method names, variable names, file names, or other identifiers that directly Proc

    Identify and list only the code-related elements that are essential for searching the codebase and understanding or reproducing the bug. These may include function names, class names, method names, variable names, file names, or other identifiers that directly Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE186. Publication date: July 2026. FSE186:24 W...

    work page 2026

  44. [44]

    feature”, “error

    Exclude non-actionable or irrelevant terms, including: - Generic words like “feature”, “error”, “problem”. - User-defined class names or model names that are created within the example bug report but are unlikely to exist in the actual codebase (e.g.,A,B,Cin a sample model definition). - Any abstract or non-code terms that do not directly contribute to debugging

    work page

  45. [45]

    If an imported element is renamed using as, restore its original module path

    Preserve the exact names or formats of the elements as written in the bug report. If an imported element is renamed using as, restore its original module path. For example, if the bug report mentions import pandas as pd , and pd.DataFrame is used in the code, extract it aspandas.DataFrame

    work page

  46. [46]

    - Additionally, class names and function/method names should be ranked higher than code fragments

    Prioritize the extracted elements by their importance for reproducing the bug: - Elements that are most likely to be useful or necessary for understanding the bug should be ranked highest. - Additionally, class names and function/method names should be ranked higher than code fragments. Output format: Provide the extracted code elements as a list in the f...

    work page

  47. [47]

    Find and recommend some appropriate test functions based on the bug report

    work page

  48. [48]

    Rank the selected test functions in order of relevance, with the most relevant one first

    work page

  49. [49]

    path/to/test_file_x.py

    Output the name of test function and its file path. The result should contain at most {topk} test cases. ### Available Tools: - list_root(): Lists all files and directories inside the root test folder of the project. You may call this function first. -list_folder(path): Lists files and directories at the given path. - list_classes_and_functions(file_path)...

    work page 2026

  50. [50]

    Understand the Bug Report and explicitly identify the following information: Observed Behavior (OB): What is the actual, incorrect behavior produced by the current code? Expected Behavior (EB): What is the correct behavior that the user expects?

    work page

  51. [51]

    code should run successfully without errors

    Design the Test Logic Based on the Expected Behavior - Scenario A: If the Expected Behavior (EB) is the “code should run successfully without errors” Your test case should simply call the problematic code directly. DO NOT wrap the code in try...except, pytest.raises, or assertRaises. Reasoning: On the buggy code, the unexpected error (the OB) will be rais...

    work page

  52. [52]

    Test Requirement The test must be minimal and focused: Only reproduce the issue — do not add extra assertions that are unrelated to the bug

    work page

  53. [53]

    Focus on understanding the issue and its context

    Use Provided Information Effectively The input includes the following sections to assist in test creation: <issue>: Contains the bug report. Focus on understanding the issue and its context. <code>: Contains relevant code snippets. These can help you understand the project Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE186. Publication date: July 2026...

    work page 2026