Otter: Generating Tests from Issues to Validate SWE Patches

Avraham Shinnar; Jatin Ganhotra; Martin Hirzel; Rangeet Pan; Saurabh Sinha; Toufique Ahmed

REVIEW 3 major objections 5 minor 6 cited by

Otter: Generating Tests from Issues to Validate SWE Patches

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that it is feasible to generate correct tests from a software issue before any fix exists, that the Otter system does so in over a third of benchmark cases, and that these tests can vet patches produced by…

desk verdict Solid empirical systems paper with a genuinely useful new benchmark, but the headline SOTA claim rests on a model-version confound and the ablation design blurs component contributions. read the letter →

arxiv 2502.05368 v2 pith:W7GWPR5E submitted 2025-02-07 cs.SE cs.LG

Toufique Ahmed , Jatin Ganhotra , Rangeet Pan , Avraham Shinnar , Saurabh Sinha , Martin Hirzel This is my paper · ORCID

classification cs.SEcs.LG

keywords testgenerationfromissuesfail-to-passtestsSWEagentstest-drivendevelopmentLLMbenchmarkpatchvalidationcodecoverage

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Otter is a system that tries to write tests for a software issue before anyone has written the fix: given only the issue text and the current code, it produces a test that fails on the old code and would pass on the fixed code. The paper argues this is both practically useful for test-driven development and a way to check the output of automated patch generators, the SWE agents that compete on benchmarks such as SWE-bench Verified. To measure this, the paper builds TDD-Bench-Verified, a filtered 449-instance benchmark mined from SWE-bench Verified, and shows that Otter produces a fail-to-pass test in 31.4% of instances on a single run, rising to 37.0% for the five-prompt ensemble Otter++. When the same tests are used to filter candidate patches from 22 leading SWE-agent systems, precision rises to 65-92% while recall stays at 30-41%.

What carries the argument

The load-bearing pieces are the self-reflective action planner and the TDD-Bench-Verified scoring harness. The planner first localizes candidate test files and focal files and functions, then asks the LLM to propose a sequence of read, write, or modify actions; a rule-based validator checks every file and function name against the repository, and a reflect-and-improve loop lets the model revise its plan up to five times. The test generator then emits complete test functions, and a rule-based import fixer repairs hallucinated imports using the codebase. The benchmark defines tddScore as the product of a binary fail-to-pass factor and an adequacy factor, where adequacy is the fraction of added and deleted lines of the golden patch covered by the generated tests; this is what lets the paper say a test is correct only when it goes from failing to passing on the real fix.

What would settle it

Run Otter on the 44 SWE-bench Verified instances that TDD-Bench-Verified discarded because their own golden tests do not satisfy fail-to-pass, and compare the fail-to-pass rate; if Otter's rate is much higher there, the benchmark's conditioning on golden-test behavior is hiding part of what Otter can do. Alternatively, take instances where the repository contains a merged alternative fix that passes the golden tests but is not the golden patch, and check whether Otter's generated test passes that alternative patch.

Watch

Extended reading notes

Core claim

The central claim is that it is possible, before any code fix exists, to generate tests from an issue description that genuinely reproduce the issue and validate its resolution. The paper supports this with Otter, which combines LLM-guided localization and a self-reflective action planner with rule-based checks and repairs, and evaluates it on TDD-Bench-Verified, where a test counts only if it fails on the old code, passes on the golden post-fix code, and covers the lines the golden patch added or removed. By this metric Otter succeeds on 31.4% of the 449 benchmark instances with one run and 37.0% with the five-prompt Otter++ ensemble, beating the prior state of the art SWE-Agent+, which reached 19.2% on its own SWT-Lite set. The paper also claims the generated tests are useful beyond test generation itself: filtering SWE-agent patches with them lifts precision from 60.8% to 91.9% for the third-ranked SWE-bench Verified system, and to 65-92% across 22 leaderboard systems, at recall of 30-41%.

Load-bearing premise

The benchmark treats the single merged developer patch as the one and only correct fix: a generated test is judged correct only if it fails on the old code, passes on that specific golden new code, and covers that patch's changed lines, so any valid but different fix that would reject the generated test counts as a failure.

Editorial extensions

If this is right

In a test-driven workflow, a developer facing an open issue can start from an Otter-generated candidate test, which makes the expected behavior explicit before any fix is written.
For automated repair, Otter-generated tests act as a high-precision filter: 65-92% precision at 30-41% recall across 22 SWE-bench Verified systems, so a human can spend review effort on far fewer candidate patches.
TDD-Bench-Verified gives the community an execution-based yardstick for test-from-issue generators, replacing similarity or format checks with fail-to-pass behavior and line-coverage scores.
Because Otter-generated tests follow the repository's existing test framework and can be added to its CI, a merged patch can be checked against the generated test as a permanent regression test.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The benchmark's tie to the golden patch suggests that Otter is likely penalized for writing behavior-level tests that pass on valid alternative fixes but not on the merged implementation, so a test that captures the issue's contract could look like a failure by the benchmark's own lights.
The ensemble result, where five heterogeneous prompts beat five high-temperature samples, suggests that diversity of context matters more than sampling noise for LLM test generation, a principle worth porting to other code-generation tasks.
The same fail-to-pass filter could be inverted: when a generated test passes on a candidate patch but fails on the old code, it forms a minimal specification that could be fed back to a patch generator, turning a validator into a ranking signal beyond binary filtering.
TDD-Bench-Verified's 449 instances come from 12 popular Python repositories, so the 31.4% and 37.0% numbers are an upper bound for what Otter would achieve on less standard codebases or other languages; the action-planning steps are mostly language-agnostic, but the localization and import-fixing rules are Python-specific.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

What you should know about arXiv:2502.05368: it's a real contribution. The benchmark TDD-Bench-Verified is genuinely useful, and Otter's fail-to-pass rates of 31.4% (single) and 37.0% (ensemble) on that benchmark are believable. But the central "outperforms state-of-the-art" claim against SWE-Agent+ is weaker than it looks: Otter runs on GPT-4o while the comparators ran on GPT-4, and the ablations pull out several components at once. The right verdict is "important, needs a cleaner comparison," not "reject."

What's actually new: the task framing—generating tests from an issue before a fix exists—is well motivated, and Otter's self-reflective action planner with rule-based repair and heterogeneous ensemble is a legitimate new combination. The benchmark is the strongest part: 449 instances filtered from SWE-bench Verified with strict fail-to-pass and coverage checks, and the harness is transparent. The 22-system precision/recall filtering experiment is real work (49k containers), and the precision jump from 60.8% to 91.9% is plausible because it's a precision/recall tradeoff, not a free lunch.

Soft spots, in order of severity. The model confound is real. Table 3 compares Otter (GPT-4o) to Mündler et al.'s GPT-4 numbers, and their own Table 1 shows a 4.5-point gap between GPT-4o and Mistral-large with the same pipeline. So the 6.2-point gap on SWT-Lite could be mostly base-model luck. They need a same-model head-to-head. Second, the ablations labeled "without Focal Localization" and "without Test Localization" also drop the action planner (the asterisk says so), so they don't isolate the localizer's contribution. That's sloppy but fixable. Third, hyperparameters like ensemble size 5 were picked using the evaluation set; no held-out tuning. Minor but annoying. Fourth, no Otter code release—the benchmark is out, but the implementation isn't, so independent reproduction is limited. The benchmark's golden-patch uniqueness assumption is a real validity limit, but it applies equally to all systems compared, so it doesn't undermine the relative result; it just means the metric rewards tests that match the merged PR's behavior.

Who this is for: anyone working on SWE agents, test generation, or LLM-based software engineering benchmarks. It deserves a serious referee. I'd send it to review with the explicit ask that the authors add a GPT-4-controlled comparison, re-run the ablations one-component-at-a-time, and release the Otter code.

Referee Report

3 major / 5 minor

Summary. The paper proposes Otter, an LLM-based pipeline that generates tests from an issue description and the old code snapshot, before a patch exists. Otter combines file and function localization, a self-reflective action planner, and rule-based import fixing and validation. Otter++ ensembles five heterogeneous prompts and uses execution logs on the old code to select one candidate. The paper also introduces TDD-Bench-Verified, a 449-instance benchmark derived from SWE-bench Verified, with an execution-based fail-to-pass metric and a changed-line coverage metric. Experiments report 31.4% fail-to-pass for Otter and 37.0% for Otter++ with GPT-4o on TDD-Bench-Verified, 25.4% and 29.0% on SWT-Lite compared to SWE-Agent+'s 19.2%, and a filtering study showing that Otter-generated tests can raise the precision of SWE-agent patches at the cost of recall.

Significance. The benchmark release, the containerized evaluation harness, and the execution-based metric are concrete assets for the test-generation-from-issues community. The study is unusually thorough in its large-scale filtering experiment across 22 SWE-bench Verified leaderboard systems, its contamination analysis, and its cost reporting. If the headline comparison were model-controlled, the paper would represent a clear step forward in a practically relevant task. The main quantitative claims are transparent and reproducible from the released artifacts, but the central state-of-the-art comparison is currently confounded by a model-version difference, and the ablation analysis does not cleanly attribute the named components. These issues are fixable within the scope of a revision.

major comments (3)

[Section 6.4, Table 3] The comparison against Mündler et al. is confounded: Otter and Otter++ are run with GPT-4o, while all listed baseline approaches, including SWE-Agent+, are run with GPT-4. The paper's own Table 1 shows that model choice alone moves Otter's fail-to-pass rate by 4.5 points on TDD-Bench-Verified (31.4% for GPT-4o vs. 26.9% for Mistral-large), which is the same order of magnitude as the 6.2-point gap over SWE-Agent+ on SWT-Lite. Since every stage of Otter is an LLM call, the stronger base model may account for much or all of the reported advantage. The claim that Otter 'outperforms state-of-the-art systems' is load-bearing for the paper, so the authors should either run Otter with GPT-4 (or run SWE-Agent+ with GPT-4o) or explicitly reframe the result as a system-plus-model comparison rather than a method comparison.
[Section 6.3, Table 2] The ablation rows labeled 'without Focal Localization*', 'without Test Localization*', and 'without Focal & Test Localization*' are marked as not followed by action planning, so each of those rows differs from the Otter (T1) row in two ways: the planner is removed and a localizer is removed. As a result, the text 'all the components contribute' and the reported percentage losses do not isolate the contribution of the named component. For example, comparing T1 (141) with T3 (115) conflates removing the planner and removing focal localization. The pairwise comparisons among T2, T3, T4, and T5 (which hold the planner fixed) should be the basis for the localization ablation claims, or the authors should run controlled ablations that remove one localizer while keeping the self-reflective planner intact.
[Section 5.2, Section 5.3, and Section 7] The benchmark metric assumes that the golden developer patch is the unique correct new code. The failToPass definition rewards a generated test only if it fails on cold and passes on the golden new code, and the adequacy metric measures coverage of exactly the lines added and deleted by that patch. If a different valid fix exists that would not pass the generated test, the generated test is counted as correct even though it would reject an acceptable resolution. Additionally, the dataset construction filters out 44 instances whose golden tests do not satisfy the same fail-to-pass or coverage criteria, so the 449-instance evaluation set is conditioned on golden-test behavior. The paper mentions human annotation in SWE-bench Verified but does not state this uniqueness and conditioning limitation in Section 7. The authors should add an explicit discussion of this validity boundary and, ideally, a small manual or synthetic study of whether generated fail-to-pass tests would also accept plausible alternative patches.

minor comments (5)

[Section 7 and Section 5.3] The Limitations section states that SymPy coverage information was unreliable and therefore coverage was removed from the final tddScore metric for SymPy instances. Since this changes the metric definition for about 15% of the dataset, the statement should be moved into Section 5.3 and repeated in the captions of Tables 1 and 2, where tddScore is compared.
[Section 6.7] The sentence 'except for one system where the precision increased by 22% to 167%' is confusing, since precision cannot exceed 100%; the authors should specify whether 167% is a relative increase over the original value and clarify the units.
[Section 6.7, Figure 6] The text reports precision/recall values and the 60.8%-to-91.9% example on what appears to be the 449-instance TDD-Bench-Verified subset, but this is not stated explicitly. The authors should clarify that the filtering results are computed on the 449-instance subset, not on the full SWE-bench Verified leaderboard.
[Section 6.6] In the paragraph on prompts complementarity, the text says 'using the GPT-4 model' when the experiments in this paper use GPT-4o (gpt-4o-2024-08-06); please correct the model name.
[Section 6.2, Table 1] The zero-shot baseline is given only the repository name and issue description, whereas Otter is given the full old code snapshot. This is a reasonable baseline, but the discrepancy should be stated in the table caption so readers do not interpret the comparison as an ablation of the full input.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Otter's outputs are evaluated against external golden patches, and its selection heuristics use only the problem input (old code), not the target new code.

full rationale

The derivation chain is self-contained. Otter receives only the issue description and the old code snapshot (Section 2), and its generated tests are scored by executing them on the old code and on the golden new-code patch mined from the merged PR (Section 5.1, Figure 3). The golden patch and golden tests are external to Otter's inputs: cnew is explicitly unavailable to genTests, so no output is being constructed from the answer. TDD-Bench-Verified's filtering (Section 5.2) drops instances whose golden tests do not fail-to-pass and do not cover the golden patch; this is benchmark curation, not a fitted input, and it conditions the evaluation set without defining Otter's outputs. Otter++'s final candidate selection uses execution logs on cold, which is part of the problem input, and merely prefers assertion failures over passes or errors; this is a heuristic on the input, not on the hidden new code. The comparison with Mündler et al. (Table 3) uses GPT-4o for Otter and GPT-4 for SWE-Agent+, which is a possible model-version confound and a validity concern, but it is not a circular reduction. There are no load-bearing self-citations: the paper's references are external prior works, and no uniqueness theorem or ansatz is imported from the authors' own prior publications. No equation reduces to another by construction, and no fitted parameter is renamed as a prediction. The golden-patch-as-reference assumption is a benchmark validity limitation, but it applies equally to all compared systems and does not make Otter's reported results circular.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the quality of the benchmark, not on fitted equations. The main free parameters are pipeline hyperparameters (retrieval count, iteration cap, ensemble size, tie-break order), several chosen via internal scaling experiments rather than a held-out split. The axioms are domain assumptions about the golden patch as oracle, coverage as adequacy, and representativeness of the filtered 449-instance set.

free parameters (4)

max retrieved files (localizer k) = 10
The localizer retrieves at most 10 file names per LLM call, chosen from initial dev-set findings on SWE-bench; this bounds context size (Section 4.1).
max plan iterations = 5
The self-reflective planner repeats the 'reflect and improve plan' step at most 5 times; Figure 4 shows most instances finish in 1-2 turns (Section 4.1).
Otter++ ensemble size = 5
Otter++ combines Otter's output with 4 additional heterogeneous prompts; Appendix C and Table 9 show the incremental gain at 5 candidates is 0.5-5.6%, and this size is used in the headline 37.0% result.
tie-breaking order in Otter++ selection = Otter, both localizers, test localizer only, focal localizer only, neither
When multiple tests fail on cold, the ranker picks a test according to a hand-chosen ordering that favors prompts with more context (Section 4.2).

assumptions (4)

domain assumption The golden developer patch is the unique correct resolution of each issue.
failToPass in Section 5.1 requires generated tests to pass on the golden new code and fail on old code, so any valid alternative fix that would not pass the generated test is scored as wrong.
domain assumption Line coverage of lines added and deleted by the golden patch is an adequate proxy for test usefulness.
The adequacy factor in Section 5.3 defines tddScore as failToPass times changed-line coverage; the paper cites SWE-Bench+ as motivation but does not independently validate this proxy.
domain assumption The 449 filtered SWE-bench Verified instances are representative of real-world issue-to-test scenarios.
TDD-Bench-Verified inherits representativeness from SWE-bench Verified and then filters out instances whose golden tests do not fail-to-pass or have zero coverage (Section 5.2), which may exclude hard or ambiguous issues.
domain assumption The Python coverage package reports accurate line coverage for the 12 repositories.
The harness integrates the coverage package (Section 5.3); the authors later found SymPy coverage unreliable and dropped it from tddScore (Section 7).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Otter: Generating Tests from Issues to Validate SWE Patches." pith.science (2026). https://pith.science/paper/W7GWPR5E

@misc{pith2026250205368,
  author       = {Pith},
  title        = {Pith review of: Otter: Generating Tests from Issues to Validate SWE Patches},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W7GWPR5E}},
  note         = {Machine review of arXiv:2502.05368}
}

read the original abstract

While there has been plenty of work on generating tests from existing code, there has been limited work on generating tests from issues. A correct test must validate the code patch that resolves the issue. This paper focuses on the scenario where that code patch does not yet exist. Doing so supports two major use-cases. First, it supports TDD (test-driven development), the discipline of "test first, write code later" that has well-documented benefits for human software engineers. Second, it also validates SWE (software engineering) agents, which generate code patches for resolving issues. This paper introduces TDD-Bench-Verified, a benchmark for generating tests from issues, and Otter, an LLM-based solution for this task. Otter augments LLMs with rule-based analysis to check and repair their outputs, and introduces a novel self-reflective action planner. Experiments show Otter outperforming state-of-the-art systems for generating tests from issues, in addition to enhancing systems that generate patches from issues. We hope that Otter helps make developers more productive at resolving issues and leads to more robust, well-tested code.

Figures

Figures reproduced from arXiv: 2502.05368 by the authors.

**Figure 1.** Overview of Otter. to our current setup because we have multiple tests but no code patch. CodeMonkeys (Ehrlich et al., 2025) is another relevant work that iteratively and jointly improves a patch with a test, which is interesting and complementary to Otter. Unlike our work, their experiments do not evaluate the failto-pass rate nor the coverage of generated tests. 4. Methodology This section describes three solutio… view at source ↗

**Figure 2.** Overall flow of TDD-bench dataset filtering starting from SWE-bench verified. test patch (candidate or golden) golden code patch fail/pass (expect fail ) old code cold git apply code with tests cold y resolve class and location tests y runTests and cov git apply golden new code ĉnew fail/pass (expect pass ) runTests and cov [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Evaluation harness for TDD-Bench-Verified. code cold. Next, the harness analyzes the resulting code cold ⊕ y to resolve the contributed test functions y. The harness then executes y while avoiding running other tests that occur in the same file but were not contributed in the patch. This yields test results, including coverage achieved on the old code. At least one of these tests should fail, indicating that the tes… view at source ↗

Figures from the paper (18 more)

**Figure 4.** Figure 4: Number of turns taken in “reflect and improve plan” step. 6.6. Characteristics of the Generated Tests Coverage of the generated tests. We compared the coverage achieved by the Otter-generates tests and the golden tests written by developers. We observe that, for the f…

**Figure 5.** Figure 5: Number of instances with fail-to-pass tests generated by different prompts [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 8.** Figure 8: Cumulative percentage of Otter-generated new tests, using GPT-4o, with maximum similarity less than the similarity value shown on the x-axis. 7. Limitations One limitation of TDD-Bench-Verified is that it is mined from 12 popular Python repositories, so our findings m…

**Figure 6.** Figure 6: Precision of all 22 systems from the leaderboard. Recall is also mentioned at the top of each bar. 2019 2020 2021 2022 2023 Year 0 20 40 60 80 100 Number of Samples 38.3% 24.2% 24.4% 31.9% 26.5% Total Sample Fail-to-Pass Sample [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

**Figure 7.** Figure 7: Sample distribution of total and fail-to-pass tests by year. performance by the model. Test similarity. We conducted another experiment following the approach of Schafer et al. ¨ (2024). We compute the similarity score between the generated test from Otter and the mos…

**Figure 9.** Figure 9: Example test patch with one contributed test. Although the test file name test regression.py and test name test missing data are available in this diff, the class TestPolyFit enclosing test missing data is missing. By applying the test patch to the base commit and pars…

**Figure 10.** Figure 10: Cumulative percent of Otter generated all tests, using GPT-4o, with maximum similarity less than the similarity value shown on the x-axis. 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Maximum Similarity 0 20 40 60 80 100 Cumulative Percentage of Test Cases with Maximum…

**Figure 11.** Figure 11: Cumulative percent of Otter generated modified tests, using GPT-4o, with maximum similarity less than the similarity value shown on the x-axis. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]

**Figure 12.** Figure 12: Cumulative percent of Otter generated fail-to-pass tests, using GPT-4o, with maximum similarity less than the similarity value shown on the x-axis. 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Maximum Similarity 0 20 40 60 80 100 Cumulative Percentage of Test Cases wit…

**Figure 13.** Figure 13: Cumulative percent of Otter generated non fail-to-pass tests, using GPT-4o, with maximum similarity less than the similarity value shown on the x-axis. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]

**Figure 14.** Figure 14: Prompt (1 of 2) for focal function localizer. Suppose you are a very experienced developer. An issue has been created, and you need to choose the best possible files and functions to make the changes. You will be given the following pieces of information. Please write…

**Figure 15.** Figure 15: Prompt (2 of 2) for focal function localizer. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_15.png]

**Figure 16.** Figure 16: Prompt (1 of 2) for test function localizer. Suppose you are a very experienced developer. An issue has been created, and you need to choose the best possible test file to write a fail-to-pass test. You will be given three pieces of information. Please write down the …

**Figure 17.** Figure 17: Prompt (2 of 2) for test function localizer. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_17.png]

**Figure 18.** Figure 18: Prompts for “make an initial plan” step. Suppose you are a very experienced developer. An issue has been created, and you need to write a test to ensure the issue has been resolved. You were given the following pieces of information. You proposed a set of actions. We …

**Figure 19.** Figure 19: Prompts for “reflect and improve the plan” step. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_19.png]

**Figure 20.** Figure 20: Prompt for modifying existing test. Suppose you are a very experienced developer. An issue has been created, and you need to write a fail-to-pass test to ensure the issue has been resolved. After reading the issue related and existing tests, you decided to write a new…

**Figure 21.** Figure 21: Prompt for writing new test. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_21.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Validation Evidence in LLM Repair Agents: How Much of What Passes Actually Tests the Bug?
cs.SE 2026-07 conditional novelty 7.0 of 10

By replaying each validation command on buggy, candidate, and gold-fix code, this study finds 46% of positive checks do not discriminate the bug, and a reminder alone explains part of the feedback benefit.
Semantic Drift in Bug Resolution: How Behavioral Signals Propagate from Reports to Tests and Patches
cs.SE 2026-07 conditional novelty 7.0 of 10

Behavioral signals in bug reports propagate only partially into tests and fixes; alignment is measurable but representation-dependent, and LLM judges are systematically optimistic versus human ratings.
FaultLine: Automated Proof-of-Vulnerability Generation Using LLM Agents
cs.SE 2025-07 conditional novelty 7.0 of 10

A staged LLM agent that reasons about data flow and branch conditions generated valid proof-of-vulnerability tests for 16 of 100 Java, C, and C++ projects, versus 9 for the CodeAct 2.1 baseline.
ExplainBench: Evaluating Code Explanations from Agents
cs.SE 2026-07 conditional novelty 6.0 of 10

Agent explanation quality can be scored automatically with LLM questionnaires, and this score ranks agents differently from SWE-bench Verified.
DiffTestGen: Change-Directed LLM-Based Testing for Exposing Behavioral Differences
cs.SE 2026-07 conditional novelty 6.0 of 10

Change-directed LLM-based testing with call-graph access information and union-coverage feedback exposes behavioral differences in 78.2% of 463 Python pull requests.
Understanding Automated Program Repair Agents Through the Lens of Traceability: An Empirical Study
cs.SE 2025-06 accept novelty 6.0 of 10

An empirical study of five LLM-based program repair agents shows they excel on simple fixes but overfit tests and struggle with test generation, regression selection, and advanced tooling.

Reference graph

Works this paper leans on

17 extracted references · 17 canonical work pages · cited by 6 Pith papers

[3]

Write one file name in each line

List of file(s), that will be updated or added for addressing the pull request: $LIST_FILES Please write down the most suitable 10 file names from the list above based on the issue description. Write one file name in each line. Do not add any number or index. Also do not add any explanation. Answer: Figure 14.Prompt (1 of 2) for focal function localizer. ...

work page
[6]

You can choose multiple functions from different files but keep the list as short as possible

List of file(s) and function(s), that will be updated or added for addressing the pull request: $LIST_FILE_FUNCTION Please write down the most suitable program files and relevant function names based on the issue description. You can choose multiple functions from different files but keep the list as short as possible. Filename should be written between <...

work page
[9]

Write one file name in each line

List of file(s), that will be updated or added for writing fail-to-pass test: $LIST_FILES Please write down the most suitable 10 test file names from the list above based on the issue description. Write one file name in each line. Do not add any number or index. Also do not add any explanation. Answer: Figure 16.Prompt (1 of 2) for test function localizer...

work page
[12]

You can choose multiple test from different files but keep the list as short as possible

List of file(s), that will be updated or added for writing fail-to-pass test: $LIST_FILE_FUNCTION Please write down the most suitable test files and relevant function names based on the issue description. You can choose multiple test from different files but keep the list as short as possible. Filename should be written between <Filename> & </Filename> ta...

work page
[15]

Test file(s) and function(s), that will be updated or added for testing the pull request : $LIST_TEST_FILE_FUNCTION

work page
[16]

make an initial plan

Program file(s) and functions(s), that will be updated or added for addressing the issue : $LIST_FOCAL_FILE_FUNCTION Now make an action list to write a fail-to-pass test. You can read necessary test and focal function to finish the task. You canonly perform read actions. Read: You can read a focal function/test given a file. You can read only the current ...

work page
[17]

Valid Action(s): $VALID_ACTION

work page
[18]

We are sharing the following information

Invalid Action(s): $INVALID_ACTION Now, based on the action rule, you can modify the action provided with the information above (e.g., replace wrong or None fun ction name and file name w ith existing functions and files). We are sharing the following information

work page

Show all 17 references

[21]

Test file(s) and function(s), that may be relevant to the issue: $LIST_TEST_FILE_FUNCTION
[22]

Write"/"Modify

Program file(s) and functions(s), that will be updated or added for addressing the issue : $LIST_FOCAL_FILE_FUNCTION Action Rules to Follow: Now make an action list to write a fail-to-pass test. You can read necessary test and program files to finish the task. You have 3 possi...
[24]

Issue Description: $ISSUE DESCRIPTION
[27]

Relevant Function

Name of the function to be modified: $NAME Writing Rule: Write the complete test between the <COMPLETE_FUNC> & </COMPLETE_FUNC> tags. No need to write any explanation or add any class within the tags. Write down the fail-to- pass test. T o keep the indentation intact, start wr...
[28]

Repository name: $REPO_NAME
[29]

Issue Description: $ISSUE_DESCRIPTION
[30]

Relevant Function: $FUNCTION
[31]

Test File imports and structure: $FILE_FUNCTION
[32]

Please consider class name and inden tation provided in 4 (Test File imports and structure) while proposing the name

Name of the function to be written: $NAME Writing Rule: Please write the name of the prior function, after which the new function will be added. Please consider class name and inden tation provided in 4 (Test File imports and structure) while proposing the name. Prior function...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [3]

Write one file name in each line

List of file(s), that will be updated or added for addressing the pull request: $LIST_FILES Please write down the most suitable 10 file names from the list above based on the issue description. Write one file name in each line. Do not add any number or index. Also do not add any explanation. Answer: Figure 14.Prompt (1 of 2) for focal function localizer. ...

work page

[2] [6]

You can choose multiple functions from different files but keep the list as short as possible

List of file(s) and function(s), that will be updated or added for addressing the pull request: $LIST_FILE_FUNCTION Please write down the most suitable program files and relevant function names based on the issue description. You can choose multiple functions from different files but keep the list as short as possible. Filename should be written between <...

work page

[3] [9]

Write one file name in each line

List of file(s), that will be updated or added for writing fail-to-pass test: $LIST_FILES Please write down the most suitable 10 test file names from the list above based on the issue description. Write one file name in each line. Do not add any number or index. Also do not add any explanation. Answer: Figure 16.Prompt (1 of 2) for test function localizer...

work page

[4] [12]

You can choose multiple test from different files but keep the list as short as possible

List of file(s), that will be updated or added for writing fail-to-pass test: $LIST_FILE_FUNCTION Please write down the most suitable test files and relevant function names based on the issue description. You can choose multiple test from different files but keep the list as short as possible. Filename should be written between <Filename> & </Filename> ta...

work page

[5] [15]

Test file(s) and function(s), that will be updated or added for testing the pull request : $LIST_TEST_FILE_FUNCTION

work page

[6] [16]

make an initial plan

Program file(s) and functions(s), that will be updated or added for addressing the issue : $LIST_FOCAL_FILE_FUNCTION Now make an action list to write a fail-to-pass test. You can read necessary test and focal function to finish the task. You canonly perform read actions. Read: You can read a focal function/test given a file. You can read only the current ...

work page

[7] [17]

Valid Action(s): $VALID_ACTION

work page

[8] [18]

We are sharing the following information

Invalid Action(s): $INVALID_ACTION Now, based on the action rule, you can modify the action provided with the information above (e.g., replace wrong or None fun ction name and file name w ith existing functions and files). We are sharing the following information

work page