Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

YATE: The Role of Test Repair in LLM-Based Unit Test Generation

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Repairing broken LLM tests lifts coverage 32 percent.

desk verdict Solid empirical paper: repairing near-miss LLM tests buys real coverage and mutation gains, but single-run stochastic evaluation and no artifact leave the exact effect sizes less certain than the prose claims. read the letter →

arxiv 2507.18316 v1 pith:T5TACAHF submitted 2025-07-24 cs.SE

Michael Konstantinou , Renzo Degiovanni , Jie M. Zhang , Mark Harman , Mike Papadakis This is my paper · ORCID
classification cs.SE
keywords LLM-basedunittestgenerationrepairnear-misstestscompilationerrorfixingoraclecoverage-guidedaugmentationmutationtesting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that 'near-miss' unit tests — tests an LLM generates that fail to compile or fail to pass — should be repaired rather than thrown away, because they already target the program logic that needs testing. It presents YATE, a four-part repair pipeline: chain-of-thought test planning, compilation fixing that retrieves class dependencies and method signatures from the project, oracle fixing that adjusts failing assertions to match observed behavior, and branch-coverage feedback for extra tests. In an evaluation on six Java projects with four LLMs, YATE's tests cover on average 32.06 percentage points more lines and kill 21.77 percentage points more mutants than a plain LLM-based method, and it beats four published generators on line, branch, and mutation metrics at comparable or lower LLM-call cost. The authors conclude that repair, especially compilation-relevant feedback, is what unlocks the value of LLM-generated tests.

What carries the argument

The load-bearing mechanism is the repair loop. Rather than treating a compiler error or failing assertion as a dead end, YATE uses rule-based static analysis on the project repository to obtain the missing context — imports, constructor signatures, classes implementing an interface, method bodies on the call graph — and feeds that context, plus the error log, back to the LLM in a new prompt. A second repair layer fixes oracles by substituting observed runtime values into assertions and by converting exception expectations, with a final LLM pass for complex values; a third pass re-prompts with uncovered branches. The core finding is that the combination of all four components outperforms any single component or their removal, and that compilation-relevant dependency feedback is the most valuable ingredient.

What would settle it

Run YATE and the plain baseline ten times over the same classes and compare the coverage and mutation distributions: if the 32-percentage-point line-coverage advantage and the 21.77-percentage-point mutation advantage fall within run-to-run noise, the central claim does not hold.

Watch

Extended reading notes

Core claim

YATE's central claim is that the discarded invalid outputs of an LLM are a recoverable source of testing value: when the model produces a test that almost compiles or almost passes, the near-miss contains the targeting logic needed to test the class, and repairing it is cheaper and more effective than discarding and re-prompting blindly. The paper operationalizes this with a workflow that (1) has the LLM first produce a coverage-oriented test plan, (2) repairs compilation errors by statically analyzing the repository, directly fixing imports and constructors, walking the method call graph for dependencies, and re-prompting with the retrieved class and method code, (3) fixes failing oracles by replacing assertion values with observed outputs and by inverting or generating exception assertions, and (4) prompts again with uncovered branches. On 393 class/method problems across six open-source Java projects, YATE reaches 51% average branch coverage, reporting about 22 percentage points higher line coverage, 20 percentage points higher branch coverage, and 20 percentage points more mutants killed than four published methods, with class-level prompting needing about five times fewer LLM calls than method-level prompting. The ablation study supports the paper's emphasis on repair: removing compilation fixing costs the most, and dependency-aware compilation feedback outweighs bare coverage feedback.

Load-bearing premise

The comparison treats a single run of each LLM-based method on each class as a trustworthy measurement of that method's performance, even though LLM generation is stochastic; if repeated runs shift coverage by even a few percentage points per class, the reported 20–32% margins and the significance tests could change.

Editorial extensions

If this is right

  • Repairing near-miss tests directly converts wasted LLM output into passing tests: YATE raises the passing rate from about 50% for plain prompting to 90.18% at class level and 82.96% at method level, producing thousands more usable tests.
  • Class-level prompting is the cost-effective operating point for LLM test generation: it needs about five times fewer LLM calls than method-level prompting while losing only about 5% coverage, and it still beats all baselines.
  • Compilation-relevant feedback matters more than coverage feedback: replacing YATE's dependency-aware compilation fixing drops branch coverage the most, suggesting that generators should invest in dependency retrieval before branch guidance.
  • The gains are not tied to one model: YATE outperforms plain prompting across GPT-4o-mini, GPT-4.1, DeepSeek-V3, and Codestral, so the repair mechanism transfers across LLMs.
  • Combining class-level and method-level suites yields higher coverage than either alone, indicating that the two granularities supply complementary contextual information.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test this, but a natural extension is that the repair components could be grafted onto other generators: if the value lies in recovering near-misses, then feeding dependency code and error logs to any LLM test generator's output could produce stacked gains rather than redundant ones.
  • Because the compilation-fixing step retrieves project code by walking the call graph, it is effectively a retrieval-augmented generation strategy; an isolated ablation that supplies the same retrieved context in the initial prompt rather than after a failure would reveal how much of the gain comes from the repair framing versus the added context alone.
  • The stochastic one-run evaluation implies a methodological corollary: before relying on these margins, practitioners should estimate run-to-run variability, since a few percentage points of coverage noise could change which configurations are actually distinguishable.
  • If the missed-opportunity view is right, test generators should be designed to provoke and collect informative invalid tests rather than to avoid them, and cost models should count the price of repairing a near-miss against the price of a fresh generation attempt.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents YATE, an LLM-based unit test generation technique built around four components: chain-of-thought initial generation, compilation-error fixing via static analysis and re-prompting, oracle fixing using execution outcomes, and coverage-based test augmentation. The authors evaluate YATE on six Java projects (393 classes) with four LLMs, comparing it against a plain LLM prompting baseline and four published tools (HITS, SymPrompt, TestSpark, CoverUp) using line coverage, branch coverage, mutation score, and number of LLM calls as the cost metric. The central claims are that YATE improves line coverage by 32.06 percentage points and mutation score by 21.77 percentage points over plain LLM generation at class level, and that it outperforms all four baseline tools by roughly 20 percentage points in coverage and mutation score at comparable cost.

Significance. If the reported results hold, YATE would be a useful contribution to LLM-based unit test generation: it provides evidence that repairing near-miss invalid tests, rather than discarding them, can substantially improve the effectiveness of LLM-generated test suites, and it identifies class-level prompting as a cost-effective configuration. The study has several strengths: a standard benchmark design with six projects and 393 classes, a four-component ablation, a multi-LLM sensitivity analysis, use of Mann-Whitney U tests and VDA effect sizes, and explicit measurement of LLM-call cost. However, the stochastic nature of the LLM pipeline means the current single-run evaluation is not sufficient to support the strength of the headline claims, and the paper's reporting of percentage-point differences as percentage improvements needs correction.

major comments (3)
  1. [§5.2, Tables 4, 7, 9] The effectiveness evaluation is based on a single execution of each stochastic LLM pipeline per class; no repeated runs, confidence intervals, or variance estimates are reported for any coverage or mutation value, yet Tables 5 and 8 report Mann-Whitney U p-values and VDA effect sizes computed from these single-sample values. The headline advantages (e.g., 32.06 percentage points line coverage over plain, and roughly 20/20/20 over HITS) are therefore point estimates. This is not a purely hypothetical concern: in RQ4 (Table 7), GPT-4.1 YATE exceeds its plain baseline by only 5.31 pp line coverage (52.61 vs 47.30), and in RQ5 (Table 9), YATE-C branch coverage on CH (37.02%) is below HITS (44.3%). If repeated runs alter per-class values by even a few percentage points, the 'consistently' and 'significantly' claims in Findings 3 and 6 could change. Please add repeated runs or, at minimum, a perturbation-based sensitivity analysis, and report distributions or confidence intervals for all headline numbers.
  2. [Abstract and §5.2] The phrase 'cover on average 32.06% more lines and kill 21.77% more mutants' is misleading. These numbers are percentage-point differences in average coverage: from Table 4, average class-level line coverage is 64.94% for YATE vs 32.88% for plain, and average mutation score is 43.98% vs 22.22%. The corresponding relative improvements on the averaged values are approximately 97.5% and 97.9%, while the 131.45%, 143.02%, and 156.07% figures in Section 5.2 appear to be averages of per-project relative improvements. Please report absolute coverage values, percentage-point differences, and relative improvements with unambiguous labels throughout, and correct the abstract so that '32.06% more lines' does not overstate the relative gain.
  3. [§5.2, Finding 4, Table 5] The claim that combining class-level and method-level YATE test suites 'consistently yields significantly higher coverage' is not supported by any statistical test. Table 5 only compares YATE(C) versus YATE(M), and no p-values or effect sizes are reported for the Y-Comb column against either constituent. The Y-Comb advantages in Table 4 are also reported without variance estimates. Please add the missing statistical comparisons or soften the claim to an observation about the observed totals and averages.
minor comments (6)
  1. [§7] The validity section states that 'we use ChatUniTest, a widely used plugin with the implementation of all algorithms we compare with,' but HITS, SymPrompt, TestSpark, and CoverUp are separate tools, not implementations inside ChatUniTest. Please correct this sentence or clarify what was actually reused from existing tooling.
  2. [Tables 2–9] Several tables distinguish 'Total' (aggregated) from 'Average' (unweighted per-project) rows, but the text sometimes cites one without specifying which. Please define both quantities explicitly in Section 4.4 and refer to them consistently in the findings.
  3. [§3.4–3.6, §4.3] Several configuration choices that directly control the pipeline are stated as fixed defaults without sensitivity analysis or full specification: the maximum number of oracle-fixing iterations (3), the call-graph depth used in compilation fixing, the single extra coverage-augmentation cycle, the exception-oracle avoid list, and the LLM temperature and decoding parameters. Please report the exact settings used and, where feasible, include a limited sensitivity analysis for the most consequential parameters.
  4. [§4.2 and §7] The seen/unseen cutoff date is justified for GPT-4o-mini only, but RQ4 uses other models with different training data; the paper acknowledges this in Section 4.2 but does not revisit the data-leakage threat for those models in the validity discussion. Please address this explicitly.
  5. [Throughout] Please fix typographical and wording issues: 'utilises' in the abstract, 'adn' in Section 3.5.1, 'BGP' in Figure 3, 'askes' in Section 3.6, 'threads' in Section 7, and 'living out' in Section 2.
  6. [General] No artifact or replication instructions are provided. Given the complexity of the YATE pipeline and the number of configurable components, a public artifact with the scripts, prompts, and parameter settings would substantially strengthen the paper's reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: YATE's effectiveness claims are empirical measurements against external benchmarks and not derivations from the method's own definitions.

full rationale

The paper makes no mathematical derivation; its claims are empirical comparisons (Tables 2-10) against an LLM-Plain baseline and four published tools on six external Java projects. The oracle-fixing phase (Section 3.5) replaces failing expected values with runtime outcomes and inverts exception oracles; this makes repaired tests pass by construction, but the paper presents this as the mechanism being evaluated, not as a prediction, and the measured pass, coverage, and mutation scores are outcomes of executing the resulting suites against external subjects. Coverage and mutation scores are not defined in terms of the method's own outputs, and no fitted parameter is renamed as a prediction. Self-citations (e.g., [1], [4], [13], [20]) provide background context and metrics, not load-bearing justification, and no uniqueness theorem or ansatz is imported from the authors' prior work. The single-run stochastic evaluation and thin margins in some cells (e.g., GPT-4.1 YATE vs. plain, YATE-C branch on CH) are threats to statistical validity, not circularity, and should be assessed separately.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

YATE introduces no new scientific entities; it configures existing LLMs and static analysis. The listed hand-chosen hyperparameters (iteration limits, call graph depth, avoid list, cutoff date, decoding settings) shape the reported results and are engineering choices rather than fitted constants. The main unstated assumptions are the single-run stability of stochastic LLM outputs and the accuracy of the external analysis tools.

free parameters (6)
  • Oracle fixing max iterations = 3 (default)
    Section 3.5.3: 'By default, the maximum number of iterations is 3.' This hand-chosen limit controls how many LLM repair attempts are made and affects both cost and repair success.
  • Compilation fixing call graph depth = unspecified ('up to a certain depth')
    Sections 3.4.2 and 3.4.4 describe walking the method call graph 'up to a certain depth' to fetch dependency classes; the exact depth is not specified, which affects context size and repair outcomes.
  • Coverage augmentation cycles = 1 (stops after second generation cycle)
    Section 3.6: coverage augmentation starts a new process and 'The process stops at the end of this (second) generation cycle.' This limits how many coverage feedback rounds are attempted.
  • Exception oracle avoid list = configurable; default includes incorrect mock usage
    Section 3.5.2: YATE has a configurable list of exception oracles to avoid, including exceptions related to incorrect mock usage. The list shapes which failing tests become exception-based oracles.
  • Seen/unseen cutoff date = October 1, 2023
    Section 4.2: projects created after this date are labeled unseen for GPT-4o-mini. This is a hand-chosen assumption about training data boundaries and affects the leakage analysis.
  • LLM temperature and decoding parameters = not reported
    The paper does not report sampling temperature or other decoding settings for the four LLMs; these unstated choices can affect test generation quality and the stability of the results.
assumptions (5)
  • domain assumption Spoon's static analysis and method call graph construction are accurate
    Section 7 (Internal Validity): YATE relies on Spoon for code analysis and call graph construction. The authors state they manually verified accuracy, but the tool's correctness remains an external dependency that could affect the compilation fixing component.
  • domain assumption LLM outputs from a single run are representative of typical performance
    No repeated runs are reported; all coverage and mutation tables treat one output per class as the measurement. Stochastic variance is not quantified, so the comparisons assume single-sample stability.
  • domain assumption Jacoco and PIT provide valid proxies for test effectiveness
    Section 4.4: line and branch coverage from Jacoco and mutation score from PIT are used as the effectiveness metrics. These are standard but remain proxy measurements of test quality.
  • domain assumption The six selected projects are representative of Java open-source software
    Section 4.2: projects are drawn from ChatUniTest, HITS, and Gitbug-Java datasets. The authors note they selected all classes, but external validity is limited to similar Java projects.
  • domain assumption The seen/unseen cutoff date approximates LLM training data boundaries
    Section 4.2: October 1, 2023 is assumed to be the training cutoff for GPT-4o-mini. The paper acknowledges in Section 5.4 that some LLMs may have seen different sets of projects, which weakens the leakage control.

how reviews work

0 comments
Cite this review

Pith. "Pith review of YATE: The Role of Test Repair in LLM-Based Unit Test Generation." pith.science (2026). https://pith.science/paper/T5TACAHF

@misc{pith2026250718316,
  author       = {Pith},
  title        = {Pith review of: YATE: The Role of Test Repair in LLM-Based Unit Test Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T5TACAHF}},
  note         = {Machine review of arXiv:2507.18316}
}
read the original abstract

Recent advances in automated test generation utilises language models to produce unit tests. While effective, language models tend to generate many incorrect tests with respect to both syntax and semantics. Although such incorrect tests can be easily detected and discarded, they constitute a "missed opportunity" -- if fixed, they are often valuable as they directly add testing value (they effectively target the underlying program logic to be tested) and indirectly form good seeds for generating additional tests. To this end, we propose a simple technique for repairing some of these incorrect tests through a combination of rule-based static analysis and re-prompting. We evaluate this simple approach, named YATE, on a set of 6 open-source projects and show that it can effectively produce tests that cover on average 32.06% more lines and kill 21.77% more mutants than a plain LLM-based method. We also compare YATE with four other LLM-based methods, namely HITS, SYMPROMPT, TESTSPARK and COVERUP and show that it produces tests that cover substantially more code. YATE achieves 22% higher line coverage, 20% higher branch coverage and kill 20% more mutants at a comparable cost (number of calls to LLMs).

Figures

Figures reproduced from arXiv: 2507.18316 by the authors.

Figure 1
Figure 1. YATE workflow, based on the four components [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Coverage improvement of YATE over LLM-Plain for [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Plain Vs YATE when using different LLMs. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Distribution of line and branch coverage achieved, [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Knowledge-Guided Synthetic Bug Feedback for LLM-Based Unit Test Generation

    cs.SE 2026-07 conditional novelty 6.5 of 10

    Mechanism-guided synthetic-bug feedback raises real-bug detection of LLM unit tests to 72.67% RBDR on 172 Defects4J tasks, outperforming mutation and coverage baselines by ~20 points.

  2. On the risk of coding before testing: An empirical study on LLM-based test generation workflow

    cs.SE 2026-07 conditional novelty 6.0 of 10

    Generating LLM tests after faulty LLM code cuts fault detection to about 14% versus about 25% when tests are generated from the task description alone, across models and prompting strategies.

Reference graph

Works this paper leans on

50 extracted references · 15 canonical work pages · cited by 2 Pith papers

  1. [1]

    Nadia Alshahwan, Jubin Chheda, Anastasia Finegenova, Mark Harman, Alexan- dru Marginean, Shubho Sengupta, and Eddy Wang. 2024. Automated unit test improvement using Large Language Models at Meta. In ACM International Con- ference on the Foundations of Software Engineering (FSE 2024) (Porto de Galinhas, Brazil, Brazil)

    2024

  2. [2]

    Nadia Alshahwan, Jubin Chheda, Anastasia Finogenova, Beliz Gokkaya, Mark Harman, Inna Harper, Alexandru Marginean, Shubho Sengupta, and Eddy Wang

  3. [3]

    Nadia Alshahwan, Mark Harman, and Alexandru Marginean. 2023. Software Testing Research Challenges: An Industrial Perspective (keynote paper). In 2023 IEEE Conference on Software Testing, Verification and Validation (ICST 2023) . IEEE, 1–10

    2023

  4. [4]

    Nadia Alshahwan, Mark Harman, Alexandru Marginean, Shubho Sengupta, and Eddy Wang. 2024. Assured LLM-Based Software Engineering (keynote paper). In 2𝑛𝑑 . ICSE workshop on Interoperability and Robustness of Neural Software Engineering (InteNSE) (Lisbon, Portugal)

    work page 2024

  5. [5]

    Nadia Alshahwan, Mark Harman, Alexandru Marginean, and Eddy Wang. 2024. Observation-based unit test generation at Meta. In Foundations of Software Engi- neering (FSE 2024)

    2024

  6. [6]

    Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo

    Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo

  7. [7]

    Mohamed Salah Bouafif, Mohammad Hamdaqa, and Edward Zulkoski. 2025. PRIMG : Efficient LLM-driven Test Generation Using Mutant Prioritiza- tion. CoRR abs/2505.05584 (2025). https://doi.org/10.48550/ARXIV.2505.05584 arXiv:2505.05584

    arXiv doi:10.48550/arxiv.2505.05584 2025

  8. [8]

    Yinghao Chen, Zehao Hu, Chen Zhi, Junxiao Han, Shuiguang Deng, and Jian- wei Yin. 2024. ChatUniTest: A Framework for LLM-Based Test Generation. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering (Porto de Galinhas, Brazil) (FSE 2024). As- sociation for Computing Machinery, New York, NY, USA, 572–57...

    arXiv 2024

Show all 50 references
  1. [9]

    Henry Coles, Thomas Laurent, Christopher Henard, Mike Papadakis, and An- thony Ventresque. 2016. PIT: a practical mutation testing tool for Java (demo). In ISSTA. ACM, 449–452

    2016

  2. [10]

    Desmarais

    Arghavan Moradi Dakhel, Amin Nikanjam, Vahid Majdinasab, Foutse Khomh, and Michel C. Desmarais. 2024. Effective test generation using pre-trained Large Language Models and mutation testing. Inf. Softw. Technol. 171 (2024), 107468. https://doi.org/10.1016/J.INFSOF.2024.107468

    2024

  3. [11]

    Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei L...

    2024 arXiv

  4. [12]

    Angela Fan, Beliz Gokkaya, Mitya Lyubarskiy, Mark Harman, Shubho Sengupta, Shin Yoo, and Jie Zhang. 2023. Large Language Models for Software Engineering: Survey and Open Problems. In ICSE Future of Software Engineering (FoSE 2023)

    2023

  5. [14]

    Christopher Foster, Abhishek Gulati, Mark Harman, Inna Harper, Ke Mao, Jillian Ritchey, Hervé Robert, and Shubho Sengupta. 2025. Mutation-Guided LLM-based Test Generation at Meta. CoRR abs/2501.12862 (2025)

    2025 arXiv

  6. [15]

    Gordon Fraser and Andrea Arcuri. 2011. Evolutionary Generation of Whole Test Suites. In 2011 11th International Conference on Quality Software . 31–40. https://doi.org/10.1109/QSIC.2011.19

    2011 doi

  7. [16]

    Gordon Fraser and Andreas Zeller. 2011. Exploiting Common Object Usage in Test Case Generation. InFourth IEEE International Conference on Software Testing, Verification and Validation, ICST 2011, Berlin, Germany, March 21-25, 2011 . IEEE Computer Society, 80–89. https://doi.or...

    2011 doi

  8. [17]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997 2, 1 (2023)

    2023 arXiv

  9. [18]

    Siqi Gu, Chunrong Fang, Quanjun Zhang, Fangyuan Tian, Jianyi Zhou, and Zhenyu Chen. 2024. TestART: Improving LLM-based Unit Test via Co-evolution of Automated Generation and Repair Iteration. CoRR abs/2408.03095 (2024). https://doi.org/10.48550/ARXIV.2408.03095 arXiv:2408.03095

    2024 doi arXiv

  10. [19]

    Sijia Gu, Noor Nashid, and Ali Mesbah. 2025. LLM Test Generation via Iterative Hybrid Program Analysis. CoRR abs/2503.13580 (2025)

    2025 arXiv

  11. [20]

    Mark Harman, Peter O’Hearn, and Shubho Sengupta. 2025. Harden and Catch for Just-in-Time Assured LLM-Based Software Testing: Open Research Challenges (keynote paper). In 2025 ACM Conference on Foundations of Software Engineering (FSE 2025). ACM. Also available as arXiv preprin...

    2025 arXiv

  12. [21]

    Runlin Liu, Zhe Zhang, Yunge Hu, Yuhang Lin, Xiang Gao, and Hailong Sun. 2025. LLM-based Unit Test Generation for Dynamically-Typed Pro- grams. CoRR abs/2503.14000 (2025). https://doi.org/10.48550/ARXIV.2503.14000 arXiv:2503.14000

    2025 doi

  13. [22]

    Henry B Mann and Donald R Whitney. 1947. On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics (1947), 50–60

    1947

  14. [23]

    Mistral AI. 2024. Meet Codestral: A powerful and efficient open-weight model for code. https://mistral.ai/news/codestral. Introduces Codestral, a 22B-parameter open-weight code generation model supporting over 80 programming languages

    2024

  15. [24]

    Zifan Nan, Zhaoqiang Guo, Kui Liu, and Xin Xia. 2025. Test Intention Guided LLM-Based Unit Test Generation. In ICSE. IEEE, 1026–1038

    2025

  16. [25]

    Chao Ni, Xiaoya Wang, Liushan Chen, Dehai Zhao, Zhengong Cai, Shaohua Wang, and Xiaohu Yang. 2024. CasModaTest: A Cascaded and Model-agnostic Self-directed Framework for Unit Test Generation. CoRR abs/2406.15743 (2024)

    2024 arXiv

  17. [26]

    OpenAI. 2024. GPT -4o-mini: advancing cost -efficient intelligence. OpenAI blog post, July 18 2024. https://openai.com/index/gpt-4o-mini-advancing-cost- efficient-intelligence Introduces GPT -4o-mini, a small multimodal model with superior reasoning, coding, and math performan...

    2024

  18. [27]

    OpenAI. 2025. Introducing GPT-4.1 in the API. OpenAI Blog. https://openai. com/index/gpt-4-1/ Announces GPT-4.1 model series with improvements in coding, instruction following, and long-context capabilities

    2025

  19. [28]

    Rangeet Pan, Myeongsoo Kim, Rahul Krishna, Raju Pavuluri, and Saurabh Sinha

  20. [29]

    Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, and Mark Harman. 2019. Chapter Six - Mutation Testing Advances: An Analysis and Survey. Adv. Comput. 112 (2019), 275–378. https://doi.org/10.1016/BS.ADCOM. 2018.03.015

    2019 doi

  21. [30]

    Renaud Pawlak, Martin Monperrus, Nicolas Petitprez, Carlos Noguera, and Lionel Seinturier. 2016. SPOON: A library for implementing analyses and transforma- tions of Java source code. Softw. Pract. Exp. 46, 9 (2016), 1155–1179

    2016

  22. [31]

    arXiv preprint arXiv:2409.03093 (2024)

    Aster: Natural and multi-language unit test generation with llms. arXiv preprint arXiv:2409.03093 (2024)

    2024 arXiv

  23. [32]

    Gabriel Ryan, Siddhartha Jain, Mingyue Shang, Shiqi Wang, Xiaofei Ma, Mu- rali Krishna Ramanathan, and Baishakhi Ray. 2024. Code-Aware Prompt- ing: A Study of Coverage-Guided Test Generation in Regression Setting us- ing LLM. Proc. ACM Softw. Eng. 1, FSE, Article 43 (July 2024...

    2024 doi

  24. [33]

    Arkadii Sapozhnikov, Mitchell Olsthoorn, Annibale Panichella, Vladimir Ko- valenko, and Pouria Derakhshanfar. 2024. TestSpark: IntelliJ IDEA’s Ultimate Test Generation Companion. In Proceedings of the 2024 IEEE/ACM 46th Interna- tional Conference on Software Engineering: Compa...

    2024

  25. [34]

    Juan Altmayer Pizzorno and Emery D. Berger. 2024. CoverUp: Coverage-Guided LLM-Based Test Generation. CoRR abs/2403.16218 (2024). https://doi.org/10. 48550/ARXIV.2403.16218 arXiv:2403.16218 Konstantinou et al

    2024 doi arXiv

  26. [35]

    André Silva, Nuno Saavedra, and Martin Monperrus. 2024. GitBug-Java: A Reproducible Benchmark of Recent Java Bugs. In MSR. ACM, 118–122

    2024

  27. [36]

    Philipp Straubinger, Marvin Kreis, Stephan Lukasczyk, and Gordon Fraser. 2025. Mutation Testing via Iterative Large Language Model-Driven Scientific Debug- ging. In IEEE International Conference on Software Testing, Verification and Valida- tion, ICST 2025 - Workshops, Naples,...

    2025

  28. [37]

    Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2024. An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation. IEEE Trans. Software Eng. 50, 1 (2024), 85–105

    2024

  29. [38]

    Briand, and Kui Liu

    Guancheng Wang, Qinghua Xu, Lionel C. Briand, and Kui Liu. 2025. On Mutation- Guided Unit Test Generation. CoRR abs/2506.02954 (2025)

    2025 arXiv

  30. [39]

    Le, Ed H

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In The Eleventh International Conference on Learning Representations, ICLR 2023, K...

    2023

  31. [40]

    András Vargha and Harold D. Delaney. 2000. A Critique and Improvement of the "CL" Common Language Effect Size Statistics of McGraw and Wong. Journal of Educational and Behavioral Statistics 25, 2 (2000), 101–132. http: //www.jstor.org/stable/1165329

    2000

  32. [41]

    Chi, Quoc Le, and Denny Zhou

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. 2022. Chain of Thought Prompting Elicits Reasoning in Large Language Models. CoRR abs/2201.11903 (2022). arXiv:2201.11903 https://arxiv.org/abs/2201.11903

    2022 arXiv

  33. [42]

    Wei Wei. 2025. Static Analysis and LLM for Comprehensive Java Unit Test Generation. In 2025 8th International Conference on Advanced Electronic Materials, Computers and Software Engineering (AEMCSE) . 87–92. https://doi.org/10.1109/ AEMCSE65292.2025.11042526

    2025

  34. [43]

    Zejun Wang, Kaibo Liu, Ge Li, and Zhi Jin. 2024. HITS: High-coverage LLM-based Unit Test Generation via Method Slicing. In Proceedings of the 39th IEEE/ACM In- ternational Conference on Automated Software Engineering, ASE 2024, Sacramento, CA, USA, October 27 - November 1, 202...

    2024

  35. [44]

    Chen Yang, Junjie Chen, Bin Lin, Jianyi Zhou, and Ziqi Wang. 2024. Enhancing LLM-based Test Generation for Hard-to-Cover Branches via Program Analysis. CoRR abs/2404.04966 (2024)

    2024 arXiv

  36. [45]

    Xin Yin, Chao Ni, Xinrui Li, Liushan Chen, Guojun Ma, and Xiaohu Yang

  37. [46]

    Briand, and Kui Liu

    Qinghua Xu, Guancheng Wang, Lionel C. Briand, and Kui Liu. 2025. Hallucination to Consensus: Multi-Agent LLMs for End-to-End Test Generation with Accurate Oracles. CoRR abs/2506.02943 (2025)

    2025

  38. [47]

    Hong Zhu, Patrick A. V. Hall, and John H. R. May. 1997. Software Unit Test Coverage and Adequacy. ACM Comput. Surv. 29, 4 (1997), 366–427. https: //doi.org/10.1145/267580.267590

    1997

  39. [50]

    Zhiqiang Yuan, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, Xin Peng, and Yiling Lou. 2024. Evaluating and Improving ChatGPT for Unit Test Generation. Proc. ACM Softw. Eng. 1, FSE, Article 76 (July 2024), 24 pages. https://doi.org/10.1145/3660783

    2024 doi

  40. [2015]

    IEEE Transactions on Software Engineering 41, 5 (May 2015), 507–525

    The Oracle Problem in Software Testing: A Survey. IEEE Transactions on Software Engineering 41, 5 (May 2015), 507–525

    2015

  41. [2024]

    In SIGSOFT FSE Companion

    Automated Unit Test Improvement using Large Language Models at Meta. In SIGSOFT FSE Companion. ACM, 185–196

  42. [2025]

    CoRR abs/2501.07425 (2025)

    Enhancing LLM’s Ability to Generate More Repository-Aware Unit Tests Through Precise Contextual Information Injection. CoRR abs/2501.07425 (2025)

    2025 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.