Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

With current LLMs, a plain prompt-and-repair loop generates better test suites than four engineered state-of-the-art test generators, at comparable query cost.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 10:30 UTC pith:FXHW6TOG

load-bearing objection A timely empirical study with a plausible finding, but the ChatUniTest-based reimplementations of the four SOTA tools are never validated against the originals—so the headline claim is conditional. the 3 major comments →

arxiv 2601.09695 v2 pith:FXHW6TOG submitted 2026-01-14 cs.SE

How well LLM-based test generation techniques perform with newer LLM versions?

Michael Konstantinou , Renzo Degiovanni , Mike Papadakis This is my paper
classification cs.SE
keywords LLM-based test generationunit test generationcode coveragemutation testingprompt engineeringlarge language modelsempirical evaluationtest repair
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether the engineering inside recent LLM-based unit-test generators still pays off when the underlying LLM is a strong current model. The authors replicate four published techniques—method slicing, symbolic path enumeration, coverage feedback, and class-context prompting—and measure them against a deliberately simple baseline: one prompt containing the class (or method) source, followed by up to five repair rounds that feed compiler and runtime errors back to the model. Across 393 Java classes and three modern LLMs, the plain baseline scored higher than every engineered tool on line coverage, branch coverage, and mutation score, at a comparable number of LLM queries. The paper also shows that class-level and method-level prompts produce complementary tests, and that a hybrid that starts with classes and then targets uncovered methods keeps the effectiveness while using about 20% fewer LLM requests. This matters because, if true, the sophisticated machinery that dominates current test-generation research adds little value on modern models, and the real bottleneck is the large share of generated tests that fail to compile or contain wrong oracles.

Core claim

The paper's central claim is that with recent LLMs, a plain prompt-and-repair loop is at least as effective as—and here slightly better than—four previously published, elaborately engineered test generators. On the full 393-class, 3,657-method dataset, the plain method reached 49.95% line coverage, 35.33% branch coverage, and 33.82% mutation score; the best engineered baseline reached 42.43%, 29.49%, and 27.97%. The result held across three model families, and the plain method used fewer LLM queries than two of the four baselines and a comparable number to the others. The same experiments show that prompting at class granularity and method granularity produce mutually complementary tests, an

What carries the argument

The central object is the LLM-Plain workflow: a single zero-shot prompt containing the full source of a class or method, asking for a complete test suite, followed by up to five repair iterations that feed compiler and runtime errors back to the model, plus minimal parsing to fix package, import, and class-name mismatches. The second mechanism is granularity: the paper varies whether the prompt targets the whole class or an individual method, and then combines the two in a hybrid that first does class-level generation and then method-level generation only for methods with uncovered branches. Counting LLM queries serves as the cost model for each workflow.

Load-bearing premise

The comparison assumes that a shared plugin faithfully reproduces the four baseline tools' original prompts, repair loops, and termination behavior, and that assumption is not independently verified in the paper.

What would settle it

Run the four baselines from their original public code repositories on a subset of the 393 classes using the same three LLMs and temperature, and check whether any of them beats the plain prompt-and-repair loop on line coverage, branch coverage, or mutation score; if one does, the paper's conclusion that plain prompting is superior is refuted.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If correct, the elaborate aids in recent test generators—method slicing, symbolic path enumeration, coverage feedback, and parent-class context—add little or nothing over a basic prompt-plus-repair loop once the model is strong enough.
  • The bottleneck shifts from generation guidance to post-generation quality: since only about half of generated tests compile and pass, future tools should concentrate on repairing invalid tests and improving oracle quality rather than on guiding the LLM.
  • A class-then-uncovered-method hybrid offers almost the same effectiveness as running both granularities separately, while using about 20% fewer LLM requests, making it an attractive efficiency lever for paid models.
  • Evaluation practice should adopt full-project datasets and current model versions; older baselines and cherry-picked classes inflate the apparent contribution of engineering components.
  • The pattern held across three model families, so the finding is not an artifact of a single vendor's model.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the trend continues, the optimal unit-test generator may become a thin wrapper around a strong model plus a repair and oracle-fixing loop; traditional code-analysis guidance may survive only as a fallback for hard-to-reach branches.
  • The larger gap observed for the model with the later training cutoff hints that memorization of these repositories could inflate plain-LLM performance; testing on genuinely unseen code would be needed to separate capability from data contamination.
  • A natural next experiment is to add a lightweight test-repair or oracle-fixing stage to the hybrid granularity loop; given that roughly 20% of tests fail to compile and around 43% fail at runtime, repairing even half of those could raise coverage well beyond the engineered tools.
  • The complementarity of class-level and method-level suites suggests that multiple granularities encode different knowledge, so a prompt explicitly mixing cross-method integration tests at class level with deep per-method tests might beat both.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper asks whether four previously proposed LLM-based test generation tools with substantial engineering—HITS (method slicing), SymPrompt (symbolic-execution-guided prompting), TestSpark (context-enriched prompting), and CoverUp (coverage-guided augmentation)—still outperform a simple "Plain-LLM" baseline when equipped with newer LLMs (gpt-4o-mini, DeepSeek V3, Llama 3.3 70B). The authors evaluate all approaches on 393 classes / 3,657 methods from six Java projects, measuring line coverage, branch coverage, and PIT mutation score, as well as number of LLM requests. They report that Plain-LLM at method level achieves the highest aggregate coverage (49.95% line, 35.33% branch, 33.82% mutation vs. best baseline HITS 42.43%/29.49%/27.97%), with comparable or lower request counts. They further show class-level and method-level suites are complementary, and propose a hybrid strategy (class-level first, then method-level for uncovered methods) that yields comparable effectiveness with ~20% fewer requests than the combined approach. The paper also documents low compilation/passing rates and LLM-generated helper classes as open challenges.

Significance. If the central comparison is accepted, the paper's finding that a plain prompt-plus-repair loop matches or beats four published tools on newer LLMs is important for test-generation research; it suggests that complex guided generation may become less valuable as LLMs improve. The study's strengths include whole-project evaluation on 393 classes, three model families, standard coverage/mutation metrics, low temperature, and uniform invalid-test filtering. The hybrid granularity strategy is a practical, falsifiable contribution. However, the unvalidated ChatUniTest implementations and incomplete statistical reporting currently prevent full confidence in the headline claim. The paper also contributes useful negative evidence on LLM-generated test quality (compilation/passing rates, hallucinated helper classes).

major comments (3)
  1. [§IV-C, §VI-C.1, §VIII-A] The central comparison assumes that the ChatUniTest plugin reproduces the original HITS, SymPrompt, TestSpark, and CoverUp algorithms faithfully. §IV-C states only that the authors 'relied on ChatUniTest... that implements all the algorithms', and §VI-C.1 refers to 'the implementation of ChatUniTest, which we use to execute the four state-of-the-art tools'. No evidence is provided that the prompts, repair loops, coverage augmentation, or symbolic path enumeration match the published tools. If these reimplementations are degraded, the headline superiority of Plain-LLM (abstract: +17.72% line, +19.80% branch, +20.92% mutation) is an artifact of comparing against weaker baselines. The threat is acknowledged in §VIII-A, but calibration against the original tools on even a subset of classes is necessary.
  2. [§IV-B, §V-A] The main conclusion that existing techniques offer 'no advantage over Plain-LLM' (Finding 1) is not supported by the statistical analysis as reported. The paper says Mann-Whitney U tests were used, but no p-values or effect sizes are given for the principal comparisons; only the hybrid-vs-combined comparison reports p-values (p=0.92, p=0.75 in §VI-B). With six projects (n=6), a non-significant result is weak evidence of equivalence. Please specify the unit of analysis (per-class? per-project?), report distributions/effect sizes, and interpret equivalence bounds rather than mere absence of significance.
  3. [§VI-B, §VII-C] The hybrid approach's cost savings depend on the rule for selecting which methods receive method-level generation ('methods that are not fully covered at the branch level', §VI-B). The threshold is not precisely defined, and no sensitivity analysis is provided. If 'fully covered' means 100% branch coverage, most methods will be selected; if it means a different cutoff, the 20% request saving (Table VIII) may change drastically. Since the hybrid is one of the paper's practical contributions, this free parameter needs explicit definition and robustness analysis.
minor comments (5)
  1. [§VI-B / §VI-C.1] Finding 6 appears twice: once for the hybrid approach and once for the invalid-test analysis. Renumber to avoid ambiguity.
  2. [§V-C] The text says 'Across the dataset, class-level testing generated a total of 1,510 tests, which is approximately 7.76 times fewer than the number generated at the method level.' 1,510 is the number of LLM requests, not generated tests; Table V shows 3,232 class-level tests for GPT-4o-mini. Please correct.
  3. [Fig. 2, Table II] Typo 'SymPromt' in Figure 2 text; also use consistent capitalization for DeepSeek V3 and CoverUp/CUP.
  4. [Author block] Typo in affiliation: 'Luxuembourg' should be 'Luxembourg'.
  5. [§II, §IV-D] The term 'Plain-LLM' is defined in contrast to 'Raw LLM', but the implementation includes up to five repair iterations with execution feedback (Fig. 1). Consider clarifying that the baseline is not prompt-only but a minimal repair loop, so readers do not conflate it with zero-shot generation without feedback.

Circularity Check

0 steps flagged

No significant circularity: the study is an empirical benchmark with standard external metrics; no fitted parameter or self-citation chain defines the outcome.

full rationale

This paper is an empirical comparison, not a derivation, so the circularity patterns do not apply. The central claim (plain LLM prompting outperforms four engineered tools) is established by running external tools and measuring with standard metrics (JaCoCo, PIT, Mann–Whitney U). No parameter is fitted to a subset of data and then renamed as a prediction. The hybrid class-then-uncovered-method strategy is presented as an observed post-hoc result (Sections V-C and VI-B), not as a fitted model, so it is a selection-bias/overfitting concern rather than circularity. The only self-citation (Ref. [36] on test oracles) is not load-bearing for the main result. The PIT reference (Ref. [14]) shares an author, but PIT is an external, established mutation-testing tool used as a metric, not as evidence for the paper's claims. The one substantive validity threat—that the four baselines are executed via ChatUniTest rather than the original tool repositories (Sections IV-C and VIII-A, quoting 'we relied on ChatUniTest, a widely adopted plugin that implements all the algorithms included in our evaluation')—concerns baseline fidelity and external validity, not circularity. It could weaken the comparison but does not make the outcome equivalent to the inputs by definition. Therefore no circular step is identified.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

The paper's claims rest on benchmark validity, faithful tool replication, and contamination proxies rather than free parameters or invented entities. The only hand-chosen design element is the hybrid selection rule, which is tuned on the same dataset it evaluates. The main non-standard assumptions are the fidelity of the ChatUniTest-based tool reimplementations and the use of cutoff dates to estimate contamination.

free parameters (1)
  • branch-coverage threshold for hybrid method selection = 0 uncovered branches (fully covered method)
    The hybrid generates method-level tests only for methods with uncovered branches after class-level generation. This decision rule was chosen after observing class/method complementarity on the same six projects, so it is effectively tuned to the evaluation set rather than independently validated.
axioms (4)
  • ad hoc to paper The ChatUniTest plugin reproduces the original HITS, SymPrompt, TestSpark, and CoverUp algorithms closely enough for a fair comparison.
    Sections IV-C and VIII-A rely on ChatUniTest to execute all four tools, but no evidence is provided that the plugin matches the original implementations, prompts, or repair loops.
  • ad hoc to paper Repository creation date relative to a model's announced cutoff is a valid proxy for whether training-data contamination occurred.
    Section IV-A labels projects as seen/unseen by cutoff date; Section VIII-C acknowledges the uncertainty. DeepSeek V3 may have seen all projects, which the paper itself notes.
  • ad hoc to paper Aggregating coverage across six projects and testing with Mann-Whitney U is sufficiently powerful to support the 'no advantage over Plain-LLM' conclusion.
    Section V-A reports statistical confirmation but no p-values for the main comparisons; with N=6 the power to detect small differences is low.
  • domain assumption Line coverage, branch coverage, and PIT mutation score are accepted proxies for unit-test suite effectiveness.
    These are standard software-engineering adequacy metrics cited from [13] and [14]; all effectiveness conclusions depend on them.

pith-pipeline@v1.3.0-alltime-deepseek · 16752 in / 10987 out tokens · 114080 ms · 2026-08-03T10:30:07.833335+00:00 · methodology

0 comments
read the original abstract

The rapid evolution of Large Language Models (LLMs) has strongly impacted software engineering, leading to a growing number of studies on automated unit test generation. However, the standalone use of LLMs without post-processing has proven insufficient, often producing tests that fail to compile or achieve high coverage. Several techniques have been proposed to address these issues, reporting improvements in test compilation and coverage. While important, LLM-based test generation techniques have been evaluated against relatively weak baselines (for todays' standards), i.e., old LLM versions and relatively weak prompts, which may exacerbate the performance contribution of the approaches. In other words, stronger (newer) LLMs may obviate any advantage these techniques bring. We investigate this issue by replicating four state-of-the-art LLM-based test generation tools, HITS, SymPrompt, TestSpark, and CoverUp that include engineering components aimed at guiding the test generation process through compilation and execution feedback, and evaluate their relative effectiveness and efficiency over a plain LLM test generation method. We integrate current LLM versions in all approaches and run an experiment on 393 classes and 3,657 methods. Our results show that the plain LLM approach can outperform previous state-of-the-art approaches in all test effectiveness metrics we used: line coverage (by 17.72%), branch coverage (by 19.80%) and mutation score (by 20.92%), and it does so at a comparable cost (LLM queries). We also observe that the granularity at which the plain LLM is applied has a significant impact on the cost. We therefore propose targeting first the program classes, where test generation is more efficient, and then the uncovered methods to reduce the number of LLM requests. This strategy achieves comparable (slightly higher) effectiveness while requiring about 20% fewer LLM requests.

Figures

Figures reproduced from arXiv: 2601.09695 by Michael Konstantinou, Mike Papadakis, Renzo Degiovanni.

Figure 1
Figure 1. Figure 1: LLM-Plain unit test generation workflow is on evaluating LLM-based tools that have been previously assessed using earlier model versions. CoverUp [10] extends prior test generation workflows by introducing an additional coverage augmentation phase. Af￾ter generating an initial test suite, CoverUp measures code coverage, identifies uncovered lines, and subsequently guides the LLM to generate additional test… view at source ↗
Figure 2
Figure 2. Figure 2: Number of API requests between different approaches [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Number of API requests for LLM-Plain across multiple LLMs [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Additional content generated by LLMs instead of fixing unit tests [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Average additional content generated on whole dataset [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. SemaDiff: Identifying Semantic-Changing Commits with Generated Code and Tests

    cs.SE 2026-07 conditional novelty 6.0

    SemaDiff identifies semantic-preserving refactoring commits by generating and executing LLM-produced tests on both pre- and post-commit versions, achieving 100% precision in detecting behavior changes.

  2. Prompt Coverage Adequacy

    cs.SE 2026-07 unverdicted novelty 6.0

    Prompt Coverage Adequacy, measured via attention boosting in LLMs, is associated with fault detection and uncovers over 30% more faults than traditional code coverage when guiding test generation across two datasets.

Reference graph

Works this paper leans on

39 extracted references · 1 canonical work pages · cited by 2 Pith papers · 1 internal anchor

  1. [1]

    Large language models for software engineering: A systematic literature review,

    X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engineering: A systematic literature review,”ACM Trans. Softw. Eng. Methodol., Sep. 2024, just Accepted. [Online]. Available: https://doi.org/10.1145/3695988

    doi:10.1145/3695988 2024

  2. [3]

    Automated unit test improvement using Large Language Models at Meta,

    N. Alshahwan, J. Chheda, A. Finegenova, M. Harman, A. Marginean, S. Sengupta, and E. Wang, “Automated unit test improvement using Large Language Models at Meta,” inACM International Conference on the Foundations of Software Engineering (FSE 2024), July 2024

    2024

  3. [4]

    Evaluating and improving chatgpt for unit test generation,

    Z. Yuan, M. Liu, S. Ding, K. Wang, Y . Chen, X. Peng, and Y . Lou, “Evaluating and improving chatgpt for unit test generation,”Proc. ACM Softw. Eng., vol. 1, no. FSE, Jul. 2024. [Online]. Available: https://doi.org/10.1145/3660783

    doi:10.1145/3660783 2024

  4. [5]

    HITS: high-coverage llm-based unit test generation via method slicing,

    Z. Wang, K. Liu, G. Li, and Z. Jin, “HITS: high-coverage llm-based unit test generation via method slicing,” inProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ASE 2024, Sacramento, CA, USA, October 27 - November 1, 2024, V . Filkov, B. Ray, and M. Zhou, Eds. ACM, 2024, pp. 1258–

    2024

  5. [6]

    Software testing with large language models: Survey, landscape, and vision,

    J. Wang, Y . Huang, C. Chen, Z. Liu, S. Wang, and Q. Wang, “Software testing with large language models: Survey, landscape, and vision,” IEEE Trans. Softw. Eng., vol. 50, no. 4, p. 911–936, Feb. 2024. [Online]. Available: https://doi.org/10.1109/TSE.2024.3368208

    arXiv 2024

  6. [7]

    Chain of thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. H. Chi, Q. Le, and D. Zhou, “Chain of thought prompting elicits reasoning in large language models,”CoRR, vol. abs/2201.11903, 2022. [Online]. Available: https://arxiv.org/abs/2201.11903

    Pith/arXiv arXiv 2022

  7. [8]

    Self-consistency improves chain of thought reasoning in language models,

    X. Wang, J. Wei, D. Schuurmans, Q. V . Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models,” inThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. [Online]. Available: https://openreview.net/forum?id=1PL1NIMMrw

    2023

  8. [9]

    Retrieval-augmented generation for large language models: A survey,

    Y . Gao, Y . Xiong, X. Gao, K. Jia, J. Pan, Y . Bi, Y . Dai, J. Sun, H. Wang, and H. Wang, “Retrieval-augmented generation for large language models: A survey,”arXiv preprint arXiv:2312.10997, vol. 2, no. 1, 2023

    Pith/arXiv arXiv 2023

  9. [10]

    Coverup: Coverage-guided llm-based test generation,

    J. A. Pizzorno and E. D. Berger, “Coverup: Coverage-guided llm-based test generation,”CoRR, vol. abs/2403.16218, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2403.16218

    Pith/arXiv arXiv doi:10.48550/arxiv.2403.16218 2024

  10. [11]

    Code-aware prompting: A study of coverage-guided test generation in regression setting using llm,

    G. Ryan, S. Jain, M. Shang, S. Wang, X. Ma, M. K. Ramanathan, and B. Ray, “Code-aware prompting: A study of coverage-guided test generation in regression setting using llm,”Proc. ACM Softw. Eng., vol. 1, no. FSE, Jul. 2024. [Online]. Available: https://doi.org/10.1145/3643769

    doi:10.1145/3643769 2024

  11. [12]

    Testspark: Intellij idea’s ultimate test generation companion,

    A. Sapozhnikov, M. Olsthoorn, A. Panichella, V . Kovalenko, and P. Derakhshanfar, “Testspark: Intellij idea’s ultimate test generation companion,” inProceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings, 2024, pp. 30–34

    2024

  12. [13]

    Software unit test coverage and adequacy,

    H. Zhu, P. A. V . Hall, and J. H. R. May, “Software unit test coverage and adequacy,”ACM Comput. Surv., vol. 29, no. 4, pp. 366–427, 1997. [Online]. Available: https://doi.org/10.1145/267580.267590

    arXiv 1997

  13. [14]

    PIT: a practical mutation testing tool for java (demo),

    H. Coles, T. Laurent, C. Henard, M. Papadakis, and A. Ventresque, “PIT: a practical mutation testing tool for java (demo),” inISSTA. ACM, 2016, pp. 449–452

    2016

  14. [15]

    Aster: Natural and multi-language unit test generation with llms,

    R. Pan, M. Kim, R. Krishna, R. Pavuluri, and S. Sinha, “Aster: Natural and multi-language unit test generation with llms,”arXiv preprint arXiv:2409.03093, 2024

    Pith/arXiv arXiv 2024

  15. [16]

    Llm-based unit test generation for dynamically-typed programs,

    R. Liu, Z. Zhang, Y . Hu, Y . Lin, X. Gao, and H. Sun, “Llm-based unit test generation for dynamically-typed programs,”CoRR, vol. abs/2503.14000, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2503.14000

    doi:10.48550/arxiv.2503.14000 2025

  16. [17]

    Static analysis and llm for comprehensive java unit test gen- eration,

    W. Wei, “Static analysis and llm for comprehensive java unit test gen- eration,” in2025 8th International Conference on Advanced Electronic Materials, Computers and Software Engineering (AEMCSE), 2025, pp. 87–92

    2025

  17. [18]

    Hallucination to consensus: Multi-agent llms for end-to-end test generation with accurate oracles,

    Q. Xu, G. Wang, L. C. Briand, and K. Liu, “Hallucination to consensus: Multi-agent llms for end-to-end test generation with accurate oracles,” CoRR, vol. abs/2506.02943, 2025

    arXiv 2025

  18. [19]

    An empirical evaluation of using large language models for automated unit test generation,

    M. Sch ¨afer, S. Nadi, A. Eghbali, and F. Tip, “An empirical evaluation of using large language models for automated unit test generation,”IEEE Trans. Software Eng., vol. 50, no. 1, pp. 85–105, 2024

    2024

  19. [20]

    Automated unit test improvement using large language models at meta,

    N. Alshahwan, J. Chheda, A. Finogenova, B. Gokkaya, M. Harman, I. Harper, A. Marginean, S. Sengupta, and E. Wang, “Automated unit test improvement using large language models at meta,” inSIGSOFT FSE Companion. ACM, 2024, pp. 185–196

    2024

  20. [21]

    Test intention guided llm-based unit test generation,

    Z. Nan, Z. Guo, K. Liu, and X. Xia, “Test intention guided llm-based unit test generation,” inICSE. IEEE, 2025, pp. 1026–1038

    2025

  21. [22]

    Casmodatest: A cascaded and model-agnostic self-directed framework for unit test generation,

    C. Ni, X. Wang, L. Chen, D. Zhao, Z. Cai, S. Wang, and X. Yang, “Casmodatest: A cascaded and model-agnostic self-directed framework for unit test generation,”CoRR, vol. abs/2406.15743, 2024

    Pith/arXiv arXiv 2024

  22. [23]

    Chatunitest: A framework for llm-based test generation,

    Y . Chen, Z. Hu, C. Zhi, J. Han, S. Deng, and J. Yin, “Chatunitest: A framework for llm-based test generation,” inCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, ser. FSE 2024. New York, NY , USA: Association for Computing Machinery, 2024, p. 572–576. [Online]. Available: https://doi.org/10.1145/3...

    arXiv 2024

  23. [24]

    Testart: Improving llm-based unit test via co-evolution of automated generation and repair iteration,

    S. Gu, C. Fang, Q. Zhang, F. Tian, J. Zhou, and Z. Chen, “Testart: Improving llm-based unit test via co-evolution of automated generation and repair iteration,”CoRR, vol. abs/2408.03095, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2408.03095

    Pith/arXiv arXiv doi:10.48550/arxiv.2408.03095 2024

  24. [25]

    LLM test generation via iterative hybrid program analysis,

    S. Gu, N. Nashid, and A. Mesbah, “LLM test generation via iterative hybrid program analysis,”CoRR, vol. abs/2503.13580, 2025

    Pith/arXiv arXiv 2025

  25. [26]

    Enhancing llm-based test generation for hard-to-cover branches via program analysis,

    C. Yang, J. Chen, B. Lin, J. Zhou, and Z. Wang, “Enhancing llm-based test generation for hard-to-cover branches via program analysis,”CoRR, vol. abs/2404.04966, 2024

    Pith/arXiv arXiv 2024

  26. [27]

    Mutation testing via iterative large language model-driven scientific debugging,

    P. Straubinger, M. Kreis, S. Lukasczyk, and G. Fraser, “Mutation testing via iterative large language model-driven scientific debugging,” inIEEE International Conference on Software Testing, Verification and Validation, ICST 2025 - Workshops, Naples, Italy, March 31 - April 4, 2025. IEEE, 2025, pp. 358–367. [Online]. Available: https://doi.org/10.1109/ICS...

    arXiv 2025

  27. [28]

    On mutation-guided unit test generation,

    G. Wang, Q. Xu, L. C. Briand, and K. Liu, “On mutation-guided unit test generation,”CoRR, vol. abs/2506.02954, 2025

    Pith/arXiv arXiv 2025

  28. [29]

    Effective test generation using pre-trained large language models and mutation testing,

    A. M. Dakhel, A. Nikanjam, V . Majdinasab, F. Khomh, and M. C. Desmarais, “Effective test generation using pre-trained large language models and mutation testing,”Inf. Softw. Technol., vol. 171, p. 107468,

  29. [30]

    Mutation-guided llm-based test generation at meta,

    C. Foster, A. Gulati, M. Harman, I. Harper, K. Mao, J. Ritchey, H. Robert, and S. Sengupta, “Mutation-guided llm-based test generation at meta,”CoRR, vol. abs/2501.12862, 2025

    Pith/arXiv arXiv 2025

  30. [31]

    PRIMG : Efficient LLM-driven Test Generation Using Mutant Prioritization

    M. S. Bouafif, M. Hamdaqa, and E. Zulkoski, “PRIMG : Efficient llm-driven test generation using mutant prioriti- zation,”CoRR, vol. abs/2505.05584, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2505.05584

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2505.05584 2025

  31. [32]

    Gitbug-java: A reproducible benchmark of recent java bugs,

    A. Silva, N. Saavedra, and M. Monperrus, “Gitbug-java: A reproducible benchmark of recent java bugs,” inMSR. ACM, 2024, pp. 118–122

    2024

  32. [33]

    On a test of whether one of two random variables is stochastically larger than the other,

    H. B. Mann and D. R. Whitney, “On a test of whether one of two random variables is stochastically larger than the other,”The annals of mathematical statistics, pp. 50–60, 1947

    1947

  33. [34]

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,

    L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, and T. Liu, “A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,” ACM Trans. Inf. Syst., vol. 43, no. 2, Jan. 2025

    2025

  34. [35]

    Test oracle automation in the era of llms,

    F. Molina, A. Gorla, and M. d’Amorim, “Test oracle automation in the era of llms,”ACM Transactions on Software Engineering and Methodology, vol. 34, no. 5, pp. 1–24, 2025

    2025

  35. [36]

    Do llms generate test oracles that capture the actual or the expected program behaviour?

    M. Konstantinou, R. Degiovanni, and M. Papadakis, “Do llms generate test oracles that capture the actual or the expected program behaviour?” arXiv preprint arXiv:2410.21136, 2024

    Pith/arXiv arXiv 2024

  36. [37]

    Toga: A neural method for test oracle generation,

    E. Dinella, G. Ryan, T. Mytkowicz, and S. K. Lahiri, “Toga: A neural method for test oracle generation,” inProceedings of the 44th International Conference on Software Engineering, 2022, pp. 2130– 2141

    2022

  37. [38]

    Togll: Correct and strong test oracle generation with llms,

    S. B. Hossain and M. Dwyer, “Togll: Correct and strong test oracle generation with llms,”arXiv preprint arXiv:2405.03786, 2024

    Pith/arXiv arXiv 2024

  38. [1268]

    Available: https://doi.org/10.1145/3691620.3695501

    [Online]. Available: https://doi.org/10.1145/3691620.3695501

    arXiv

  39. [2024]

    Available: https://doi.org/10.1016/j.infsof.2024.107468

    [Online]. Available: https://doi.org/10.1016/j.infsof.2024.107468

    arXiv 2024