Pith. sign in

REVIEW 5 major objections 4 minor 2 cited by

Injecting project-specific and testing-domain knowledge into an LLM-based pipeline yields unit tests that compile, run, and cover complex methods better than direct code-to-code generation.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 21:37 UTC pith:VQ4VHAE2

load-bearing objection The method is genuinely new and the ablation is well-designed, but the execution pass-rate claim is internally contradictory—§3.2.2 says 90.05%, Table 2 says 77.07%—so the headline numbers need to be corrected before the central claim can be assessed. the 5 major comments →

arxiv 2511.14224 v3 pith:VQ4VHAE2 submitted 2025-11-18 cs.SE

Knowledge Matters: Injecting Project and Testing Knowledge into LLM-based Unit Test Generation

Anji Li , Mingwei Liu , Zhenxi Chen , Zheng Pei , Zike Li , Dekun Dai , Yanlin Wang , Zibin Zheng This is my paper
classification cs.SE
keywords LLM-based unit test generationproject knowledge injectionstatic analysisusage trace extractiontest case designmulti-perspective promptingtest maintainabilitycode coverage
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the main reason LLM-written unit tests fail in real projects is missing knowledge: the model does not know how the project's classes are actually constructed and used, and it does not apply basic testing heuristics such as boundary analysis or exception coverage. To fix this, the proposed framework, KTester, statically extracts a reusable project knowledge base—class structure, signatures, Javadoc, and realistic invocation traces—and feeds it into a pipeline that separates test-case design from test-method generation. The paper reports large gains over the strongest baseline on a 110-method benchmark of complex Java methods: execution pass rate up 5.69 percentage points, line coverage up 8.83 percentage points, and fewer test cases generated. A human study also rates the resulting tests as more correct, readable, and maintainable.

Core claim

KTester's central claim is that two complementary injections of knowledge—project awareness and testing-domain awareness—make LLM-based unit test generation substantially more correct and more maintainable. The project knowledge comes from static analysis of the whole codebase, including how focal methods are invoked by real callers, so the LLM no longer has to guess at object construction or input preparation. The testing knowledge is applied by explicitly decoupling 'what to test' from 'how to test': test scenarios are first designed in a structured intermediate format using multiple testing perspectives, then transformed into executable Java test methods. On the benchmark, KTester reaches

What carries the argument

The load-bearing mechanism is a two-stage pipeline with a critical separation between test design and test implementation. Offline, static analysis builds a per-project knowledge base: structural metadata, Javadoc semantics, dependency relations, and path-sensitive usage traces extracted from caller control-flow graphs via slicing. Online, the LLM generates a test class framework, designs grouped test cases in a lightweight JSON format under three prompting perspectives (condition-branch, functionality, and exception-oriented), transforms each group into executable test methods with retrieved related-function context, integrates the methods into a coherent class, and repairs failures through

Load-bearing premise

The benchmark (110 complex Java methods from ten projects) and the choice of a single LLM backbone are treated as representative; if a broader set of projects, languages, or models shrinks the measured gap, the attribution of the gains to knowledge injection would weaken.

What would settle it

Run KTester and a direct one-step generation method on a fresh set of at least 100 complex methods from unseen Java projects, using the same LLM backend; if execution pass rate and line coverage do not differ materially, or if the separation step can be removed without a comparable drop, the paper's central causal claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the result holds, project-aware knowledge injection becomes a standard component of LLM test generation rather than an optional prompt enhancement.
  • The ablation indicates that separating test-case design from test-method generation is the highest-impact component: removing it drops execution pass rate by roughly 24 percentage points.
  • Because the knowledge base is built once per project and reused across methods, the cost of the offline analysis amortizes, making whole-repository test generation practical.
  • The paper's own model-swap experiments suggest the gains transfer beyond the single LLM used in the main comparison.
  • Generated tests are fewer and more structured, which points toward cheaper and more maintainable regression suites.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • As an extension, the same per-project knowledge base could feed other code-generation tasks—bug repair, documentation, refactoring—where realistic invocation patterns matter.
  • The JSON test plans produced at the design stage could be reused as executable specifications, enabling cross-language test generation or automatic regression-suite maintenance.
  • Replacing static usage traces with dynamic execution traces could close the remaining coverage gap on paths that callers rarely exercise.
  • A natural test of the framework's boundary is to apply it to simpler methods or non-Java languages, where the value of heavy project-knowledge injection may shrink.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes KTester, a framework for LLM-based unit test generation that augments the model prompt with project-specific knowledge (extracted offline via static analysis) and testing-domain knowledge, and separates test-case design from test-method generation. The framework is evaluated on the HITS benchmark (110 methods from 10 Java projects) against four baselines (ChatUnitTest, ChatTester, HITS, and UTGen), with correctness, coverage, and efficiency metrics, an ablation study, and a human study on readability and maintainability. The central claim is that KTester outperforms all baselines across six key metrics and that the modular separation of test design and test code generation is the largest contributor to these gains.

Significance. If the empirical claims are upheld, the work makes a useful contribution to LLM-based test generation: it demonstrates a reusable offline knowledge base, a multi-perspective prompting strategy, and a modular generation pipeline that are plausibly responsible for improved coverage and maintainability. The paper ships code and data, uses an external benchmark (HITS), and compares against independently published baselines, which are strengths. However, the headline quantitative claims are currently undermined by multiple internal numerical inconsistencies — most importantly the EPR contradiction between the prose and Table 2 — so the significance of the central result cannot be assessed from the manuscript as written.

major comments (5)
  1. [Section 3.2.2 vs. Table 2] The prose states: 'Regarding the Execution Pass Rate (EPR), KTester attains 90.05%, slightly lower than UTGen.' Table 2 lists KTester EPR=77.07 and UTGen EPR=90.05. The prose value 90.05 is exactly UTGen's EPR. In the same paragraph, prose reports KTester LC=62.78%, BC=54.71% and HITS LC/BC=49.74%/43.74%, while Table 2 lists KTester LC/BC=61.10%/52.59% and HITS=52.27%/45.93%. Consequently the abstract's claim of 'improving execution pass rate by 5.69% over the strongest baseline' and Finding 1's 'outperforms all baselines across eight metrics' cannot be verified. If Table 2 is correct, KTester is 12.98 pp below UTGen on EPR, so the headline claim is false; if the prose is correct, Table 2 must be regenerated. Please reconcile all numbers and update the claims accordingly.
  2. [Section 3.3.2 / Finding 2] Ablation numbers are internally inconsistent. The prose reports that removing the modular test-case-design/transformation step (KTester-DGT) causes EPR to drop by 24.08% and LC by 12.61%, while Finding 2 reports drops of 22.15% and 14.29%. Table 4 (KTester 77.07->52.99; 61.10->48.49) supports the Section 3.3.2 prose. Additionally, Finding 2 refers to the variant as 'KTester-TCG' although the actual variant name is KTester-DGT. Please correct these values and the variant name.
  3. [Section 3.2.1 (experimental design)] The evaluation is based on three runs per configuration, but no variance information, confidence intervals, or significance tests are reported. The abstract uses 'significantly outperforms,' yet differences on some metrics are only a few percentage points (e.g., LC 61.10 vs. 52.27; BC 52.59 vs. 45.93). Without per-run data or a statistical test, it is impossible to know whether these gaps are stable or within noise. Please report distributions, standard deviations, or appropriate significance tests, and soften 'significant' unless statistically supported.
  4. [Section 3.1.2 (baselines)] UTGen is not evaluated with the same LLM backend as the other methods: it uses EvoSuite with CodeLlama-7b, while all other methods use gpt-4o-mini. This confounds the comparison on execution pass rate (UTGen 90.05) and makes the label 'strongest baseline' ambiguous — the paper's '5.69% over the strongest baseline' appears to compare against HITS, not against UTGen, even though UTGen is included as a baseline. Please either exclude UTGen from the headline 'strongest baseline' comparison or provide a principled justification for comparing across different backends.
  5. [Section 3.4.1 / Section 5 (user study)] The user study is described as targeting 'professional developers,' but the actual participants are 5 Ph.D. students and 10 Master's students (15 total, recruited from university departments). Moreover, Section 5 states 'The study includes 14 professional developers.' The RQ3 conclusion that KTester 'consistently outperforms all baselines in human evaluation' should be scoped to the student participants actually recruited, and the 14-vs-15 discrepancy should be resolved.
minor comments (4)
  1. [Throughout] Typos and inconsistent naming: 'baslines' in Section 3.1.2; 'Lables' in Figure 1; 'InputCharactor' in Figure 6; 'simiar' in the description of Figure 3; 'Undertstanding' in Section 4.2 (if present in the original text).
  2. [Table 4] KTester's BC is 52.49 in Table 4 but 52.59 in Table 2. Please ensure all tables refer to the same set of runs.
  3. [Section 3.4.2] The readability distribution is described as 'over 90%' for KTester; 0.57 + 0.35 = 0.92, which is accurate, but please standardize capitalization of Likert scale labels (e.g., 'Strongly Agree' vs. 'strongly agree').
  4. [Section 4.2] Some referenced URLs and papers (e.g., [31], [32]) are point-in-time model names; please specify the exact access dates or versions, as model versions change quickly.

Circularity Check

0 steps flagged

No meaningful circularity: KTester's gains are empirical measurements against external baselines; the sole self-citation is non-load-bearing.

full rationale

KTester is an empirical evaluation paper, not a derivation. The pipeline's components (static project knowledge extraction, multi-view test-case design, transformation, integration, repair) are described operationally and evaluated on the external HITS benchmark against independently published baselines (ChatUnitTest, ChatTester, HITS, UTGen). No metric in Table 2 is computed from a fitted parameter or defined in terms of the target result; EPR, LC, BC, LCP, and BCP are measured by executing generated tests with JaCoCo, and the ablation variants are concrete pipeline modifications rather than renamings of the input. The only self-citation is ChatTester [22], which appears as a baseline in Section 3.1.2 and in related work; the authors re-run that baseline from its official implementation, so it is not used as evidence for KTester's central claims. There is no uniqueness theorem, no ansatz smuggled in via citation, and no target quantity that reduces by construction to an input. The manuscript does contain internal reporting inconsistencies (e.g., Section 3.2.2 prose assigns UTGen's EPR value 90.05 to KTester, contradicting Table 2; Finding 2's variant label and drop values differ from the Section 3.3.2 text; Section 3.4.1 reports 15 user-study participants while Section 5 reports 14). These are correctness/consistency threats, not circularity, and they do not satisfy the reduction-by-construction criterion required for a circularity finding.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 0 invented entities

No new entities. The main 'free parameters' are the prompt designs and threshold choices (repair iterations limited to 5, top-N similar functions, Jaccard similarity combination), which are not tuned per dataset in the text but remain unstated choices.

free parameters (2)
  • top-N similar functions
    The similarity context construction selects 'top-N most similar functions' (Eq. 1-3). The exact value of N is never specified; it is chosen by hand to balance prompt size and accuracy, and could affect the method's performance.
  • repair iterations limit = 5
    The paper states 'We limited automatic repair iterations to 5 in KTester' (§3.2.1). This is a hand-chosen budget that directly caps the correctness improvements; a different budget could change EPR and coverage.
axioms (3)
  • domain assumption LLM test generation with gpt-4o-mini is representative of LLM-based test generation
    All conclusions rely on one model; the generalizability table with claude-3.5-haiku and deepseek-v3.1 is a partial counter but still a small set.
  • domain assumption Jacoco line and branch coverage are the right proxies for test sufficiency
    The paper uses Jacoco coverage of the focal method only, not mutation score or fault-detection capability.
  • domain assumption HITS dataset (110 methods) is a representative sample of complex real-world methods
    The evaluation is entirely on one dataset from 10 projects; generalizability outside this sample is assumed.

pith-pipeline@v1.3.0-alltime-deepseek · 20046 in / 8470 out tokens · 73829 ms · 2026-08-03T21:37:05.509608+00:00 · methodology

0 comments
read the original abstract

Automated unit test generation using large language models (LLMs) holds great promise but often struggles with generating tests that are both correct and maintainable in real-world projects. This paper presents KTester, a novel framework that integrates project-specific knowledge and testing domain knowledge to enhance LLM-based test generation. Our approach first extracts project structure and usage knowledge through static analysis, which provides rich context for the model. It then employs a testing-domain-knowledge-guided separation of test case design and test method generation, combined with a multi-perspective prompting strategy that guides the LLM to consider diverse testing heuristics. The generated tests follow structured templates, improving clarity and maintainability. We evaluate KTester on multiple open-source projects, comparing it against state-of-the-art LLM-based baselines using automatic correctness and coverage metrics, as well as a human study assessing readability and maintainability. Results demonstrate that KTester significantly outperforms existing methods across six key metrics, improving execution pass rate by 5.69% and line coverage by 8.83% over the strongest baseline, while requiring less time and generating fewer test cases. Human evaluators also rate the tests produced by KTester significantly higher in terms of correctness, readability, and maintainability, confirming the practical advantages of our knowledge-driven framework.

Figures

Figures reproduced from arXiv: 2511.14224 by Anji Li, Dekun Dai, Mingwei Liu, Yanlin Wang, Zheng Pei, Zhenxi Chen, Zibin Zheng, Zike Li.

Figure 2
Figure 2. Figure 2: Motivational Examples(b): Insufficient Assert State [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 1
Figure 1. Figure 1: Motivational Examples(a): Setting Tested Object [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Motivational Examples(c): Hard Encoding Values [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The Framework of KTester Context Construction) and enable accurate call path tracing for generating invocation examples (Section 2.1.2). class Lexer_nextToken_Test { private Lexer lexer; private Token token; …… @BeforeEach void setupBeforeEach() { // Initialize token for tests token = new Token(); ……; } @AfterEach void teardownAfterEach() { // Close lexer lexer.close(); ……; } @Test void testNextToken_Valid… view at source ↗
Figure 6
Figure 6. Figure 6: Illustration of extracting invocation examples from [PITH_FULL_IMAGE:figures/full_fig_p004_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Prompts for multi-view test case design. [PITH_FULL_IMAGE:figures/full_fig_p006_8.png] view at source ↗
Figure 7
Figure 7. Figure 7: Prompt for test class framework generation. [PITH_FULL_IMAGE:figures/full_fig_p006_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: Prompt for test method transformation. metric [21] by combining with method usage similarity 𝑆𝑖𝑚𝑚 (𝑎, 𝑏) and field usage similarity 𝑆𝑖𝑚𝑓 (𝑎, 𝑏): 𝑆𝑖𝑚(𝑎, 𝑏) = 𝑆𝑖𝑚𝑚 (𝑎, 𝑏) + 𝑆𝑖𝑚𝑓 (𝑎, 𝑏) (1) 𝑆𝑖𝑚𝑚 (𝑎, 𝑏) = |𝑀(𝑎) ∩ 𝑀(𝑏)| |𝑀(𝑎) ∪ 𝑀(𝑏)| (2) 𝑆𝑖𝑚𝑓 (𝑎, 𝑏) = |𝐹 (𝑎) ∩ 𝐹 (𝑏)| |𝐹 (𝑎) ∪ 𝐹 (𝑏)| (3) Here, 𝑀(𝑎) and 𝐹 (𝑎) denote the sets of methods and fields used by function 𝑎, respectively. We select the top-N most similar … view at source ↗
Figure 10
Figure 10. Figure 10: Test Classes generated by HITS and KTester. [PITH_FULL_IMAGE:figures/full_fig_p009_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Score Distribution of Correctness, Readability and [PITH_FULL_IMAGE:figures/full_fig_p011_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Knowledge-Guided Synthetic Bug Feedback for LLM-Based Unit Test Generation

    cs.SE 2026-07 conditional novelty 6.5

    Mechanism-guided synthetic-bug feedback raises real-bug detection of LLM unit tests to 72.67% RBDR on 172 Defects4J tasks, outperforming mutation and coverage baselines by ~20 points.

  2. TATG: Tracking-Aware Testing Objective for LLM-based Test Generation

    cs.SE 2026-07 conditional novelty 6.0

    Tracking individual static and feedback testing objectives across a coverage-then-mutation two-stage LLM workflow raises line/branch coverage and mutation score on complex Java methods versus KTester and PANTA.

Reference graph

Works this paper leans on

62 extracted references · 5 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Insights into regression testing (software testing),

    H. K. Leung and L. White, “Insights into regression testing (software testing), ” in Proceedings. Conference on Software Maintenance-1989 . IEEE, 1989, pp. 60–69

    1989

  2. [2]

    A survey of unit testing practices,

    P. Runeson, “A survey of unit testing practices, ”IEEE software, vol. 23, no. 4, pp. 22–29, 2006

    2006

  3. [3]

    A survey on unit testing practices and problems,

    E. Daka and G. Fraser, “A survey on unit testing practices and problems, ” in2014 IEEE 25th International Symposium on Software Reliability Engineering . IEEE, 2014, pp. 201–211

    2014

  4. [4]

    The oracle problem in software testing: A survey,

    E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “The oracle problem in software testing: A survey, ” IEEE Trans. Software Eng. , vol. 41, no. 5, pp. 507–525, 2015. [Online]. Available: https://doi.org/10.1109/TSE.2014.2372785

    arXiv 2015

  5. [5]

    A large-scale evaluation of automated unit test genera- tion using evosuite,

    G. Fraser and A. Arcuri, “A large-scale evaluation of automated unit test genera- tion using evosuite, ”ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 24, no. 2, pp. 1–42, 2014

    2014

  6. [6]

    A theoretical and empirical study of search-based testing: Local, global, and hybrid search,

    M. Harman and P. McMinn, “A theoretical and empirical study of search-based testing: Local, global, and hybrid search, ”IEEE Transactions on Software Engineer- ing, vol. 36, no. 2, pp. 226–247, 2009

    2009

  7. [7]

    Call me maybe: Using nlp to automatically generate unit test cases respecting temporal constraints,

    A. Blasi, A. Gorla, M. D. Ernst, and M. Pezzè, “Call me maybe: Using nlp to automatically generate unit test cases respecting temporal constraints, ” in37th IEEE/ACM International Conference on Automated Software Engineering , 2022, pp. 1–11

    2022

  8. [8]

    Interevo-tr: Interactive evolutionary test generation with readability assessment,

    P. Delgado-Pérez, A. Ramírez, K. J. Valle-Gómez, I. Medina-Bulo, and J. R. Romero, “Interevo-tr: Interactive evolutionary test generation with readability assessment, ” IEEE Transactions on Software Engineering , 2022

    2022

  9. [9]

    The daikon system for dynamic detection of likely invariants,

    M. D. Ernst, J. H. Perkins, P. J. Guo, S. McCamant, C. Pacheco, M. S. Tschantz, and C. Xiao, “The daikon system for dynamic detection of likely invariants, ”Science of computer programming, vol. 69, no. 1-3, pp. 35–45, 2007

    2007

  10. [10]

    Dysy: Dynamic symbolic execution for invariant inference,

    C. Csallner, N. Tillmann, and Y. Smaragdakis, “Dysy: Dynamic symbolic execution for invariant inference, ” inProceedings of the 30th international conference on Software engineering, 2008, pp. 281–290

    2008

  11. [11]

    Characteristic studies of loop problems for structural test generation via symbolic execution,

    X. Xiao, S. Li, T. Xie, and N. Tillmann, “Characteristic studies of loop problems for structural test generation via symbolic execution, ” in2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 2013, pp. 246–256

    2013

  12. [12]

    The fuzzing book,

    A. Zeller, R. Gopinath, M. Böhme, G. Fraser, and C. Holler, “The fuzzing book, ” 2019

    2019

  13. [13]

    Feedback-directed random test generation,

    C. Pacheco, S. K. Lahiri, M. D. Ernst, and T. Ball, “Feedback-directed random test generation, ” in29th International Conference on Software Engineering (ICSE’07) . IEEE, 2007, pp. 75–84

    2007

  14. [14]

    An industrial evaluation of unit test generation: Finding real faults in a financial application,

    M. M. Almasi, H. Hemmati, G. Fraser, A. Arcuri, and J. Benefelds, “An industrial evaluation of unit test generation: Finding real faults in a financial application, ” in 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP) . IEEE, 2017, pp. 263–272

    2017

  15. [15]

    Ktester,

    “Ktester, ” 2025. [Online]. Available: https://github.com/SYSUSELab/KTester

    2025

  16. [16]

    Control flow analysis,

    F. E. Allen, “Control flow analysis, ”SIGPLAN Not., vol. 5, no. 7, p. 1–19, Jul. 1970. [Online]. Available: https://doi.org/10.1145/390013.808479

    arXiv 1970

  17. [17]

    http://javaparser.org/,

    “http://javaparser.org/, ” 2025

    2025

  18. [18]

    https://spoon.gforge.inria.fr/about.html,

    “https://spoon.gforge.inria.fr/about.html, ” 2025

    2025

  19. [19]

    ChatUniTest: A Framework for LLM-Based Test Generation,

    Y. Chen, Z. Hu, C. Zhi, J. Han, S. Deng, and J. Yin, “ChatUniTest: A Framework for LLM-Based Test Generation, ” in Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering , ser. FSE

  20. [20]

    HITS: High-coverage LLM-based Unit Test Generation via Method Slicing,

    Z. Wang, K. Liu, G. Li, and Z. Jin, “HITS: High-coverage LLM-based Unit Test Generation via Method Slicing, ” inProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering , ser. ASE ’24. New York, NY, USA: Association for Computing Machinery, Oct. 2024, pp. 1258–1268. [Online]. Available: https://dl.acm.org/doi/10.1145/369...

    arXiv 2024

  21. [21]

    Lois de distribution florale dans la zone alpine,

    P. Jaccard, “Lois de distribution florale dans la zone alpine, ”Bulletin de la Société vaudoise des sciences naturelles , vol. 38, pp. 69–130, 01 1902

    1902

  22. [22]

    Evaluating and improving chatgpt for unit test generation,

    Z. Yuan, Y. Lou, M. Liu, S. Ding, K. Wang, Y. Chen, and X. Peng, “Evaluating and improving chatgpt for unit test generation, ” Proc. ACM Softw. Eng. , vol. 1, no. FSE, 2024. [Online]. Available: https://doi.org/10.1145/3660783

    doi:10.1145/3660783 2024

  23. [23]

    https://github.com/zju-aces-ise/chatunitest-maven-plugin,

    “https://github.com/zju-aces-ise/chatunitest-maven-plugin, ” 2025

    2025

  24. [24]

    https://platform.openai.com,

    “https://platform.openai.com, ” 2025

    2025

  25. [25]

    Deljouyi, R

    A. Deljouyi, R. Koohestani, M. Izadi, and A. Zaidman, Leveraging Large Language Models for Enhancing the Understandability of Generated Unit Tests . IEEE Press, 2025, p. 1449–1461. [Online]. Available: https://doi.org/10.1109/ICSE55347.2025. 00032

    arXiv 2025

  26. [26]

    Utgen replication package,

    “Utgen replication package, ” 2025. [Online]. Available: https://github.com/ amirdeljouyi/UTGen

    2025

  27. [27]

    EvoSuite: automatic test suite generation for object- oriented software,

    G. Fraser and A. Arcuri, “EvoSuite: automatic test suite generation for object- oriented software, ” inProceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering , ser. ESEC/FSE ’11. New York, NY, USA: Association for Computing Machinery, Sep. 2011, pp. 416–419. [Online]. Available: https://dl.acm...

    arXiv 2011

  28. [28]

    codellama-7b,

    “codellama-7b, ” 2025

    2025

  29. [29]

    TestART: Improving LLM-based Unit Testing via Co-evolution of Automated Generation and Repair Iteration,

    S. Gu, Q. Zhang, K. Li, C. Fang, F. Tian, L. Zhu, J. Zhou, and Z. Chen, “TestART: Improving LLM-based Unit Testing via Co-evolution of Automated Generation and Repair Iteration, ” Mar. 2025, arXiv:2408.03095 [cs]. [Online]. Available: http://arxiv.org/abs/2408.03095

    Pith/arXiv arXiv 2025

  30. [30]

    https://www.jacoco.org/jacoco/,

    “https://www.jacoco.org/jacoco/, ” 2025

    2025

  31. [31]

    claude-3-5-haiku-20241022 model overview,

    “claude-3-5-haiku-20241022 model overview, ” 2024. [Online]. Available: https: //docs.claude.com/en/docs/about-claude/models/overview#legacy-models

    2024

  32. [32]

    deepseek-v3.1 release,

    “deepseek-v3.1 release, ” 2025. [Online]. Available: https://api-docs.deepseek.com/ news/news250821

    2025

  33. [33]

    ASTER: natural and multi-language unit test generation with llms,

    R. Pan, M. Kim, R. Krishna, R. Pavuluri, and S. Sinha, “ASTER: natural and multi-language unit test generation with llms, ” in47th IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice, SEIP@ICSE 2025, Ottawa, ON, Canada, April 27 - May 3, 2025 . IEEE, 2025, pp. 413–424. [Online]. Available: https://doi.org/10.1109/IC...

    arXiv 2025

  34. [34]

    Achievements, open problems and challenges for search based software testing,

    M. Harman, Y. Jia, and Y. Zhang, “Achievements, open problems and challenges for search based software testing, ” in2015 IEEE 8th international conference on software testing, verification and validation (ICST) . IEEE, 2015, pp. 1–12

    2015

  35. [35]

    Search-based software testing: Past, present and future,

    P. McMinn, “Search-based software testing: Past, present and future, ” in2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops. IEEE, 2011, pp. 153–163

    2011

  36. [36]

    A survey of symbolic execution techniques,

    R. Baldoni, E. Coppa, D. C. D’elia, C. Demetrescu, and I. Finocchi, “A survey of symbolic execution techniques, ”ACM Computing Surveys (CSUR), vol. 51, no. 3, pp. 1–39, 2018

    2018

  37. [37]

    Klee: unassisted and automatic generation of high-coverage tests for complex systems programs

    C. Cadar, D. Dunbar, D. R. Engleret al., “Klee: unassisted and automatic generation of high-coverage tests for complex systems programs. ” inOSDI, vol. 8, 2008, pp. 209–224. KTester: Leveraging Domain and Testing Knowledge for More Effective LLM-based Test Generation Conference’17, July 2017, Washington, DC, USA

    2008

  38. [38]

    Unleashing mayhem on binary code,

    S. K. Cha, T. Avgerinos, A. Rebert, and D. Brumley, “Unleashing mayhem on binary code, ” in2012 IEEE Symposium on Security and Privacy . IEEE, 2012, pp. 380–394

    2012

  39. [39]

    Unit test case generation with transformers and focal context,

    M. Tufano, D. Drain, A. Svyatkovskiy, S. K. Deng, and N. Sundaresan, “Unit test case generation with transformers and focal context, ” arXiv preprint arXiv:2009.05617, 2020

    Pith/arXiv arXiv 2009

  40. [40]

    Effective test generation using pre-trained large language models and mutation testing,

    A. M. Dakhel, A. Nikanjam, V. Majdinasab, F. Khomh, and M. C. Desmarais, “Effective test generation using pre-trained large language models and mutation testing, ”Information and Software Technology, vol. 171, p. 107468, 2024

    2024

  41. [41]

    Large language models are edge-case generators: Crafting unusual programs for fuzzing deep learning libraries,

    Y. Deng, C. S. Xia, C. Yang, S. D. Zhang, S. Yang, and L. Zhang, “Large language models are edge-case generators: Crafting unusual programs for fuzzing deep learning libraries, ” inProceedings of the 46th IEEE/ACM international conference on software engineering, 2024, pp. 1–13

    2024

  42. [42]

    Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models,

    Y. Deng, C. S. Xia, H. Peng, C. Yang, and L. Zhang, “Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models, ” in Proceedings of the 32nd ACM SIGSOFT international symposium on software testing and analysis, 2023, pp. 423–435

    2023

  43. [43]

    Learning deep semantics for test completion,

    P. Nie, R. Banerjee, J. J. Li, R. J. Mooney, and M. Gligoric, “Learning deep semantics for test completion, ”arXiv preprint arXiv:2302.10166, 2023

    Pith/arXiv arXiv 2023

  44. [44]

    exlong: Generating exceptional behavior tests with large language models,

    J. Zhang, Y. Liu, P. Nie, J. J. Li, and M. Gligoric, “exlong: Generating exceptional behavior tests with large language models, ”arXiv preprint arXiv:2405.14619, 2024

    Pith/arXiv arXiv 2024

  45. [45]

    An empirical evaluation of using large language models for automated unit test generation,

    M. Schäfer, S. Nadi, A. Eghbali, and F. Tip, “An empirical evaluation of using large language models for automated unit test generation, ”IEEE Transactions on Software Engineering, vol. 50, no. 1, pp. 85–105, 2023

    2023

  46. [46]

    Testspark: Intellij idea’s ultimate test generation companion,

    A. Sapozhnikov, M. Olsthoorn, A. Panichella, V. Kovalenko, and P. Derakhshanfar, “Testspark: Intellij idea’s ultimate test generation companion, ” inProceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings, 2024, pp. 30–34

    2024

  47. [47]

    Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models,

    C. Lemieux, J. P. Inala, S. K. Lahiri, and S. Sen, “Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models, ” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 2023, pp. 919–931

    2023

  48. [48]

    Coverup: Effective high coverage test generation for python,

    J. Altmayer Pizzorno and E. D. Berger, “Coverup: Effective high coverage test generation for python, ”Proceedings of the ACM on Software Engineering , vol. 2, no. FSE, pp. 2897–2919, 2025

    2025

  49. [49]

    Code- aware prompting: A study of coverage-guided test generation in regression setting using llm,

    G. Ryan, S. Jain, M. Shang, S. Wang, X. Ma, M. K. Ramanathan, and B. Ray, “Code- aware prompting: A study of coverage-guided test generation in regression setting using llm, ”Proceedings of the ACM on Software Engineering , vol. 1, no. FSE, pp. 951–971, 2024

    2024

  50. [51]

    Automated program repair via conversation: Fixing 162 out of 337 bugs for $0.42 each using chatgpt,

    C. S. Xia and L. Zhang, “Automated program repair via conversation: Fixing 162 out of 337 bugs for $0.42 each using chatgpt, ” in Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis , ser. ISSTA

  51. [52]

    Out of context: How important is local context in neural program repair?

    J. A. Prenner2024 and R. Robbes, “Out of context: How important is local context in neural program repair?” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , ser. ICSE ’24. New York, NY, USA: Association for Computing Machinery, 2024. [Online]. Available: https://doi.org/10.1145/3597503.3639086

    arXiv 2024

  52. [53]

    New York, NY, USA: Association for Computing Machinery, 2024, p. 819–831. [Online]. Available: https://doi.org/10.1145/3650212.3680323

    arXiv 2024

  53. [54]

    Impact of code language models on automated program repair,

    N. Jiang, K. Liu, T. Lutellier, and L. Tan, “Impact of code language models on automated program repair, ” inProceedings of the 45th International Conference on Software Engineering, ser. ICSE ’23. IEEE Press, 2023, p. 1430–1442. [Online]. Available: https://doi.org/10.1109/ICSE48619.2023.00125

    arXiv 2023

  54. [55]

    The plastic surgery hypothesis in the era of large language models,

    C. S. Xia, Y. Ding, and L. Zhang, “The plastic surgery hypothesis in the era of large language models, ” in Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering , ser. ASE ’23. IEEE Press, 2024, p. 522–534. [Online]. Available: https://doi.org/10.1109/ASE56229.2023.00047

    arXiv 2024

  55. [56]

    Nl2fix: Generating functionally correct code edits from bug descriptions,

    S. Fakhoury, S. Chakraborty, M. Musuvathi, and S. K. Lahiri, “Nl2fix: Generating functionally correct code edits from bug descriptions, ” in Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings , ser. ICSE-Companion ’24. New York, NY, USA: Association for Computing Machinery, 2024, p. 410–411. [Onli...

    arXiv 2024

  56. [57]

    Towards effectively leveraging execution traces for program repair with code LLMs,

    M. Haque, P. Babkin, F. Farmahinifarahani, and M. Veloso, “Towards effectively leveraging execution traces for program repair with code LLMs, ” inProceedings of the 4th International Workshop on Knowledge-Augmented Methods for Natural Language Processing, W. Shi, W. Yu, A. Asai, M. Jiang, G. Durrett, H. Hajishirzi, and L. Zettlemoyer, Eds. Albuquerque, Ne...

    2025

  57. [58]

    Repoformer: selective retrieval for repository-level code completion,

    D. Wu, W. U. Ahmad, D. Zhang, M. K. Ramanathan, and X. Ma, “Repoformer: selective retrieval for repository-level code completion, ” inProceedings of the 41st International Conference on Machine Learning , ser. ICML’24. JMLR.org, 2024

    2024

  58. [59]

    Reacc: A retrieval-augmented code completion framework,

    S. Lu, N. Duan, H. Han, D. Guo, S. won Hwang, and A. Svyatkovskiy, “Reacc: A retrieval-augmented code completion framework, ” 2022. [Online]. Available: https://arxiv.org/abs/2203.07722

    Pith/arXiv arXiv 2022

  59. [60]

    Knowledge graph based repository-level code generation,

    M. Athale and V. Vaddina, “Knowledge graph based repository-level code generation, ” in 2025 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code) . IEEE, May 2025, p. 169–176. [Online]. Available: http://dx.doi.org/10.1109/LLM4Code66737.2025.00026

    arXiv 2025

  60. [61]

    Coderag: Supportive code retrieval on bigraph for real-world code generation,

    J. Li, X. Shi, K. Zhang, L. Li, G. Li, Z. Tao, J. Li, F. Liu, C. Tao, and Z. Jin, “Coderag: Supportive code retrieval on bigraph for real-world code generation, ”

  61. [2024]

    2024, pp

    New York, NY, USA: Association for Computing Machinery, Jul. 2024, pp. 572–576. [Online]. Available: https://dl.acm.org/doi/10.1145/3663529.3663801

    arXiv 2024

  62. [2025]

    Available: https://arxiv.org/abs/2504.10046

    [Online]. Available: https://arxiv.org/abs/2504.10046

    arXiv