Pith. sign in

REVIEW 4 major objections 5 minor 47 references

Benchmarking LLMs for Unit Test Generation from Real-World Functions

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that prior LLM unit-test benchmarks are inflated by memorization and simple code, and that a new decontaminated, high-complexity benchmark of 3,909 real-world Python functions shows much lower true performance and a…

desk verdict Useful benchmark, but the 'unleaked' claim and the PLT-based memorization conclusion do not hold as stated; the abstract also misreports accuracy. read the letter →

arxiv 2508.00408 v1 pith:C4E35LTM submitted 2025-08-01 cs.SE cs.CL

Dong Huang , Jie M. Zhang , Mark Harman , Qianru Zhang , Mingzhe Du , See-Kiong Ng This is my paper · ORCID
classification cs.SEcs.CL
keywords unittestgenerationlargelanguagemodelsdatacontaminationbenchmarkdesigncyclomaticcomplexitycodecoveragemutationtestingPython
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Existing benchmarks make LLMs look better at writing unit tests than they are, the paper argues, because the functions are too simple and because the tests may already sit in the models' training data. To fix this, the paper builds ULT: 3,909 real-world Python functions, each with cyclomatic complexity at least 10, from which functions with already-public tests were filtered out. Across twelve current LLMs, generated tests on ULT pass at 12.69% on the first query and reach 45.10% line coverage and 30.22% branch coverage after five queries, far below the same models' scores on a LeetCode-derived baseline benchmark. ULT also shows a strong correlation between test-generation accuracy and coding ability (Pearson $r=0.79$, $p=0.002$), while the simpler and leaked benchmarks do not. If the filtering worked as intended, ULT and its leaked counterpart PLT give a way to separate genuine test-writing reasoning from memorization.

What carries the argument

The mechanism that carries the argument is a paired-benchmark construction. Functions are drawn from a large real-world Python corpus, kept only if they are self-contained and testable, and filtered by cyclomatic complexity (at least 10, mean 14.87), a count of decision points and independent paths through a function's control flow. The decontamination step searches the same corpus for test definitions or assertions matching each function's name; functions with matches form the leaked set PLT, and functions without matches form ULT. The paper's inferences come from comparing ULT with PLT at matched complexity and from correlating ULT scores with an external coding-ability measure: if ULT and PLT differ only in leakage, any performance gap is a direct estimate of memorization.

What would settle it

Check whether the withheld ULT test suites appear in the pretraining corpora of the twelve evaluated models, for example under renamed test functions or as assertions embedded in other files; if a substantial fraction are found, the 'unleaked' label, the ULT/PLT gap, and the memorization explanation would fail.

Watch

Extended reading notes

Core claim

The paper's central claim is that a benchmark built from real-world, structurally complex functions whose tests have been removed provides a more challenging and more valid measure of LLM unit-test generation. The authors report that on ULT, model-generated test cases achieve much lower accuracy, line coverage, branch coverage, and mutation score than on the comparison benchmark and than on the paired leaked benchmark PLT. The gap persists even when functions are matched on cyclomatic complexity, which the paper interprets as evidence that ULT demands reasoning rather than recall. The paper also reports that test-generation performance on ULT correlates strongly with coding ability ($r=0.79$, $p=0.002$), whereas the correlations on the simpler and leaked benchmarks are weak, supporting the conclusion that ULT measures generalization.

Load-bearing premise

The load-bearing premise is that the filtering step really removed every pre-existing way the models could have memorized these tests: it assumes that searching for functions with matching test names or assertions catches all leakage, and that the remaining ULT functions differ from PLT only in that leakage.

Editorial extensions

If this is right

  • LLM test-generation results on existing benchmarks may substantially overstate real-world skill; on ULT, twelve models averaged 12.69% accuracy at the first query, with 45.10% line coverage and 30.22% branch coverage after five queries.
  • Performance on ULT is not saturated: line and branch coverage keep rising beyond twenty generated tests, so the benchmark can distinguish models over long interactions.
  • The leaked benchmark PLT scores much higher than ULT at equal cyclomatic complexity, quantifying how much of prior reported performance could come from memorized test cases.
  • Accuracy on the comparison benchmark decays sharply as more tests are requested, which the paper reads as a contamination signature: memorized tests help at the first query but not when diversification is required.
  • A strong correlation between ULT test-generation accuracy and general coding ability suggests that test generation on clean, complex functions reflects a model's true generalization rather than recall.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the decontamination assumption holds, then a large part of what earlier test-generation benchmarks measured was memorization, and new benchmarks should treat public test suites as contaminated by default.
  • The same paired clean-versus-leaked design could be applied to other programming languages and to class-level or bug-reproduction test tasks, where leakage is likely even harder to detect.
  • The strong ULT correlation with coding ability suggests test generation could serve as a cheap diagnostic signal for general code understanding during model development, before expensive human evaluation.
  • Because ULT withholds ground-truth tests, future comparisons of new models should use the provided evaluation script rather than static test files, otherwise the benchmark could become contaminated after public release.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces ULT, a benchmark of 3,909 real-world Python functions extracted from The Stack v2, selected to have cyclomatic complexity at least 10, self-contained and testable code, and to be filtered with a decontamination heuristic that searches for function-name-based test definitions and assertions. The authors also construct PLT, a superset containing all ULT functions plus functions flagged as potentially contaminated, and they evaluate 12 LLMs on an iterative K-query test generation task, comparing accuracy, line coverage, branch coverage, and mutation score against TestEval and PLT. They report that ULT is substantially harder than both, and they argue that ULT's stronger correlation with BigCodeBench performance shows it measures generalization rather than memorization.

Significance. If the central claims are correct, ULT would be a timely and valuable resource for the LLM-based test generation community, addressing realistic concerns about data contamination and task complexity, and the public release philosophy of withholding ground-truth tests while providing an evaluation script is a thoughtful response to future contamination. The study also has strengths in experimental setup: a fixed deterministic decoding configuration, a unified Docker execution environment, and a broad model selection. However, the current manuscript contains a clear inconsistency in the headline accuracy figure, the PLT control is not actually a controlled paired comparison, and the decontamination evidence is weaker than the text claims. Because these issues directly affect the paper's main contributions, the manuscript requires substantive revision before the findings can be considered reliable.

major comments (4)
  1. [Abstract; Section 5.1.1; Table 2; Table 3] The abstract and conclusion state that LLM-generated tests on ULT achieve 41.32% accuracy, but Table 2 reports ULT Pass@1 of 12.69% and Pass@5 of 12.57% on average; the value 41.32% corresponds instead to the Qwen2.5-Coder-32B line-coverage at k=1 in Table 3. This is a load-bearing numerical error because the headline comparison between ULT, TestEval, and PLT is built on these numbers, and the same incorrect figure is repeated in the introduction and conclusion.
  2. [Section 3.2; Section 6.2.1; Figure 2] PLT is defined as a superset containing all ULT functions plus all functions identified as potentially contaminated, yet Section 6.2.1 claims that PLT and ULT were collected in exactly the same way with only one difference being whether the test cases were available to LLMs for training. A superset is not a paired control: the higher PLT performance could arise from the different composition of the two task sets rather than from memorization, and even the cyclomatic-complexity-binned comparison in Figure 2 compares different function sets within each bin. To support the memorization attribution, PLT should be restructured as a matched subset (e.g., one-to-one matching on cyclomatic complexity, code length, and repository origin) or the analysis restricted to a matched subpopulation.
  3. [Section 3.2 (Decontamination)] The decontamination filter searches The Stack v2 for function-name-based markers such as 'def test_func_name' or 'assert func_name', but The Stack v2 is not the pretraining corpus for all twelve evaluated models, and test cases can be parametrized, aliased, or written without repeating the function name. Therefore the absence of a matching test file does not establish that a function's tests were unavailable to a given LLM during pretraining. Since the 'unleaked' status of ULT and the interpretation of the ULT/PLT gap as memorization both depend on this filter, the authors need stronger evidence, such as content-based membership checks for each model family or checks against corpora known to be used in training those models.
  4. [Section 6.2.2; Figure 3] The claim that ULT's correlation with coding ability is stronger than that of TestEval and PLT is not statistically justified. The paper reports only per-benchmark Pearson correlations and never tests whether r = 0.79 is significantly different from r = 0.56 or r = 0.52 at n = 12; a Fisher r-to-z test or a bootstrap difference test is needed. In addition, Figure 3 shows significant rank correlations for TestEval (Spearman rho = 0.71, p = 0.010; Kendall tau = 0.55, p = 0.014), contradicting the text's assertion that there is 'no evidence for any correlation' for TestEval. The authors should report the full correlation matrix, include formal tests of correlation differences, and revise the conclusion accordingly.
minor comments (5)
  1. [Section 5.1.3] The heading '6 RQ1.3' appears to be a leftover formatting artifact and should be removed.
  2. [Figure 2 caption] The caption says the figure shows 'four different LLMs', but it shows twelve panels; the caption should be corrected.
  3. [Table 5 footnote] The mutation testing subsection states that only a random sample of 1,000 PLT functions (outside the 3,909 ULT functions) was used; the paper should clarify how this sample is representative and report confidence intervals for the PLT mutation score.
  4. [Section 7.4] The discussion says that a self-contained evaluation script can compute coverage and mutation metrics without exposing ground-truth tests, but it is unclear how mutation analysis is performed without reference tests; the relationship between the evaluation script and the withheld ground-truth tests should be specified.
  5. [Throughout] There are minor grammar and typographical issues, including 'avaliable' in Section 3.2, 'we will requires' in Section 3.2, and 'free LM performance' in Section 6.2.1, which should be corrected in a copyedit pass.

Circularity Check

1 steps flagged · score 2.0 of 10

No load-bearing circularity: the core difficulty claim is benchmarked externally, but the PLT/ULT memorization analysis is a self-referential validation of the decontamination filter.

  1. self definitional [Section 3.2 (Decontamination) and Section 6.2.1 (RQ3.1)]
    "Our second benchmark, PLT, is a superset that contains all the functions from ULT plus all the functions that were identified as potentially contaminated. ... PLT and ULT were collected in exactly the same way by the same researchers using the same process, with only one difference: whether or not the test cases were available to LLMs for training. ... This confirms that our decontamination process was successful in creating a benchmark that genuinely challenges the models' reasoning abilities rather than their capacity for memorization."

    The decontamination filter defines both sides of the comparison: ULT is the complement of the flagged set, and PLT is ULT plus the flagged set. The RQ3.1 conclusion that the performance gap is 'undoubtedly' due to memorization treats PLT as a matched control, but the paper's own definition makes PLT a superset with a different function composition. The gap is therefore not evidence that the same functions are easier when leaked; it can reflect the addition of the flagged functions. Since the filter's output is used to validate the filter itself, the decontamination claim is self-referential rather than independently tested.

full rationale

The paper's central claim that ULT is more challenging and more realistic is grounded in external benchmarks: absolute Pass@k, coverage, and mutation scores on ULT are compared with TestEval, and cyclomatic-complexity distributions are compared with TestEval and TestGenEval. These comparisons do not reduce to the paper's inputs. The BigCodeBench correlation (r=0.79) is also an external, independent proxy. The only self-referential element is the ULT/PLT contrast: PLT is constructed as ULT plus all functions flagged by the paper's own search for test names, and Section 6.2.1 then uses the ULT/PLT gap to confirm that the decontamination filter removed memorized tests. Because PLT is a superset rather than a matched pair, the 'only one difference' assertion is false by construction, and the gap cannot validate the filter. This is a localized circularity in the memorization analysis, not in the main difficulty result. No load-bearing self-citations or fitted-parameter-as-prediction steps were found.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The benchmark's central claims rest on several design choices that are asserted rather than independently validated: a complexity threshold, a corpus search as a proxy for contamination, comparability between ULT and PLT, and the validity of BigCodeBench as a coding-ability proxy.

free parameters (6)
  • cyclomatic_complexity_threshold = 10
    Minimum cyclomatic complexity chosen by hand to select complex functions; affects the entire dataset and all headline metrics.
  • testability_debug_limit = 3
    Maximum GPT-4o repair attempts before discarding a function during curation; chosen by hand and affects which functions enter ULT.
  • query_count_K = 5 (main), 20 (analysis)
    Number of iterative test cases requested; a design choice that shapes Pass@k, coverage, and mutation values.
  • mutation_timeout = 120 seconds
    Per-mutant execution timeout for Cosmic-Ray; chosen by hand and affects reported mutation scores.
  • PLT_mutation_sample_size = 1,000 functions
    PLT is large, so mutation testing was run only on 3,909 ULT functions plus 1,000 other PLT functions; this sampling choice affects comparability.
  • temperature = 0.0
    Greedy decoding setting chosen to reduce randomness; reproducible but not a scientific free parameter.
assumptions (5)
  • domain assumption Cyclomatic complexity >=10 is a meaningful proxy for the structural complexity that makes test generation hard.
    Used in Section 3.2 as the primary filter; the paper acknowledges it correlates with size and can be controversial.
  • domain assumption Absence of 'def test_func_name' or 'assert func_name' in The Stack v2 implies the function's tests are not in LLM training corpora.
    Load-bearing for the unleaked claim in Section 3.2; no direct check of the 12 models' training data.
  • domain assumption PLT and ULT are otherwise identical because they use the same collection pipeline.
    Used in Section 6.2.1 to attribute the performance gap to memorization; functions with public tests may differ in maintenance, domain, or popularity.
  • domain assumption BigCodeBench performance is a valid proxy for intrinsic coding ability.
    Used in Section 6.2.2 to interpret correlations as evidence that ULT measures generalization.
  • domain assumption GPT-4o can detect untestable functions and repair bugs without changing the function's realism.
    Used in Section 3.2 Testability Guarantee; repaired functions may no longer be exact real-world code.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Benchmarking LLMs for Unit Test Generation from Real-World Functions." pith.science (2026). https://pith.science/paper/C4E35LTM

@misc{pith2026250800408,
  author       = {Pith},
  title        = {Pith review of: Benchmarking LLMs for Unit Test Generation from Real-World Functions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C4E35LTM}},
  note         = {Machine review of arXiv:2508.00408}
}
read the original abstract

Recently, large language models (LLMs) have shown great promise in automating unit test generation, significantly reducing the manual effort required by developers. To effectively evaluate the capabilities of LLMs in this domain, it is crucial to have a well-designed benchmark that accurately reflects real-world scenarios and mitigates common pitfalls. Existing LLM test generation benchmarks are limited by two critical drawbacks: data contamination and structurally simple function code. As a result, we often cannot rely on the validity of scientific conclusions drawn from empirical studies using these limited benchmarks. The empirical evidence presented may be biased due to contamination and may fail to generalize beyond toy programs due to structural simplicity. To address these problems, we introduce ULT (UnLeakedTestbench), a new benchmark specifically designed for function-level unit test generation from real-world Python functions. ULT is constructed through a multi-stage curation process that ensures high cyclomatic complexity and mitigates test case contamination. With 3,909 carefully selected function-level tasks, ULT provides a more realistic and challenging evaluation of LLMs' test generation capabilities. We also provide PLT (PreLeakedTestbench), a pair benchmark of ULT with leaked tests designed to enable a controlled analysis of memorization versus reasoning in test generation. Our evaluation results demonstrate that ULT is significantly more challenging. For example, test cases generated by LLMs only achieve 41.32\%, 45.10\%, 30.22\%, and 40.21\% for accuracy, statement coverage, branch coverage, and mutation score on average for all LLMs, respectively. These results are substantially lower than the corresponding metrics on TestEval (91.79\%, 92.18\%, 82.04\%, and 49.69\%) and PLT (47.07\%, 55.13\%, 40.07\%, and 50.80\%).

Figures

Figures reproduced from arXiv: 2508.00408 by the authors.

Figure 1
Figure 1. Branch Coverage as a function of Cyclomatic Complexity for ULT and TestEval across different LLMs. [PITH_FULL_IMAGE:figures/full_fig_p015_1.png] view at source ↗
Figure 4
Figure 4. On ULT, the accuracy of generated tests remains relatively low but stable across all values of [PITH_FULL_IMAGE:figures/full_fig_p018_4.png] view at source ↗
Figure 2
Figure 2. Branch Coverage as a function of Cyclomatic Complexity for ULT and PLT across four different LLMs. [PITH_FULL_IMAGE:figures/full_fig_p019_2.png] view at source ↗
Figures from the paper (2 more)
Figure 3
Figure 3. Figure 3: Correlation analysis between code generation performance on BigCodeBench (x-axis) and test generation performance metrics [PITH_FULL_IMAGE:figures/full_fig_p020_3.png]
Figure 4
Figure 4. Figure 4: Performance trends on ULT (solid lines) and TestEval (dashed lines) as the number of generated test cases ( [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 30 canonical work pages

  1. [1]

    An orchestrated survey of methodologies for automated software test case generation,

    S. Anand, E. K. Burke, T. Y. Chen, J. Clark, M. B. Cohen, W. Grieskamp, M. Harman, M. J. Harrold, P. McMinn, A. Bertolinoet al., “An orchestrated survey of methodologies for automated software test case generation, ”Journal of systems and software , vol. 86, no. 8, pp. 1978–2001, 2013

    work page 1978

  2. [2]

    A survey on model-based testing tools for test case generation,

    W. Li, F. Le Gall, and N. Spaseski, “A survey on model-based testing tools for test case generation, ” inInternational Conference on Tools and Methods for Program Analysis. Springer, 2017, pp. 77–89

    work page 2017

  3. [3]

    An empirical evaluation of using large language models for automated unit test generation,

    M. Schäfer, S. Nadi, A. Eghbali, and F. Tip, “An empirical evaluation of using large language models for automated unit test generation, ” IEEE Transactions on Software Engineering , vol. 50, no. 1, pp. 85–105, 2023

    work page 2023

  4. [4]

    Rethinking the influence of source code on test case generation,

    D. Huang, J. M. Zhang, M. Du, M. Harman, and H. Cui, “Rethinking the influence of source code on test case generation, ”arXiv preprint arXiv:2409.09464, 2024

    arXiv 2024

  5. [5]

    Testeval: Benchmarking large language models for test case generation,

    W. Wang, C. Yang, Z. Wang, Y. Huang, Z. Chu, D. Song, L. Zhang, A. R. Chen, and L. Ma, “Testeval: Benchmarking large language models for test case generation, ”arXiv preprint arXiv:2406.04531, 2024

    arXiv 2024

  6. [6]

    Testgeneval: A real world unit test generation and test completion benchmark,

    K. Jain, G. Synnaeve, and B. Rozière, “Testgeneval: A real world unit test generation and test completion benchmark, ”arXiv preprint arXiv:2410.00752, 2024

    arXiv 2024

  7. [7]

    A survey on unit testing practices and problems,

    E. Daka and G. Fraser, “A survey on unit testing practices and problems, ” in2014 IEEE 25th International Symposium on Software Reliability Engineering. IEEE, 2014, pp. 201–211

    work page 2014

  8. [8]

    DeCon: Detecting Incorrect Assertions via Postconditions Generated by a Large Language Model

    H. Yu, T. Chen, J. Huang, Z. Li, D. Ran, X. Wang, Y. Li, A. Marron, D. Harel, Y. Xieet al., “Decon: Detecting incorrect assertions via postconditions generated by a large language model, ”arXiv preprint arXiv:2501.02901, 2025

    work page Pith review arXiv 2025

Show all 47 references
  1. [9]

    Codet: Code generation with generated tests,

    B. Chen, F. Zhang, A. Nguyen, D. Zan, Z. Lin, J.-G. Lou, and W. Chen, “Codet: Code generation with generated tests, ”arXiv preprint arXiv:2207.10397, 2022

    2022 arXiv

  2. [10]

    Codecot: Tackling code syntax errors in cot reasoning for code generation,

    D. Huang, Q. Bu, Y. Qing, and H. Cui, “Codecot: Tackling code syntax errors in cot reasoning for code generation, ”arXiv preprint arXiv:2308.08784, 2023

    2023 arXiv

  3. [11]

    Agentcoder: Multi-agent-based code generation with iterative testing and optimisation,

    D. Huang, J. M. Zhang, M. Luck, Q. Bu, Y. Qing, and H. Cui, “Agentcoder: Multi-agent-based code generation with iterative testing and optimisation, ” arXiv preprint arXiv:2312.13010, 2023

    2023 arXiv

  4. [12]

    Mercury: A code efficiency benchmark for code large language models,

    M. Du, A. T. Luu, B. Ji, Q. Liu, and S.-K. Ng, “Mercury: A code efficiency benchmark for code large language models, ”arXiv preprint arXiv:2402.07844, 2024

    2024 arXiv

  5. [13]

    Reflexion: Language agents with verbal reinforcement learning,

    N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, “Reflexion: Language agents with verbal reinforcement learning, ”Advances in Neural Information Processing Systems, vol. 36, pp. 8634–8652, 2023

    2023

  6. [14]

    Kernelgpt: Enhanced kernel fuzzing via large language models,

    C. Yang, Z. Zhao, and L. Zhang, “Kernelgpt: Enhanced kernel fuzzing via large language models, ” in Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 , 2025, pp. 560–573

    2025

  7. [15]

    Universal fuzzing via large language models,

    C. S. Xia, M. Paltenghi, J. Le Tian, M. Pradel, and L. Zhang, “Universal fuzzing via large language models, ” CoRR, 2023

    2023

  8. [16]

    Large language models are edge-case generators: Crafting unusual programs for fuzzing deep learning libraries,

    Y. Deng, C. S. Xia, C. Yang, S. D. Zhang, S. Yang, and L. Zhang, “Large language models are edge-case generators: Crafting unusual programs for fuzzing deep learning libraries, ” inProceedings of the 46th IEEE/ACM international conference on software engineering , 2024, pp. 1–13

    2024

  9. [17]

    Whitefox: White-box compiler fuzzing empowered by large language models,

    C. Yang, Y. Deng, R. Lu, J. Yao, J. Liu, R. Jabbarvand, and L. Zhang, “Whitefox: White-box compiler fuzzing empowered by large language models, ” Proceedings of the ACM on Programming Languages , vol. 8, no. OOPSLA2, pp. 709–735, 2024

    2024

  10. [18]

    Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models,

    Y. Deng, C. S. Xia, H. Peng, C. Yang, and L. Zhang, “Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models, ” inProceedings of the 32nd ACM SIGSOFT international symposium on software testing and analysis , 2023, pp. 423–435

    2023

  11. [19]

    Large language models are edge-case fuzzers: Testing deep learning libraries via fuzzgpt,

    Y. Deng, C. S. Xia, C. Yang, S. D. Zhang, S. Yang, and L. Zhang, “Large language models are edge-case fuzzers: Testing deep learning libraries via fuzzgpt, ”arXiv preprint arXiv:2304.02014, 2023

    2023 arXiv

  12. [20]

    Swt-bench: Testing and validating real-world bug-fixes with code agents,

    N. Mündler, M. Müller, J. He, and M. Vechev, “Swt-bench: Testing and validating real-world bug-fixes with code agents, ” Advances in Neural Information Processing Systems, vol. 37, pp. 81 857–81 887, 2024

    2024

  13. [21]

    Testbench: Evaluating class-level test case generation capability of large language models,

    Q. Zhang, Y. Shang, C. Fang, S. Gu, J. Zhou, and Z. Chen, “Testbench: Evaluating class-level test case generation capability of large language models, ” arXiv preprint arXiv:2409.17561, 2024

    2024 arXiv

  14. [22]

    Swe-bench: Can language models resolve real-world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “Swe-bench: Can language models resolve real-world github issues?” arXiv preprint arXiv:2310.06770, 2023

    2023 arXiv

  15. [23]

    Lessleak-bench: A first investigation of data leakage in llms across 83 software engineering benchmarks,

    X. Zhou, M. Weyssow, R. Widyasari, T. Zhang, J. He, Y. Lyu, J. Chang, B. Zhang, D. Huang, and D. Lo, “Lessleak-bench: A first investigation of data leakage in llms across 83 software engineering benchmarks, ”arXiv preprint arXiv:2502.06215, 2025

    2025 arXiv

  16. [24]

    Large-scale, independent and comprehensive study of the power of llms for test case generation,

    W. C. Ouédraogo, K. Kaboré, H. Tian, Y. Song, A. Koyuncu, J. Klein, D. Lo, and T. F. Bissyandé, “Large-scale, independent and comprehensive study of the power of llms for test case generation, ”arXiv preprint arXiv:2407.00225, 2024

    2024

  17. [25]

    Starcoder 2 and the stack v2: The next generation,

    A. Lozhkov, R. Li, L. B. Allal, F. Cassano, J. Lamy-Poirier, N. Tazi, A. Tang, D. Pykhtar, J. Liu, Y. Weiet al., “Starcoder 2 and the stack v2: The next generation, ”arXiv preprint arXiv:2402.19173, 2024. Manuscript submitted to ACM 26 Dong HUANG, Jie M. Zhang, Mark Harman, Qi...

    2024 arXiv

  18. [26]

    Software engineering (ed.),

    I. Sommerville, “Software engineering (ed.), ” America: Pearson Education Inc, 2011

    2011

  19. [27]

    G. J. Myers, C. Sandler, and T. Badgett, The art of software testing . John Wiley & Sons, 2011

    2011

  20. [28]

    A survey of unit testing practices,

    P. Runeson, “A survey of unit testing practices, ” IEEE software, vol. 23, no. 4, pp. 22–29, 2006

    2006

  21. [29]

    Meszaros, xUnit test patterns: Refactoring test code

    G. Meszaros, xUnit test patterns: Refactoring test code . Pearson Education, 2007

    2007

  22. [30]

    Automated unit test generation for evolving software,

    S. Shamshiri, “Automated unit test generation for evolving software, ” in Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, 2015, pp. 1038–1041

    2015

  23. [31]

    Symbolic execution and program testing,

    J. C. King, “Symbolic execution and program testing, ” Communications of the ACM, vol. 19, no. 7, pp. 385–394, 1976

    1976

  24. [32]

    The s2e platform: Design, implementation, and applications,

    V. Chipounov, V. Kuznetsov, and G. Candea, “The s2e platform: Design, implementation, and applications, ”ACM Transactions on Computer Systems (TOCS), vol. 30, no. 1, pp. 1–49, 2012

    2012

  25. [33]

    Search-based software testing: Past, present and future,

    P. McMinn, “Search-based software testing: Past, present and future, ” in2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops. IEEE, 2011, pp. 153–163

    2011

  26. [34]

    An empirical study of the reliability of unix utilities,

    B. P. Miller, L. Fredriksen, and B. So, “An empirical study of the reliability of unix utilities, ”Communications of the ACM, vol. 33, no. 12, pp. 32–44, 1990

    1990

  27. [35]

    Large language models for software engineering: A systematic literature review,

    X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engineering: A systematic literature review, ”ACM Transactions on Software Engineering and Methodology , vol. 33, no. 8, pp. 1–79, 2024

    2024

  28. [36]

    Mutation testing advances: an analysis and survey,

    M. Papadakis, M. Kintis, J. Zhang, Y. Jia, Y. Le Traon, and M. Harman, “Mutation testing advances: an analysis and survey, ” inAdvances in computers. Elsevier, 2019, vol. 112, pp. 275–378

    2019

  29. [37]

    An analysis and survey of the development of mutation testing,

    Y. Jia and M. Harman, “An analysis and survey of the development of mutation testing, ”IEEE transactions on software engineering , vol. 37, no. 5, pp. 649–678, 2010

    2010

  30. [38]

    Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions,

    T. Y. Zhuo, M. C. Vu, J. Chim, H. Hu, W. Yu, R. Widyasari, I. N. B. Yusuf, H. Zhan, J. He, I. Paulet al., “Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions, ”arXiv preprint arXiv:2406.15877, 2024

    2024 arXiv

  31. [39]

    Using large language models to generate junit tests: An empirical study,

    M. L. Siddiq, J. C. Da Silva Santos, R. H. Tanvir, N. Ulfat, F. Al Rifat, and V. Carvalho Lopes, “Using large language models to generate junit tests: An empirical study, ” inProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering ...

    2024

  32. [40]

    Using github copilot for test generation in python: An empirical study,

    K. El Haji, C. Brandt, and A. Zaidman, “Using github copilot for test generation in python: An empirical study, ” inProceedings of the 5th ACM/IEEE International Conference on Automation of Software Test (AST 2024) , 2024, pp. 45–55

    2024

  33. [41]

    Testspark: Intellij idea’s ultimate test generation companion,

    A. Sapozhnikov, M. Olsthoorn, A. Panichella, V. Kovalenko, and P. Derakhshanfar, “Testspark: Intellij idea’s ultimate test generation companion, ” in Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings , 2024, pp. 30–34

    2024

  34. [42]

    Code generation tools (almost) for free? a study of few-shot, pre-trained language models on code,

    P. Bareiß, B. Souza, M. d’Amorim, and M. Pradel, “Code generation tools (almost) for free? a study of few-shot, pre-trained language models on code, ”arXiv preprint arXiv:2206.01335, 2022

    2022 arXiv

  35. [43]

    Retrieval-based prompt selection for code-related few-shot learning,

    N. Nashid, M. Sintaha, and A. Mesbah, “Retrieval-based prompt selection for code-related few-shot learning, ” in2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 2023, pp. 2450–2462

    2023

  36. [44]

    Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models,

    C. Lemieux, J. P. Inala, S. K. Lahiri, and S. Sen, “Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models, ” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 2023, pp. 919–931

    2023

  37. [45]

    Testart: Improving llm-based unit test via co-evolution of automated generation and repair iteration,

    S. Gu, C. Fang, Q. Zhang, F. Tian, and Z. Chen, “Testart: Improving llm-based unit test via co-evolution of automated generation and repair iteration, ” arXiv e-prints, pp. arXiv–2408, 2024

    2024

  38. [46]

    Aster: Natural and multi-language unit test generation with llms,

    R. Pan, M. Kim, R. Krishna, R. Pavuluri, and S. Sinha, “Aster: Natural and multi-language unit test generation with llms, ”arXiv preprint arXiv:2409.03093, 2025

    2025 arXiv

  39. [47]

    Effective test generation using pre-trained large language models and mutation testing,

    A. M. Dakhel, A. Nikanjam, V. Majdinasab, F. Khomh, and M. C. Desmarais, “Effective test generation using pre-trained large language models and mutation testing, ”Information and Software Technology, vol. 171, p. 107468, 2024. Manuscript submitted to ACM

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.