Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

Improving the Readability of Automatically Generated Tests using Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that renaming test identifiers and test names with large language models preserves test coverage and raises the readability of automatically generated tests to the level of developer-written ones.

desk verdict Solid empirical study of LLM-based renaming of Evosuite tests, but the semantic-preservation claim is stronger than the coverage proxy supports. read the letter →

arxiv 2412.18843 v1 pith:7GO7Y3AF submitted 2024-12-25 cs.SE

Matteo Biagiola , Gianluca Ghislotti , Paolo Tonella This is my paper · ORCID
classification cs.SE
keywords LLM-basedtestreadabilityimprovementsearch-basedgenerationsemanticpreservationcoverageEvosuiteJavaunittestshumanstudypromptengineering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Search-based test generators produce high-coverage unit tests, but their identifiers and test names are arbitrary, so developers find them hard to read and maintain. This paper proposes to keep the generator's coverage advantage and use a large language model to rename only the identifiers and the test name, leaving everything else in each test untouched. In the paper's evaluation, five of nine LLMs preserved the tests' JaCoCo coverage (the lines and branches each test executes) in every repetition, and ten professional developers judged the improved tests to be as readable as tests written by developers. If the result holds, teams can take the otherwise illegible output of automated generators, rename it with an LLM, and commit tests that developers can actually read.

What carries the argument

The mechanism is a multi-step prompt pipeline that constrains what the LLM may change. An informational prompt first gives the model a persona, the task goal, and the focal context of the class under test (class name, constructors, attributes, method signatures) and is kept in memory; then each test case is processed by a separate improvement prompt that includes the test and the source of the methods it calls, with explicit instructions to modify only identifiers and the test name. A third prompt resolves duplicate test names by renaming while keeping test content identical. The other load-bearing piece is the evaluation proxy: semantic preservation is approximated by equality of JaCoCo coverage reports between original and improved tests, and stability is measured by cosine similarity of code embeddings across repetitions.

What would settle it

Rerun the pipeline on classes where two execution paths share the same line and branch coverage, instrument the improved tests to log which path they actually execute, and compare coverage reports: if any improved test's reported JaCoCo coverage matches the original while its executed path, exception type, or assertion outcome differs, the coverage-preservation proxy is shown to overstate semantic preservation.

Watch

Extended reading notes

Core claim

The paper's claim is that readability of search-based generated tests can be improved without changing their behavior by asking an LLM to rename test names and local identifiers, and that the result is comparable in readability to human-written tests. The evidence comes from two measurements: identical JaCoCo coverage reports before and after transformation (used, as the paper states, as a proxy for semantic preservation), and readability scores from a randomized survey of ten professional developers. Five of the nine tested LLMs had a 100% success rate on the coverage-preservation check across all projects and repetitions, and the developer survey found no statistically significant readability difference between LLM-improved and developer-written tests, regardless of which of those five models produced the improvement. The transformation is deliberately conservative: only identifiers and test names may change, method calls and control flow must stay fixed.

Load-bearing premise

The result stands on the assumption that identical line, branch, method, and statement coverage between the original and improved test means the test's behavior has not changed.

Editorial extensions

If this is right

  • Automatic test suites can be made human-comprehensible without losing their coverage, so the main practical objection to using search-based generated tests is weakened.
  • The multi-step prompt pipeline can be applied as a post-processing step after any coverage-based generator, because it operates on individual test cases rather than regenerating the suite.
  • The five reliable LLMs are interchangeable for this task in terms of resulting readability, letting users choose a cheaper or faster model without sacrificing readability.
  • Because the improved tests remain highly similar across repetitions, the results are stable enough to be committed into a codebase and maintained over time.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct ablation would reveal how much of the readability gain comes from test names versus variable names, and whether the renaming-only constraint is what keeps coverage intact.
  • The same multi-step prompt likely transfers to other languages and coverage-guided generators, since it only asks for identifier and test-name changes; a rerun on non-Java generator outputs would test whether the 100% preservation rate is Java-specific.
  • A larger human study with more developers and classes would sharpen the comparison with developer-written tests, because the paper reports that its ten-developer sample did not reach conventional statistical power.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes using large language models to rename identifiers and test names in EvoSuite-generated Java test suites, with the goal of improving readability while preserving test semantics. The approach uses a multi-step prompt that first provides the LLM with focal context about the class under test and then asks it to rewrite only identifiers and test names of individual test cases. The evaluation covers nine LLMs across five Java projects. RQ1 measures semantic preservation via equality of JaCoCo coverage reports; RQ2 measures stability via cosine similarity of code embeddings across repetitions; RQ3 reports a human study in which ten professional developers scored the readability of LLM-improved tests versus developer-written tests. The paper reports that five of the nine LLMs achieved 100% coverage-preservation, and that LLM-improved tests received readability scores statistically indistinguishable from developer-written tests.

Significance. If the results hold, the approach addresses a real practical problem: EvoSuite-generated tests are effective but often unreadable, and the proposed pipeline offers a way to make them more human-friendly without discarding the coverage achieved by search-based generation. The paper's strengths include a replication package, evaluation of nine LLMs from multiple providers, a systematic class-selection procedure, and a human study with professional developers recruited via a qualification task. The central semantic-preservation claim, however, is supported only by a coarse coverage proxy, and the human-study design lacks the unimproved EvoSuite baseline needed to demonstrate an actual readability improvement. The repetition-count inconsistency and the low statistical power of the equivalence claim further weaken the conclusions as currently stated. The work is a useful empirical contribution, but the load-bearing claims need to be either strengthened with additional evidence or appropriately qualified.

major comments (4)
  1. [V-C1, V-G] The claim that the transformations are 'semantically-preserving' (RQ1, Table II) rests entirely on equality of JaCoCo coverage reports, as stated in V-C1: 'we approximate semantic-preservation with coverage-preservation.' JaCoCo records lines, branches, methods, and statements executed; it does not capture the assertions made by a test. A transformation that changes assertSame(object1, object0) to assertSame(object0, object1), replaces a literal inside an assertion, or swaps an expected exception type can preserve all JaCoCo metrics while changing the test's failure behavior. Because RQ1's 100% success rate is used to screen which LLMs advance to the human study and to pair tests with developer-written counterparts in V-E1, the coverage proxy is load-bearing. The acknowledgment in V-G does not mitigate the mismatch between the safety claim ('never change the semantics of a test case') and the metric actually reported. I recommend adding an assertion-level comparison or a mutation-based oracle check to the RQ1 procedure, or repositioning RQ1 explicitly as a study of coverage preservation and limiting the semantic-preservation claim accordingly.
  2. [V-C2, V-F1, Table II] There is an internal inconsistency in the number of repetitions for RQ1. Section V-C2 states that 'we executed our readability improvement approach ten times' for each LLM, while Table II's caption and Section V-F1 both report results 'across five repetitions.' The denominator of the success rate directly depends on this number, and the stability analysis in RQ2 also uses five repetitions. Please clarify which number is correct: if ten repetitions were actually run, report results for all ten or explain why only five are analyzed; if only five were run, correct the procedure description in V-C2.
  3. [V-E1, V-F3] The human study compares LLM-improved tests only against developer-written tests; unimproved EvoSuite tests are not scored by the developers. Consequently, the study cannot support the conclusion that the approach 'improves' readability relative to the original automatically generated tests; it supports only the weaker claim that improved tests are comparable to developer-written tests. Since the paper's stated objective is to improve readability (Section I), the absence of the original EvoSuite baseline is a significant gap. Adding the unimproved tests to the survey, or explicitly rephrasing the RQ3 conclusion to avoid implying an improvement over EvoSuite, is necessary to make the result interpretable.
  4. [V-F3] The paper acknowledges that the statistical power β did not reach the conventional threshold of 0.8, so the Wilcoxon test's failure to find a significant difference is not evidence that LLM-improved tests are equivalent in readability to developer-written tests. The abstract and conclusion nevertheless state that 'LLM-improved tests are as readable as developer-written tests.' Please temper the wording to 'no statistically significant difference was observed' or provide an equivalence test, confidence intervals, or a larger sample that can support the stronger claim.
minor comments (6)
  1. [Abstract, Section I] The abstract and introduction state that the approach is 'overall semantically-preserving' across the nine evaluated LLMs, but Table II shows that only five of the nine models achieved 100% success, with the others ranging from 0% to 50%. Please qualify this claim to refer specifically to the five LLMs that passed the RQ1 screening.
  2. [Table I, V-C2] The total number of test cases used in RQ1 is not reported anywhere; success rates are computed over some set of tests, but the reader cannot see how many tests were transformed per class or project. Adding test counts (e.g., as a column in Table I or in the replication package) would improve reproducibility and make the reported percentages interpretable.
  3. [V-D1, V-D2, Figure 2] The stability analysis interprets cosine similarities of 0.87–0.94 as 'high stability' without any baseline for comparison. Reporting the similarity between repeated embeddings of the same unchanged test, or between the original EvoSuite test and its improved versions, would help calibrate these numbers.
  4. [Listing 3, V-F2] Typos: 'genererated' in the Listing 3 caption and 'similiarity' in Section V-F2 should be corrected.
  5. [References] Reference [53] lists the author as 'G. Ghisolotti', but the paper's author list uses 'Gianluca Ghislotti'; please correct the spelling.
  6. [V-C1] The sentence 'These reports contain indicate how many statements...' has a grammatical error ('contain indicate'); it should read 'These reports indicate how many statements...' or 'These reports contain indicators of how many statements...'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the semantic-preservation claim is an empirical measurement with an acknowledged coverage proxy, not a result forced by definition or self-citation.

full rationale

The paper's central claims are evaluated empirically against external benchmarks. RQ1 measures whether LLM-renamed tests preserve JaCoCo coverage relative to the original Evosuite tests; this is an observed output comparison, not a quantity fitted from the data and then predicted back. The paper explicitly says 'we approximate semantic-preservation with coverage-preservation' (Section V-C1), which is an acknowledged construct-validity limitation, not a circular reduction: the success rate is not defined to be 100% by construction, and Table II reports varied outcomes across the nine LLMs (0% to 100%). RQ2 uses cosine similarity of embeddings of repeated LLM outputs; this is a measurement choice for stability, not a self-referential argument. RQ3 compares human readability ratings of LLM-improved tests against developer-written tests; the ratings come from ten external professional developers and are not derived from the paper's assumptions. The authors' self-citations (e.g., [2], [6], [10], [26], [54]) appear only as background or methodology support and are not load-bearing for the novel claim. No equation is defined in terms of its own output, no fitted parameter is renamed a prediction, and no uniqueness or ansatz is imported from the authors' prior work. The coverage-proxy issue is a real threat to construct validity, but it is explicitly acknowledged in Section V-G and does not make the derivation circular.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the coverage proxy for semantics, the embedding model for stability, the readability gold standard of manual tests, and the prompt's ability to constrain LLM behavior. These are domain assumptions rather than fitted parameters or invented entities; the only hand-chosen numbers are experimental design choices.

free parameters (3)
  • Class selection thresholds = LOC >= 50; 3 <= AML <= 20; 0 <= II <= 10
    Hand-chosen filters in Section V-A define which classes enter the study, so the external validity of the coverage and human evaluation depends on them.
  • LLM temperature = 0
    Set to 0 to reduce non-determinism in Section V-C2; affects how representative single-run outputs are of the model's behavior.
  • Duplicate-name fallback limit = 3 queries
    Hand-chosen limit in Section IV-A after which original test names are restored; affects whether duplicate names remain in the final test suite.
assumptions (4)
  • domain assumption JaCoCo coverage equality is a valid proxy for test semantic preservation
    Invoked in Section V-C1 and defended as a widely used proxy in Section V-G; if false, the RQ1 success rates do not establish semantic preservation.
  • domain assumption OpenAI text embeddings capture meaningful code similarity
    Used in Section V-D to measure stability via cosine similarity; if embeddings are insensitive to naming changes, stability results could be misleading.
  • domain assumption Developer-written tests in the selected projects are a valid readability gold standard
    The human study in Section V-E compares LLM-improved tests against manual tests from five open-source projects, assuming those manual tests are readable and of high quality.
  • domain assumption The multi-step prompt design keeps the LLM focused on renaming
    The entire approach relies on the LLM following the constraint to modify only identifiers and test names; Section IV-A describes the prompts, but adherence is an empirical property of each model, not a guarantee.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving the Readability of Automatically Generated Tests using Large Language Models." pith.science (2026). https://pith.science/paper/7GO7Y3AF

@misc{pith2026241218843,
  author       = {Pith},
  title        = {Pith review of: Improving the Readability of Automatically Generated Tests using Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7GO7Y3AF}},
  note         = {Machine review of arXiv:2412.18843}
}
read the original abstract

Search-based test generators are effective at producing unit tests with high coverage. However, such automatically generated tests have no meaningful test and variable names, making them hard to understand and interpret by developers. On the other hand, large language models (LLMs) can generate highly readable test cases, but they are not able to match the effectiveness of search-based generators, in terms of achieved code coverage. In this paper, we propose to combine the effectiveness of search-based generators with the readability of LLM generated tests. Our approach focuses on improving test and variable names produced by search-based tools, while keeping their semantics (i.e., their coverage) unchanged. Our evaluation on nine industrial and open source LLMs show that our readability improvement transformations are overall semantically-preserving and stable across multiple repetitions. Moreover, a human study with ten professional developers, show that our LLM-improved tests are as readable as developer-written tests, regardless of the LLM employed.

Figures

Figures reproduced from arXiv: 2412.18843 by the authors.

Figure 1
Figure 1. Overview of our approach to readability improvement. A test generator produces the test suite given the class under [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Results for RQ2. The figure shows the distributions of cosine similarity values for each LLM, for each project, considering all the test cases and classes. The x-axis shows the different projects for each LLM, while the y-axis shows the cosine similarity, which ranges from −1 to 1 [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Distributions of readability scores for each LLM across [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Rethinking Cognitive Complexity for Unit Tests: Toward a Readability-Aware Metric Grounded in Developer Perception

    cs.SE 2025-06 reject novelty 6.0 of 10

    CCTR is a test-aware complexity metric that adds assertion, mocking, and annotation counts to the traditional nesting score, but its claim to reflect developer perception is not yet validated.

  2. Large Language Models for Unit Testing: A Systematic Literature Review

    cs.SE 2025-06 conditional novelty 5.0 of 10

    The paper presents the first systematic literature review of large language model based unit testing, covering 105 papers up to March 2025.

Reference graph

Works this paper leans on

59 extracted references · 17 canonical work pages · cited by 2 Pith papers

  1. [1]

    Search based software engineering,

    M. Harman, “Search based software engineering,” in Computational Science - ICCS 2006, 6th International Conference, Reading, UK, May 28-31, 2006, Proceedings, Part IV , ser. Lecture Notes in Computer Science, V . N. Alexandrov, G. D. van Albada, P. M. A. Sloot, and J. J. Dongarra, Eds., vol. 3994. Springer, 2006, pp. 740–747. [Online]. Available: https://...

    work page doi:10.1007/11758549 2006

  2. [2]

    Evolutionary testing of classes,

    P. Tonella, “Evolutionary testing of classes,” in Proceedings of the ACM/SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2004, Boston, Massachusetts, USA, July 11-14, 2004 , G. S. Avrunin and G. Rothermel, Eds. ACM, 2004, pp. 119–128. [Online]. Available: https://doi.org/10.1145/1007512.1007528

    arXiv 2004

  3. [3]

    Evosuite: automatic test suite generation for object-oriented software,

    G. Fraser and A. Arcuri, “Evosuite: automatic test suite generation for object-oriented software,” in Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering , ser. ESEC/FSE ’11. New York, NY , USA: Association for Computing Machinery, 2011, p. 416–419. [Online]. Available: https://doi.org/10.1...

    arXiv 2011

  4. [4]

    Whole test suite generation,

    ——, “Whole test suite generation,” IEEE Trans. Software Eng. , vol. 39, no. 2, pp. 276–291, 2013. [Online]. Available: https: //doi.org/10.1109/TSE.2012.14

    doi:10.1109/tse.2012.14 2013

  5. [6]

    Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets,

    A. Panichella, F. M. Kifetew, and P. Tonella, “Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets,” IEEE Trans. Software Eng. , vol. 44, no. 2, pp. 122–158, 2018. [Online]. Available: https: //doi.org/10.1109/TSE.2017.2663435

    arXiv 2018

  6. [7]

    Pynguin: Automated unit test generation for python,

    S. Lukasczyk and G. Fraser, “Pynguin: Automated unit test generation for python,” in 44th IEEE/ACM International Conference on Software Engineering: Companion Proceedings, ICSE Companion 2022, Pittsburgh, PA, USA, May 22-24, 2022 . ACM/IEEE, 2022, pp. 168–172. [Online]. Available: https://doi.org/10.1145/3510454.3516829

    arXiv 2022

  7. [8]

    Syntest-javascript: Automated unit-level test case generation for javascript,

    M. Olsthoorn, D. Stallenberg, and A. Panichella, “Syntest-javascript: Automated unit-level test case generation for javascript,” in Proceedings of the 17th ACM/IEEE International Workshop on Search-Based and Fuzz Testing , ser. SBFT ’24. New York, NY , USA: Association for Computing Machinery, 2024, p. 21–24. [Online]. Available: https://doi.org/10.1145/3...

    work page arXiv 2024

  8. [9]

    Evomaster: Evolutionary multi-context automated system test generation,

    A. Arcuri, “Evomaster: Evolutionary multi-context automated system test generation,” in 11th IEEE International Conference on Software Testing, Verification and Validation, ICST 2018, V¨aster˚as, Sweden, April 9-13, 2018 . IEEE Computer Society, 2018, pp. 394–397. [Online]. Available: https://doi.org/10.1109/ICST.2018.00046

    arXiv 2018

Show all 59 references
  1. [11]

    Sapienz: multi-objective automated testing for android applications,

    K. Mao, M. Harman, and Y . Jia, “Sapienz: multi-objective automated testing for android applications,” in Proceedings of the 25th International Symposium on Software Testing and Analysis, ISSTA 2016, Saarbr ¨ucken, Germany, July 18-20, 2016 , A. Zeller and A. Roychoudhury, Eds...

    2016

  2. [12]

    Modeling readability to improve unit tests,

    E. Daka, J. Campos, G. Fraser, J. Dorn, and W. Weimer, “Modeling readability to improve unit tests,” in Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, Bergamo, Italy, August 30 - September 4, 2015 , E. D. Nitto, M. Harman, and...

    2015

  3. [13]

    Generating unit tests with descriptive names or: would you name your children thing1 and thing2?

    E. Daka, J. M. Rojas, and G. Fraser, “Generating unit tests with descriptive names or: would you name your children thing1 and thing2?” in Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis, Santa Barbara, CA, USA, July 10 - 14, 2017 ,...

    2017

  4. [14]

    Unit test case generation with transformers and focal context,

    M. Tufano, D. Drain, A. Svyatkovskiy, S. K. Deng, and N. Sundaresan, “Unit test case generation with transformers and focal context,” arXiv preprint arXiv:2009.05617, 2020

    2009 arXiv

  5. [15]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

    2017

  6. [16]

    An empirical evaluation of using large language models for automated unit test generation,

    M. Sch ¨afer, S. Nadi, A. Eghbali, and F. Tip, “An empirical evaluation of using large language models for automated unit test generation,” IEEE Trans. Software Eng. , vol. 50, no. 1, pp. 85–105, 2024. [Online]. Available: https://doi.org/10.1109/TSE.2023.3334955

    2024

  7. [18]

    Evaluating and improving chatgpt for unit test generation,

    Z. Yuan, M. Liu, S. Ding, K. Wang, Y . Chen, X. Peng, and Y . Lou, “Evaluating and improving chatgpt for unit test generation,” Proc. ACM Softw. Eng., vol. 1, no. FSE, pp. 1703–1726, 2024. [Online]. Available: https://doi.org/10.1145/3660783

    2024 doi

  8. [19]

    Testspark: Intellij idea’s ultimate test generation companion,

    A. Sapozhnikov, M. Olsthoorn, A. Panichella, V . Kovalenko, and P. Derakhshanfar, “Testspark: Intellij idea’s ultimate test generation companion,” in Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings, ICSE Companion 2...

    2024

  9. [20]

    Chatgpt vs SBST: A comparative assessment of unit test suite generation,

    Y . Tang, Z. Liu, Z. Zhou, and X. Luo, “Chatgpt vs SBST: A comparative assessment of unit test suite generation,” IEEE Trans. Software Eng., vol. 50, no. 6, pp. 1340–1359, 2024. [Online]. Available: https://doi.org/10.1109/TSE.2024.3382365

    2024

  10. [21]

    An empirical study of using large language models for unit test generation,

    M. L. Siddiqa, J. C. Santos, R. H. Tanvirb, N. Ulfatc, F. Al Rifatd, and V . C. Lopes, “An empirical study of using large language models for unit test generation,” arXiv preprint arXiv:2305.00418 , 2023

    2023 arXiv

  11. [22]

    Chatunitest: a chatgpt- based automated unit test generation tool,

    Z. Xie, Y . Chen, C. Zhi, S. Deng, and J. Yin, “Chatunitest: a chatgpt- based automated unit test generation tool,” CoRR, vol. abs/2305.04764,

    arXiv

  12. [23]

    Generating effective test suites by combining coverage criteria,

    G. Gay, “Generating effective test suites by combining coverage criteria,” in Search Based Software Engineering - 9th International Symposium, SSBSE 2017, Paderborn, Germany, September 9-11, 2017, Proceedings, ser. Lecture Notes in Computer Science, T. Menzies and J. Petke, Ed...

    2017 doi

  13. [24]

    Automated unit test improvement using large language models at meta,

    N. Alshahwan, J. Chheda, A. Finogenova, B. Gokkaya, M. Harman, I. Harper, A. Marginean, S. Sengupta, and E. Wang, “Automated unit test improvement using large language models at meta,” in Companion Proceedings of the 32nd ACM International Conference on the Foundations of Soft...

    2024

  14. [25]

    Lost in the middle: How language models use long contexts,

    N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language models use long contexts,” Trans. Assoc. Comput. Linguistics , vol. 12, pp. 157–173,

  15. [26]

    A large scale empirical comparison of state-of-the-art search-based test case generators,

    A. Panichella, F. M. Kifetew, and P. Tonella, “A large scale empirical comparison of state-of-the-art search-based test case generators,” Information and Software Technology , vol. 104, pp. 236–256, 2018. [Online]. Available: https://www.sciencedirect.com/science/article/pii/ ...

    2018

  16. [27]

    Search-based software test data generation: a survey,

    P. McMinn, “Search-based software test data generation: a survey,” Softw. Test. Verification Reliab. , vol. 14, no. 2, pp. 105–156, 2004. [Online]. Available: https://doi.org/10.1002/stvr.294

    2004 doi

  17. [28]

    A systematic review of the application and empirical investigation of search-based test case generation,

    S. Ali, L. C. Briand, H. Hemmati, and R. K. Panesar-Walawege, “A systematic review of the application and empirical investigation of search-based test case generation,” IEEE Trans. Software Eng. , vol. 36, no. 6, pp. 742–762, 2010. [Online]. Available: https: //doi.org/10.1109...

    2010 doi

  18. [29]

    Randoop: feedback-directed random testing for java,

    C. Pacheco and M. D. Ernst, “Randoop: feedback-directed random testing for java,” in Companion to the 22nd Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2007, October 21-25, 2007, Montreal, Quebec, Canada , R. P. Gab...

    2007

  19. [30]

    SBFT tool competition 2023 - java test case generation track,

    G. Jahangirova and V . Terragni, “SBFT tool competition 2023 - java test case generation track,” in IEEE/ACM International Workshop on Search-Based and Fuzz Testing, SBFT@ICSE 2023, Melbourne, Australia, May 14, 2023 . IEEE, 2023, pp. 61–64. [Online]. Available: https://doi.or...

    2023

  20. [31]

    Restful API automated test case generation with evomaster,

    A. Arcuri, “Restful API automated test case generation with evomaster,” ACM Trans. Softw. Eng. Methodol. , vol. 28, no. 1, pp. 3:1–3:37, 2019. [Online]. Available: https://doi.org/10.1145/3293455

    2019 doi

  21. [32]

    An empirical study of automated unit test generation for python,

    S. Lukasczyk, F. Kroiß, and G. Fraser, “An empirical study of automated unit test generation for python,” Empir. Softw. Eng., vol. 28, no. 2, p. 36,

  22. [33]

    Software testing with large language models: Survey, landscape, and vision,

    J. Wang, Y . Huang, C. Chen, Z. Liu, S. Wang, and Q. Wang, “Software testing with large language models: Survey, landscape, and vision,” IEEE Trans. Software Eng., vol. 50, no. 4, pp. 911–936, 2024. [Online]. Available: https://doi.org/10.1109/TSE.2024.3368208

    2024

  23. [34]

    A3test: Assertion- augmented automated test case generation,

    S. Alagarsamy, C. Tantithamthavorn, and A. Aleti, “A3test: Assertion- augmented automated test case generation,” Inf. Softw. Technol. , vol. 176, p. 107565, 2024. [Online]. Available: https://doi.org/10.1016/j. infsof.2024.107565

    2024

  24. [35]

    Available: https://doi.org/10.1007/s10664-022-10248-w

    [Online]. Available: https://doi.org/10.1007/s10664-022-10248-w

    doi

  25. [36]

    A complexity measure,

    T. McCabe, “A complexity measure,” IEEE Transactions on Software Engineering, vol. SE-2, no. 4, pp. 308–320, 1976

    1976

  26. [37]

    Cognitive complexity: an overview and evaluation,

    G. A. Campbell, “Cognitive complexity: an overview and evaluation,” in Proceedings of the 2018 International Conference on Technical Debt, TechDebt@ICSE 2018, Gothenburg, Sweden, May 27-28, 2018 , R. L. Nord, F. Buschmann, and P. Kruchten, Eds. ACM, 2018, pp. 57–58. [Online]. ...

    2018

  27. [38]

    Automatic generation of test cases based on bug reports: a feasibility study with large language models,

    L. Plein, W. C. Ou ´edraogo, J. Klein, and T. F. Bissyand ´e, “Automatic generation of test cases based on bug reports: a feasibility study with large language models,” in Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceed...

    2024

  28. [39]

    Effective test generation using pre-trained large language models and mutation testing,

    A. M. Dakhel, A. Nikanjam, V . Majdinasab, F. Khomh, and M. C. Desmarais, “Effective test generation using pre-trained large language models and mutation testing,” Inf. Softw. Technol., vol. 171, p. 107468,

  29. [40]

    Learning a metric for code readability,

    R. P. Buse and W. R. Weimer, “Learning a metric for code readability,” IEEE Transactions on software engineering, vol. 36, no. 4, pp. 546–558, 2009

    2009

  30. [41]

    Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models,

    C. Lemieux, J. P. Inala, S. K. Lahiri, and S. Sen, “Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models,” in 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023 . IEEE, 2023...

    2023

  31. [42]

    The effect of modularization and comments on program comprehension,

    S. N. Woodfield, H. E. Dunsmore, and V . Y . Shen, “The effect of modularization and comments on program comprehension,” in Proceedings of the 5th International Conference on Software Engineering, San Diego, California, USA, March 9-12, 1981 , S. Jeffrey and L. G. Stucki, Eds....

    1981

  32. [43]

    Available: https://doi.org/10.1016/j.infsof.2024.107468

    [Online]. Available: https://doi.org/10.1016/j.infsof.2024.107468

    2024

  33. [44]

    Code readability testing, an empirical study,

    T. Sedano, “Code readability testing, an empirical study,” in 29th IEEE International Conference on Software Engineering Education and Training, CSEET 2016, Dallas, TX, USA, April 5-6, 2016 . IEEE, 2016, pp. 111–117. [Online]. Available: https://doi.org/10.1109/CSEET. 2016.36

    2016 doi

  34. [45]

    An empirical validation of cognitive complexity as a measure of source code understandability,

    M. M. Bar ´on, M. Wyrich, and S. Wagner, “An empirical validation of cognitive complexity as a measure of source code understandability,” in ESEM ’20: ACM / IEEE International Symposium on Empirical Software Engineering and Measurement, Bari, Italy, October 5- 7, 2020 , M. T. ...

    2020

  35. [46]

    Deeptc-enhancer: Improving the readability of automatically generated tests,

    D. Roy, Z. Zhang, M. Ma, V . Arnaoudova, A. Panichella, S. Panichella, D. Gonzalez, and M. Mirakhorli, “Deeptc-enhancer: Improving the readability of automatically generated tests,” in 35th IEEE/ACM International Conference on Automated Software Engineering, ASE 2020, Melbourn...

    2020

  36. [47]

    Shorter identifier names take longer to comprehend,

    J. C. Hofmeister, J. Siegmund, and D. V . Holt, “Shorter identifier names take longer to comprehend,” Empir. Softw. Eng. , vol. 24, no. 1, pp. 417–443, 2019. [Online]. Available: https://doi.org/10.1007/ s10664-018-9621-x

    2019

  37. [48]

    Improving the readability of generated tests using GPT-4 and chatgpt code interpreter,

    G. Gay, “Improving the readability of generated tests using GPT-4 and chatgpt code interpreter,” in Search-Based Software Engineering - 15th International Symposium, SSBSE 2023, San Francisco, CA, USA, December 8, 2023, Proceedings , ser. Lecture Notes in Computer Science, P. ...

    2023 doi

  38. [49]

    The impact of test case summaries on bug fixing performance: an empirical investigation,

    S. Panichella, A. Panichella, M. Beller, A. Zaidman, and H. C. Gall, “The impact of test case summaries on bug fixing performance: an empirical investigation,” in Proceedings of the 38th International Conference on Software Engineering, ICSE 2016, Austin, TX, USA, May 14-22, 2...

    2016

  39. [50]

    A systematic survey of prompt engineering in large language models: Techniques and applications,

    P. Sahoo, A. K. Singh, S. Saha, V . Jain, S. Mondal, and A. Chadha, “A systematic survey of prompt engineering in large language models: Techniques and applications,” CoRR, vol. abs/2402.07927, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2402.07927

    2024 doi arXiv

  40. [51]

    Interevo-tr: Interactive evolutionary test generation with readability assessment,

    P. Delgado-P ´erez, A. Ram ´ırez, K. J. Valle-G ´omez, I. Medina- Bulo, and J. R. Romero, “Interevo-tr: Interactive evolutionary test generation with readability assessment,” IEEE Trans. Software Eng., vol. 49, no. 4, pp. 2580–2596, 2023. [Online]. Available: https://doi.org/1...

    2023

  41. [52]

    Large language models are zero-shot reasoners,

    T. Kojima, S. S. Gu, M. Reid, Y . Matsuo, and Y . Iwasawa, “Large language models are zero-shot reasoners,” in Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28...

    2022

  42. [53]

    LLM is like a box of chocolates: the non-determinism of chatgpt in code generation,

    S. Ouyang, J. M. Zhang, M. Harman, and M. Wang, “LLM is like a box of chocolates: the non-determinism of chatgpt in code generation,” CoRR, vol. abs/2308.02828, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2308.02828

    2023 doi arXiv

  43. [54]

    An empirical validation of oracle improvement,

    G. Jahangirova, D. Clark, M. Harman, and P. Tonella, “An empirical validation of oracle improvement,” IEEE Trans. Software Eng., vol. 47, no. 8, pp. 1708–1728, 2021. [Online]. Available: https://doi.org/10.1109/TSE.2019.2934409

    2021

  44. [55]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V . Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” in Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing ...

    2022

  45. [56]

    Individual comparisons by ranking methods,

    F. Wilcoxon, “Individual comparisons by ranking methods,” Biometrics Bulletin, vol. 1, no. 6, pp. 80–83, 1945. [Online]. Available: http://www.jstor.org/stable/3001968

    1945

  46. [57]

    Retrieval-augmented generation for knowledge-intensive NLP tasks,

    P. S. H. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W. Yih, T. Rockt ¨aschel, S. Riedel, and D. Kiela, “Retrieval-augmented generation for knowledge-intensive NLP tasks,” in Advances in Neural Information Processing Systems 33: Ann...

    2020

  47. [58]

    Ghisolotti, 2025

    G. Ghisolotti, 2025. [Online]. Available: https://github.com/ GhislottiGianluca/readability-llms

    2025

  48. [60]

    Gamifying a software testing course with code defenders,

    G. Fraser, A. Gambi, M. Kreis, and J. M. Rojas, “Gamifying a software testing course with code defenders,” in Proceedings of the 50th ACM Technical Symposium on Computer Science Education, SIGCSE 2019, Minneapolis, MN, USA, February 27 - March 02, 2019, E. K. Hawthorne, M. A. ...

    2019

  49. [2022]

    Available: http://papers.nips.cc/paper files/paper/2022/ hash/8bb0d291acd4acf06ef112099c16f326-Abstract-Conference.html

    [Online]. Available: http://papers.nips.cc/paper files/paper/2022/ hash/8bb0d291acd4acf06ef112099c16f326-Abstract-Conference.html

    2022

  50. [2023]

    Available: https://doi.org/10.48550/arXiv.2305.04764

    [Online]. Available: https://doi.org/10.48550/arXiv.2305.04764

    doi arXiv

  51. [2024]

    Available: https://doi.org/10.1162/tacl a 00638

    [Online]. Available: https://doi.org/10.1162/tacl a 00638

    doi

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.