{"id":"1ced2a34-620b-4d78-b6e5-fa66fce86578","arxiv_id":"2412.18843","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Using a multi-step prompt that restricts LLM edits to identifiers and test names, five of nine tested LLMs improved the readability of Evosuite-generated Java tests while fully preserving their code coverage.","lead":"This paper tests whether large language models can rename the generic identifiers and test names in automatically generated Java tests without changing what the tests do, making them easier for developers to read. Across nine LLMs, five preserved test coverage perfectly, and a small human study found the renamed tests as readable as tests written by developers.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Semantic-preservation claim rests on JaCoCo coverage equality, but coverage-equal tests can differ in assertion behavior; RQ1's 100% success may overstate true semantic preservation.","rationale":"The paper has genuine strengths: a well-designed multi-step prompt that restricts edits to identifiers and test names, an evaluation across nine LLMs, a human study with professional developers, and a public replication package. The reader's conditional verdict is appropriate. The most load-bearing concern is the use of JaCoCo coverage equality as a proxy for semantic preservation. This proxy is explicit, acknowledged in the threats to validity, and used both to filter LLMs and to match tests for the human study, so it is not a peripheral limitation. A concrete mutation-analysis check would determine whether the proxy hides meaningful oracle changes. The repetition-count inconsistency (ten in the procedure, five in results and Table II) is a reproducibility flaw but does not affect the central argument as directly. The missing original-Evosuite baseline in the human study weakens the 'improvement' wording but not the safety claim of semantic preservation. If the mutation check reveals divergences, the claim should be softened to 'coverage-preserving' and the affected RQ1 numbers re-reported; if it does not, the existing conditional verdict stands without further change.","tokens_in":19576,"tokens_out":3833,"duration_ms":39561,"concrete_test":"Run mutation testing on the ten study classes: for each class, generate a set of mutants (e.g., with PIT or Major). For each of the five LLMs that achieved 100% coverage-preservation, run both the original Evosuite test suite and the improved test suite against each mutant and record the pass/fail outcomes. Compare the two mutation-kill vectors per test suite. If any improved suite yields a different kill vector from its original counterpart despite identical JaCoCo coverage, then coverage equality is insufficient evidence of semantic preservation; quantify the frequency and report concrete examples of divergences.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section V-C1 defines success as 'coverage-preserving' and states 'we approximate semantic-preservation with coverage-preservation.' JaCoCo reports lines, branches, methods, and statements covered; it does not record which assertions are made or whether a test would fail on a mutated program. Because the LLM is instructed to rename identifiers and test names but may also alter constants, assertion arguments, or exception expectations, a transformed test can execute the same lines and branches as the original while asserting different behavior. For example, changing 'assertSame(object1, object0)' to 'assertSame(object0, object1)', or replacing a literal in an assertion with a different constant, preserves coverage but changes the oracle. Such changes would be counted as successful in RQ1 (Table II) and would also pass the coverage-based matching used to pair tests in the human study (Section V-E1). The paper acknowledges this proxy in Section V-G, but the entire RQ1 result, including the screening of which five LLMs advance to the human study, depends on it. This is not an external disagreement with consensus; it is an internal gap between the claim that LLMs 'never change the semantics of a test case' and the metric actually used to measure semantics. The missing original-Evosuite baseline in the human study and the inconsistency between the stated ten repetitions and the reported five are real secondary issues, but the coverage proxy is the foundation of the central safety guarantee.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes using large language models to rename identifiers and test names in EvoSuite-generated Java test suites, with the goal of improving readability while preserving test semantics. The approach uses a multi-step prompt that first provides the LLM with focal context about the class under test and then asks it to rewrite only identifiers and test names of individual test cases. The evaluation covers nine LLMs across five Java projects. RQ1 measures semantic preservation via equality of JaCoCo coverage reports; RQ2 measures stability via cosine similarity of code embeddings across repetitions; RQ3 reports a human study in which ten professional developers scored the readability of LLM-improved tests versus developer-written tests. The paper reports that five of the nine LLMs achieved 100% coverage-preservation, and that LLM-improved tests received readability scores statistically indistinguishable from developer-written tests.","tokens_in":19797,"tokens_out":4892,"duration_ms":44521,"significance":"If the results hold, the approach addresses a real practical problem: EvoSuite-generated tests are effective but often unreadable, and the proposed pipeline offers a way to make them more human-friendly without discarding the coverage achieved by search-based generation. The paper's strengths include a replication package, evaluation of nine LLMs from multiple providers, a systematic class-selection procedure, and a human study with professional developers recruited via a qualification task. The central semantic-preservation claim, however, is supported only by a coarse coverage proxy, and the human-study design lacks the unimproved EvoSuite baseline needed to demonstrate an actual readability improvement. The repetition-count inconsistency and the low statistical power of the equivalence claim further weaken the conclusions as currently stated. The work is a useful empirical contribution, but the load-bearing claims need to be either strengthened with additional evidence or appropriately qualified.","major_comments":[{"comment":"The claim that the transformations are 'semantically-preserving' (RQ1, Table II) rests entirely on equality of JaCoCo coverage reports, as stated in V-C1: 'we approximate semantic-preservation with coverage-preservation.' JaCoCo records lines, branches, methods, and statements executed; it does not capture the assertions made by a test. A transformation that changes assertSame(object1, object0) to assertSame(object0, object1), replaces a literal inside an assertion, or swaps an expected exception type can preserve all JaCoCo metrics while changing the test's failure behavior. Because RQ1's 100% success rate is used to screen which LLMs advance to the human study and to pair tests with developer-written counterparts in V-E1, the coverage proxy is load-bearing. The acknowledgment in V-G does not mitigate the mismatch between the safety claim ('never change the semantics of a test case') and the metric actually reported. I recommend adding an assertion-level comparison or a mutation-based oracle check to the RQ1 procedure, or repositioning RQ1 explicitly as a study of coverage preservation and limiting the semantic-preservation claim accordingly.","section":"V-C1, V-G"},{"comment":"There is an internal inconsistency in the number of repetitions for RQ1. Section V-C2 states that 'we executed our readability improvement approach ten times' for each LLM, while Table II's caption and Section V-F1 both report results 'across five repetitions.' The denominator of the success rate directly depends on this number, and the stability analysis in RQ2 also uses five repetitions. Please clarify which number is correct: if ten repetitions were actually run, report results for all ten or explain why only five are analyzed; if only five were run, correct the procedure description in V-C2.","section":"V-C2, V-F1, Table II"},{"comment":"The human study compares LLM-improved tests only against developer-written tests; unimproved EvoSuite tests are not scored by the developers. Consequently, the study cannot support the conclusion that the approach 'improves' readability relative to the original automatically generated tests; it supports only the weaker claim that improved tests are comparable to developer-written tests. Since the paper's stated objective is to improve readability (Section I), the absence of the original EvoSuite baseline is a significant gap. Adding the unimproved tests to the survey, or explicitly rephrasing the RQ3 conclusion to avoid implying an improvement over EvoSuite, is necessary to make the result interpretable.","section":"V-E1, V-F3"},{"comment":"The paper acknowledges that the statistical power β did not reach the conventional threshold of 0.8, so the Wilcoxon test's failure to find a significant difference is not evidence that LLM-improved tests are equivalent in readability to developer-written tests. The abstract and conclusion nevertheless state that 'LLM-improved tests are as readable as developer-written tests.' Please temper the wording to 'no statistically significant difference was observed' or provide an equivalence test, confidence intervals, or a larger sample that can support the stronger claim.","section":"V-F3"}],"minor_comments":[{"comment":"The abstract and introduction state that the approach is 'overall semantically-preserving' across the nine evaluated LLMs, but Table II shows that only five of the nine models achieved 100% success, with the others ranging from 0% to 50%. Please qualify this claim to refer specifically to the five LLMs that passed the RQ1 screening.","section":"Abstract, Section I"},{"comment":"The total number of test cases used in RQ1 is not reported anywhere; success rates are computed over some set of tests, but the reader cannot see how many tests were transformed per class or project. Adding test counts (e.g., as a column in Table I or in the replication package) would improve reproducibility and make the reported percentages interpretable.","section":"Table I, V-C2"},{"comment":"The stability analysis interprets cosine similarities of 0.87–0.94 as 'high stability' without any baseline for comparison. Reporting the similarity between repeated embeddings of the same unchanged test, or between the original EvoSuite test and its improved versions, would help calibrate these numbers.","section":"V-D1, V-D2, Figure 2"},{"comment":"Typos: 'genererated' in the Listing 3 caption and 'similiarity' in Section V-F2 should be corrected.","section":"Listing 3, V-F2"},{"comment":"Reference [53] lists the author as 'G. Ghisolotti', but the paper's author list uses 'Gianluca Ghislotti'; please correct the spelling.","section":"References"},{"comment":"The sentence 'These reports contain indicate how many statements...' has a grammatical error ('contain indicate'); it should read 'These reports indicate how many statements...' or 'These reports contain indicators of how many statements...'.","section":"V-C1"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on arXiv:2412.18843. The paper does something useful and narrow: it shows that a carefully designed multi-step prompt can make five of nine LLMs rename identifiers and test names in Evosuite tests without changing JaCoCo coverage, and that the rewrites are reasonably stable across runs. The nine-model comparison and the stability analysis are genuinely new relative to Gay et al. and TestGen-LLM, and the design choice to isolate the renaming task is sound. I'd give credit for the replication package too.\n\nThe soft spot is the one the stress-test flags: coverage equality is not semantic equivalence. The paper explicitly says it approximates semantic preservation with coverage preservation, and that's a defensible proxy for a screening metric, but the abstract and RQ1 answer say the LLMs 'do not change the semantics.' That's an overstatement. A test could execute the same lines and branches while asserting a different behavior—say, flipping the arguments to assertSame—and the JaCoCo report would come out identical. The five models' 100% success means 100% coverage preservation, not 100% semantic preservation. This doesn't sink the paper, but the claims should be tightened and ideally supplemented with a mutation-based or oracle-level check on a sample.\n\nTwo secondary issues are real but minor. The procedure says ten repetitions for RQ1 but the table and RQ2 text report five; that inconsistency should be corrected. The human study compares improved tests only against developer-written tests, not against the original Evosuite tests, so it can't show how much readability actually improved; and with ten developers the power is low, which the authors acknowledge. Adding an Evosuite baseline would make the readability claim much stronger.\n\nOverall: this is a credible, useful empirical paper with a fixable mismatch between its headline claim and its metric. It deserves a serious referee; the main work in revision is softening the semantics language, fixing the repetition count, and adding the baseline. I'd bring it to a reading group focused on test generation, and I'd cite it if I were working on LLM-based test improvement.","headline":"Solid empirical study of LLM-based renaming of Evosuite tests, but the semantic-preservation claim is stronger than the coverage proxy supports.","tokens_in":20387,"tokens_out":1864,"would_cite":true,"duration_ms":16150,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that renaming test identifiers and test names with large language models preserves test coverage and raises the readability of automatically generated tests to the level of developer-written ones.","keywords":["LLM-based test readability improvement","search-based test generation","semantic preservation","coverage preservation","Evosuite","Java unit tests","human study","prompt engineering"],"falsifier":"Rerun the pipeline on classes where two execution paths share the same line and branch coverage, instrument the improved tests to log which path they actually execute, and compare coverage reports: if any improved test's reported JaCoCo coverage matches the original while its executed path, exception type, or assertion outcome differs, the coverage-preservation proxy is shown to overstate semantic preservation.","tokens_in":19333,"feed_emoji":"🧪","tokens_out":8359,"duration_ms":73219,"temperature":0.7,"pith_summary":"Search-based test generators produce high-coverage unit tests, but their identifiers and test names are arbitrary, so developers find them hard to read and maintain. This paper proposes to keep the generator's coverage advantage and use a large language model to rename only the identifiers and the test name, leaving everything else in each test untouched. In the paper's evaluation, five of nine LLMs preserved the tests' JaCoCo coverage (the lines and branches each test executes) in every repetition, and ten professional developers judged the improved tests to be as readable as tests written by developers. If the result holds, teams can take the otherwise illegible output of automated generators, rename it with an LLM, and commit tests that developers can actually read.","feed_headline":"Five LLMs make auto-generated tests as readable as human ones","feed_subtitle":"Developer ratings show no gap in readability, while identical JaCoCo coverage shows semantics are preserved.","key_machinery":"The mechanism is a multi-step prompt pipeline that constrains what the LLM may change. An informational prompt first gives the model a persona, the task goal, and the focal context of the class under test (class name, constructors, attributes, method signatures) and is kept in memory; then each test case is processed by a separate improvement prompt that includes the test and the source of the methods it calls, with explicit instructions to modify only identifiers and the test name. A third prompt resolves duplicate test names by renaming while keeping test content identical. The other load-bearing piece is the evaluation proxy: semantic preservation is approximated by equality of JaCoCo coverage reports between original and improved tests, and stability is measured by cosine similarity of code embeddings across repetitions.","core_discovery":"The paper's claim is that readability of search-based generated tests can be improved without changing their behavior by asking an LLM to rename test names and local identifiers, and that the result is comparable in readability to human-written tests. The evidence comes from two measurements: identical JaCoCo coverage reports before and after transformation (used, as the paper states, as a proxy for semantic preservation), and readability scores from a randomized survey of ten professional developers. Five of the nine tested LLMs had a 100% success rate on the coverage-preservation check across all projects and repetitions, and the developer survey found no statistically significant readability difference between LLM-improved and developer-written tests, regardless of which of those five models produced the improvement. The transformation is deliberately conservative: only identifiers and test names may change, method calls and control flow must stay fixed.","pith_inferences":["A direct ablation would reveal how much of the readability gain comes from test names versus variable names, and whether the renaming-only constraint is what keeps coverage intact.","The same multi-step prompt likely transfers to other languages and coverage-guided generators, since it only asks for identifier and test-name changes; a rerun on non-Java generator outputs would test whether the 100% preservation rate is Java-specific.","A larger human study with more developers and classes would sharpen the comparison with developer-written tests, because the paper reports that its ten-developer sample did not reach conventional statistical power."],"forward_implications":["Automatic test suites can be made human-comprehensible without losing their coverage, so the main practical objection to using search-based generated tests is weakened.","The multi-step prompt pipeline can be applied as a post-processing step after any coverage-based generator, because it operates on individual test cases rather than regenerating the suite.","The five reliable LLMs are interchangeable for this task in terms of resulting readability, letting users choose a cheaper or faster model without sacrificing readability.","Because the improved tests remain highly similar across repetitions, the results are stable enough to be committed into a codebase and maintained over time."],"supporting_citations":[{"why":"Supplies the search-based Java test generator whose low-readability, high-coverage test output is the input to the readability pipeline.","marker":"[3]"},{"why":"Establishes that identifier and naming features drive perceived readability of unit tests, motivating the renaming-only scope.","marker":"[12]"},{"why":"Shows that descriptive test names matter for readable tests, directly motivating the test-name renaming step.","marker":"[13]"},{"why":"Empirically compares ChatGPT and Evosuite, documenting the coverage vs readability trade-off the hybrid approach targets.","marker":"[20]"},{"why":"Provides the closest LLM-based test improvement baseline that the paper distinguishes by aiming for semantic preservation.","marker":"[24]"},{"why":"Motivates the multi-step prompt design by showing LLMs lose information in the middle of long prompts.","marker":"[25]"},{"why":"Documents LLM output non-determinism, justifying the repeated executions used to measure semantic preservation and stability.","marker":"[49]"}],"fun_headline_variants":["LLM renames make auto-generated tests as readable as human ones","AI boosts test readability without sacrificing coverage","Five LLMs pass readability test with 100% coverage preserved","Auto-generated tests get human-level readability via LLM","Coverage-safe LLM renames close readability gap in tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result stands on the assumption that identical line, branch, method, and statement coverage between the original and improved test means the test's behavior has not changed.","fun_headline_variants_meta":{"raw":{"variants":["LLM renames make auto-generated tests as readable as human ones","AI boosts test readability without sacrificing coverage","Five LLMs pass readability test with 100% coverage preserved","Auto-generated tests get human-level readability via LLM","Coverage-safe LLM renames close readability gap in tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000215,"raw_usage":{"total_tokens":1391,"prompt_tokens":869,"completion_tokens":522,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":485,"completion_tokens_details":{"reasoning_tokens":442}},"tokens_in":485,"tokens_out":522,"duration_ms":4800,"temperature":1.0,"reasoning_tokens":442,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:24:51.313415+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the pipeline on classes where two execution paths share the same line and branch coverage, instrument the improved tests to log which path they actually execute, and compare coverage reports: if any improved test's reported JaCoCo coverage matches the original while its executed path, exception type, or assertion outcome differs, the coverage-preservation proxy is shown to overstate semantic preservation.","supporting_citations":[{"cited_title":"Lost in the middle: How language models use long contexts,","cited_arxiv_id":null,"evidence_quote":"Motivates the multi-step prompt design by showing LLMs lose information in the middle of long prompts."}],"review_version":1}