{"id":"be1dbe46-c2f8-4199-ac93-fc136a8b09ac","arxiv_id":"2507.18316","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A test-repair pipeline, combining static analysis and re-prompting, raises LLM-generated unit test coverage and mutation killing by roughly 20-30 percent over a plain prompt baseline on six Java projects.","lead":"This paper introduces YATE, a method that repairs the broken unit tests that language models produce, by fixing imports, constructors, method calls, and assertions, then re-prompting for uncovered code. On six Java projects it reports roughly 20 to 30 percent higher line, branch, and mutation coverage than a plain LLM baseline and four published tools.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-run stochastic evaluation leaves the headline 20-32 pp advantages as unverified point estimates; repeated runs are needed to rule out sampling noise, especially in per-project/per-model margins.","rationale":"The paper's central claim is well-designed and the ablation plus multi-LLM sensitivity check are genuine evidence. The main load-bearing gap is the absence of repeated runs for a stochastic pipeline; the reader's weakest_assumption identifies exactly this, and I agree. The concrete test above would resolve whether the reported effect sizes are stable. I do not see an internal inconsistency that would warrant rejection; the oracle-fixing rules are a plausible secondary threat but would require a separate analysis of repaired tests to demonstrate inflation. Note also Section 7's sentence about using 'ChatUniTest' as a plugin with implementations of all compared algorithms is garbled and should be corrected for reproducibility, but it does not affect the core argument. Because the paper ships no artifact and reports no confidence intervals, CONDITIONAL is the appropriate verdict; repeating the study on a sample would raise confidence. Thus verdict_should_be is UNCHANGED.","tokens_in":20929,"tokens_out":16740,"duration_ms":166688,"concrete_test":"Run YATE and LLM-Plain independently five times on a stratified sample of 30 classes per project (e.g., 5 projects x 30 classes = 150 classes) with the same GPT-4o-mini configuration, and record per-class line, branch, and mutation coverage for each run. Compute the per-class variance and a bootstrap 95% confidence interval for the mean difference (YATE minus Plain) for each project and metric. If any project CI includes zero, the universal-superiority claim weakens; if all CIs exclude zero, the single-run design is a presentation gap rather than a validity threat. Repeat the same for GPT-4.1 on the subset of classes where the margin was under 10 pp (RQ4).","verdict_should_be":"UNCHANGED","load_bearing_attack":"All effectiveness comparisons (Tables 4, 6, 7, 9) are computed from a single execution of each stochastic LLM pipeline per class. The Mann-Whitney U tests and VDA effect sizes in Tables 5 and 8 treat these single samples as exact values, ignoring run-to-run variance from LLM sampling. The headline figures — 32.06 pp higher line coverage and 21.77 pp higher mutation score over plain, and roughly 20/20/20% over HITS — are point estimates without confidence intervals. The margins are thin in exactly the cells that matter most: in RQ4, GPT-4.1 YATE beats its own plain baseline by only 5.31 pp line coverage (52.61% vs 47.30%); in RQ5, YATE-C branch coverage on CH (37.02%) is lower than HITS (44.3%). If a second run changes a few per-class values by even a few percentage points, these results and the p-values could shift. Because the pipeline's LLM calls are stochastic, a single run per class cannot support claims of 'consistently' or 'significantly' higher coverage without variance estimates.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents YATE, an LLM-based unit test generation technique built around four components: chain-of-thought initial generation, compilation-error fixing via static analysis and re-prompting, oracle fixing using execution outcomes, and coverage-based test augmentation. The authors evaluate YATE on six Java projects (393 classes) with four LLMs, comparing it against a plain LLM prompting baseline and four published tools (HITS, SymPrompt, TestSpark, CoverUp) using line coverage, branch coverage, mutation score, and number of LLM calls as the cost metric. The central claims are that YATE improves line coverage by 32.06 percentage points and mutation score by 21.77 percentage points over plain LLM generation at class level, and that it outperforms all four baseline tools by roughly 20 percentage points in coverage and mutation score at comparable cost.","tokens_in":21140,"tokens_out":8125,"duration_ms":81421,"significance":"If the reported results hold, YATE would be a useful contribution to LLM-based unit test generation: it provides evidence that repairing near-miss invalid tests, rather than discarding them, can substantially improve the effectiveness of LLM-generated test suites, and it identifies class-level prompting as a cost-effective configuration. The study has several strengths: a standard benchmark design with six projects and 393 classes, a four-component ablation, a multi-LLM sensitivity analysis, use of Mann-Whitney U tests and VDA effect sizes, and explicit measurement of LLM-call cost. However, the stochastic nature of the LLM pipeline means the current single-run evaluation is not sufficient to support the strength of the headline claims, and the paper's reporting of percentage-point differences as percentage improvements needs correction.","major_comments":[{"comment":"The effectiveness evaluation is based on a single execution of each stochastic LLM pipeline per class; no repeated runs, confidence intervals, or variance estimates are reported for any coverage or mutation value, yet Tables 5 and 8 report Mann-Whitney U p-values and VDA effect sizes computed from these single-sample values. The headline advantages (e.g., 32.06 percentage points line coverage over plain, and roughly 20/20/20 over HITS) are therefore point estimates. This is not a purely hypothetical concern: in RQ4 (Table 7), GPT-4.1 YATE exceeds its plain baseline by only 5.31 pp line coverage (52.61 vs 47.30), and in RQ5 (Table 9), YATE-C branch coverage on CH (37.02%) is below HITS (44.3%). If repeated runs alter per-class values by even a few percentage points, the 'consistently' and 'significantly' claims in Findings 3 and 6 could change. Please add repeated runs or, at minimum, a perturbation-based sensitivity analysis, and report distributions or confidence intervals for all headline numbers.","section":"§5.2, Tables 4, 7, 9"},{"comment":"The phrase 'cover on average 32.06% more lines and kill 21.77% more mutants' is misleading. These numbers are percentage-point differences in average coverage: from Table 4, average class-level line coverage is 64.94% for YATE vs 32.88% for plain, and average mutation score is 43.98% vs 22.22%. The corresponding relative improvements on the averaged values are approximately 97.5% and 97.9%, while the 131.45%, 143.02%, and 156.07% figures in Section 5.2 appear to be averages of per-project relative improvements. Please report absolute coverage values, percentage-point differences, and relative improvements with unambiguous labels throughout, and correct the abstract so that '32.06% more lines' does not overstate the relative gain.","section":"Abstract and §5.2"},{"comment":"The claim that combining class-level and method-level YATE test suites 'consistently yields significantly higher coverage' is not supported by any statistical test. Table 5 only compares YATE(C) versus YATE(M), and no p-values or effect sizes are reported for the Y-Comb column against either constituent. The Y-Comb advantages in Table 4 are also reported without variance estimates. Please add the missing statistical comparisons or soften the claim to an observation about the observed totals and averages.","section":"§5.2, Finding 4, Table 5"}],"minor_comments":[{"comment":"The validity section states that 'we use ChatUniTest, a widely used plugin with the implementation of all algorithms we compare with,' but HITS, SymPrompt, TestSpark, and CoverUp are separate tools, not implementations inside ChatUniTest. Please correct this sentence or clarify what was actually reused from existing tooling.","section":"§7"},{"comment":"Several tables distinguish 'Total' (aggregated) from 'Average' (unweighted per-project) rows, but the text sometimes cites one without specifying which. Please define both quantities explicitly in Section 4.4 and refer to them consistently in the findings.","section":"Tables 2–9"},{"comment":"Several configuration choices that directly control the pipeline are stated as fixed defaults without sensitivity analysis or full specification: the maximum number of oracle-fixing iterations (3), the call-graph depth used in compilation fixing, the single extra coverage-augmentation cycle, the exception-oracle avoid list, and the LLM temperature and decoding parameters. Please report the exact settings used and, where feasible, include a limited sensitivity analysis for the most consequential parameters.","section":"§3.4–3.6, §4.3"},{"comment":"The seen/unseen cutoff date is justified for GPT-4o-mini only, but RQ4 uses other models with different training data; the paper acknowledges this in Section 4.2 but does not revisit the data-leakage threat for those models in the validity discussion. Please address this explicitly.","section":"§4.2 and §7"},{"comment":"Please fix typographical and wording issues: 'utilises' in the abstract, 'adn' in Section 3.5.1, 'BGP' in Figure 3, 'askes' in Section 3.6, 'threads' in Section 7, and 'living out' in Section 2.","section":"Throughout"},{"comment":"No artifact or replication instructions are provided. Given the complexity of the YATE pipeline and the number of configurable components, a public artifact with the scripts, prompts, and parameter settings would substantially strengthen the paper's reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The central empirical claim is plausible and the study design is standard for the field, but the single-run stochastic evaluation is a substantive gap that needs to be addressed before the headline numbers can be taken as reliable. I did not see any concerning citation or novelty issues beyond what is noted in the report. Providing an artifact and a data-availability statement would also help the journal's reproducibility standards."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: YATE is a genuinely useful empirical study. The core claim—that repairing near-miss LLM-generated tests through dependency retrieval, oracle fixing, and coverage prompting improves line coverage, branch coverage, and mutation killing by roughly 20+ percentage points—holds up directionally and is worth taking seriously. The integration is new even if the individual components are not, and the ablation finding that compilation-relevant feedback matters more than coverage feedback is the most interesting result in the paper.\n\nThe evaluation is better than much of this subfield: six projects, 393 classes, four LLMs, a plain baseline, four SOTA baselines, Mann-Whitney U with VDA effect sizes, an ablation, and a cost analysis. The class-level prompting result—nearly as effective as method-level at a fraction of the LLM calls—is practically important and likely generalizable. The authors also deserve credit for running on all classes rather than a self-selected subset.\n\nThe real soft spot is exactly what your stress-test flags: every coverage and mutation number is a single stochastic run per class, with no repeated runs, confidence intervals, or variance estimates. The main margins are large enough that I doubt the headline conclusions flip, but the thin cells—GPT-4.1 vs its own plain baseline (5.31 pp line coverage), YATE class-level branch coverage on CH (37.02% vs HITS 44.3%)—could move with sampling noise, and the reported p-values treat single samples as exact. This is fixable but it is a genuine gap. Relatedly, no raw data, artifact, or complete prompts are shipped, so independent verification means reimplementation from the paper, which is harder than it should be. The oracle-fixing step (asserting observed behavior) is conventional but deserves a more explicit caveat about encoding bugs. The seen/unseen cutoff is blunt, though acknowledged as such.\n\nWho is this for: researchers and practitioners working on LLM-based unit test generation. It gives a strong data point that repair beats regeneration and that context retrieval is the highest-leverage component.\n\nRecommendation: send to peer review. The paper deserves serious refereeing. The revision should add repeated runs and variance/CI reporting, release the artifact and prompts, and tone down the 'consistently' language in the secondary comparisons.","headline":"Solid empirical paper: repairing near-miss LLM tests buys real coverage and mutation gains, but single-run stochastic evaluation and no artifact leave the exact effect sizes less certain than the prose claims.","tokens_in":21696,"tokens_out":2715,"would_cite":true,"duration_ms":27898,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Repairing broken LLM tests lifts coverage 32 percent.","keywords":["LLM-based unit test generation","test repair","near-miss tests","compilation error fixing","oracle fixing","coverage-guided test augmentation","mutation testing"],"falsifier":"Run YATE and the plain baseline ten times over the same classes and compare the coverage and mutation distributions: if the 32-percentage-point line-coverage advantage and the 21.77-percentage-point mutation advantage fall within run-to-run noise, the central claim does not hold.","tokens_in":20709,"feed_emoji":"🧪","tokens_out":7681,"duration_ms":76294,"temperature":0.7,"pith_summary":"This paper argues that 'near-miss' unit tests — tests an LLM generates that fail to compile or fail to pass — should be repaired rather than thrown away, because they already target the program logic that needs testing. It presents YATE, a four-part repair pipeline: chain-of-thought test planning, compilation fixing that retrieves class dependencies and method signatures from the project, oracle fixing that adjusts failing assertions to match observed behavior, and branch-coverage feedback for extra tests. In an evaluation on six Java projects with four LLMs, YATE's tests cover on average 32.06 percentage points more lines and kill 21.77 percentage points more mutants than a plain LLM-based method, and it beats four published generators on line, branch, and mutation metrics at comparable or lower LLM-call cost. The authors conclude that repair, especially compilation-relevant feedback, is what unlocks the value of LLM-generated tests.","feed_headline":"Repairing broken LLM tests lifts coverage 32 percent","feed_subtitle":"YATE feeds dependency code and error logs back to the model, killing 22% more mutants than plain prompting.","key_machinery":"The load-bearing mechanism is the repair loop. Rather than treating a compiler error or failing assertion as a dead end, YATE uses rule-based static analysis on the project repository to obtain the missing context — imports, constructor signatures, classes implementing an interface, method bodies on the call graph — and feeds that context, plus the error log, back to the LLM in a new prompt. A second repair layer fixes oracles by substituting observed runtime values into assertions and by converting exception expectations, with a final LLM pass for complex values; a third pass re-prompts with uncovered branches. The core finding is that the combination of all four components outperforms any single component or their removal, and that compilation-relevant dependency feedback is the most valuable ingredient.","core_discovery":"YATE's central claim is that the discarded invalid outputs of an LLM are a recoverable source of testing value: when the model produces a test that almost compiles or almost passes, the near-miss contains the targeting logic needed to test the class, and repairing it is cheaper and more effective than discarding and re-prompting blindly. The paper operationalizes this with a workflow that (1) has the LLM first produce a coverage-oriented test plan, (2) repairs compilation errors by statically analyzing the repository, directly fixing imports and constructors, walking the method call graph for dependencies, and re-prompting with the retrieved class and method code, (3) fixes failing oracles by replacing assertion values with observed outputs and by inverting or generating exception assertions, and (4) prompts again with uncovered branches. On 393 class/method problems across six open-source Java projects, YATE reaches 51% average branch coverage, reporting about 22 percentage points higher line coverage, 20 percentage points higher branch coverage, and 20 percentage points more mutants killed than four published methods, with class-level prompting needing about five times fewer LLM calls than method-level prompting. The ablation study supports the paper's emphasis on repair: removing compilation fixing costs the most, and dependency-aware compilation feedback outweighs bare coverage feedback.","pith_inferences":["The paper does not test this, but a natural extension is that the repair components could be grafted onto other generators: if the value lies in recovering near-misses, then feeding dependency code and error logs to any LLM test generator's output could produce stacked gains rather than redundant ones.","Because the compilation-fixing step retrieves project code by walking the call graph, it is effectively a retrieval-augmented generation strategy; an isolated ablation that supplies the same retrieved context in the initial prompt rather than after a failure would reveal how much of the gain comes from the repair framing versus the added context alone.","The stochastic one-run evaluation implies a methodological corollary: before relying on these margins, practitioners should estimate run-to-run variability, since a few percentage points of coverage noise could change which configurations are actually distinguishable.","If the missed-opportunity view is right, test generators should be designed to provoke and collect informative invalid tests rather than to avoid them, and cost models should count the price of repairing a near-miss against the price of a fresh generation attempt."],"forward_implications":["Repairing near-miss tests directly converts wasted LLM output into passing tests: YATE raises the passing rate from about 50% for plain prompting to 90.18% at class level and 82.96% at method level, producing thousands more usable tests.","Class-level prompting is the cost-effective operating point for LLM test generation: it needs about five times fewer LLM calls than method-level prompting while losing only about 5% coverage, and it still beats all baselines.","Compilation-relevant feedback matters more than coverage feedback: replacing YATE's dependency-aware compilation fixing drops branch coverage the most, suggesting that generators should invest in dependency retrieval before branch guidance.","The gains are not tied to one model: YATE outperforms plain prompting across GPT-4o-mini, GPT-4.1, DeepSeek-V3, and Codestral, so the repair mechanism transfers across LLMs.","Combining class-level and method-level suites yields higher coverage than either alone, indicating that the two granularities supply complementary contextual information."],"supporting_citations":[{"why":"Supplies the strongest baseline, which YATE must beat, and several of the subject projects.","marker":"[40]"},{"why":"Baseline method that prompts from execution paths; the paper compares YATE's coverage against it.","marker":"[32]"},{"why":"Baseline method that generates tests for full branch coverage; the paper compares against it.","marker":"[33]"},{"why":"Baseline that feeds uncovered code back to the LLM; the closest coverage-feedback comparator.","marker":"[31]"},{"why":"Supplies two subject projects and the earlier repair mechanism that YATE extends.","marker":"[8]"},{"why":"Supplies some of the benchmark projects, with a check that the models had not seen them.","marker":"[35]"},{"why":"Closest prior test-repair approach; the paper contrasts its oracle-only templates with YATE's dependency-aware repair.","marker":"[18]"},{"why":"Establishes the feedback-driven repair loop and the LLM-call cost model that YATE's cost comparisons use.","marker":"[46]"},{"why":"Provides the mutation testing tool used to compute the mutant-killing metric.","marker":"[9]"},{"why":"Provides the code analysis and method call graph used by YATE's compilation-fixing phase.","marker":"[30]"}],"fun_headline_variants":["Fix near-miss LLM tests to gain 32% coverage","YATE repairs bad LLM tests, boosting coverage 32%","Test repair turns discarded LLM outputs into coverage","Repairing LLM test drafts adds 20% branch coverage"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison treats a single run of each LLM-based method on each class as a trustworthy measurement of that method's performance, even though LLM generation is stochastic; if repeated runs shift coverage by even a few percentage points per class, the reported 20–32% margins and the significance tests could change.","fun_headline_variants_meta":{"raw":{"variants":["Fix near-miss LLM tests to gain 32% coverage","YATE repairs bad LLM tests, boosting coverage 32%","Test repair turns discarded LLM outputs into coverage","Repairing LLM test drafts adds 20% branch coverage"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000575,"raw_usage":{"total_tokens":2759,"prompt_tokens":1034,"completion_tokens":1725,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":650,"completion_tokens_details":{"reasoning_tokens":1654}},"tokens_in":650,"tokens_out":1725,"duration_ms":13181,"temperature":1.0,"reasoning_tokens":1654,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:14:32.089217+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run YATE and the plain baseline ten times over the same classes and compare the coverage and mutation distributions: if the 32-percentage-point line-coverage advantage and the 21.77-percentage-point mutation advantage fall within run-to-run noise, the central claim does not hold.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Baseline method that generates tests for full branch coverage; the paper compares against it."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies some of the benchmark projects, with a check that the models had not seen them."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the mutation testing tool used to compute the mutant-killing metric."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the code analysis and method call graph used by YATE's compilation-fixing phase."}],"review_version":2}