{"id":"156abdb5-d6ca-4d69-a7af-73fa6bab689c","arxiv_id":"2607.22880","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"For LLM-generated Java tests, coverage and mutation predict real-bug detection mainly when comparing models on bug-free code, not when the code under test is buggy, and suite size is not a major confounder.","lead":"This study re-tests whether code coverage and mutation scores predict whether AI-generated test suites catch real bugs, using more than 100,000 tests from 11 language models on Java bugs. It finds these proxies are useful for comparing models when the code is assumed bug-free, but fail when the code under test itself is buggy.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Inter-model correlations rely on only 13 model settings with no leave-one-out or confidence-interval analysis, and the buggy-code half of the context-dependence claim is deferred to the artifact; robustness checks are needed before the central claim is settled.","rationale":"The reader's weakest assumption identifies the same load-bearing concern I find: the inter-model evidence, which is what distinguishes this study from prior work and supports the abstract's positive claim, is based on only 13 model-level data points with no explicit robustness analysis. I agree that this warrants a conditional verdict rather than rejection. The study is otherwise well-scoped: it uses a substantial corpus of over 101,000 generated tests across 11 models, separates combined, intra-model, and inter-model views, reports both average and accumulated aggregations, and provides a public artifact. The central argument is not circular because coverage, mutation, suite size, and bug detection are measured independently. However, the manuscript's own Internal Validity paragraph in Section 8 concedes the small inter-model sample without quantifying its effect, and Section 5.4 explicitly defers the buggy-input coefficient tables to the replication package. These are not fatal, but they make the 'context-dependent' headline harder to verify from the paper alone. The proposed leave-one-out and bootstrap checks would directly test whether one or two influential model settings drive the moderate-to-strong inter-model correlations. If the correlations survive exclusion of each setting, the cross-model claim is credible; if not, the conclusions should be restricted to the particular models studied. Because the reader's conditional verdict already requires these robustness diagnostics, my assessment does not change the verdict recommendation.","tokens_in":23712,"tokens_out":6042,"duration_ms":59101,"concrete_test":"Perform a leave-one-out analysis across all 13 model settings for every inter-model correlation in Tables 6, 7, 10, 11, 13, and 14, and report the range of Pearson r and p-values under each exclusion, together with 95% bootstrap percentile intervals. If any headline coefficient (e.g., average branch coverage vs. bug detection in Table 10 or raw mutation score vs. bug detection in Table 13) drops below roughly 0.4 or loses p<0.05 when a single model setting is removed, the cross-model claim is not robust. As a secondary check, recompute the correlations while treating the two configurations of each hybrid model as non-independent; if the effective sample is below 11 and the intervals widen substantially, the conclusion should be softened accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The positive half of the central claim—coverage and mutation are meaningful signals when comparing across models—rests almost entirely on the inter-model correlations described in Section 3.7 and reported in Tables 6, 7, 10, 11, 13, and 14. These correlations are computed from one aggregate data point per model setting, yielding 13 points total. The paper acknowledges the small sample in Section 8 (Internal Validity), but it provides no leave-one-out analysis, no bootstrap confidence intervals, and no robust regression diagnostics. At n=13, a single model can move a Pearson coefficient by several tenths and change significance, and the effective number of independent points is likely even smaller because the 13 settings include paired reasoning and non-reasoning configurations of the same hybrid models. Since the combined and intra-model views are weak, the cross-model conclusion has no independent supporting line of evidence. Additionally, the buggy-input side of the claims in RQ2 is asserted without in-paper statistics: Section 5.4 states that the full coefficients are omitted for space and only available in the replication package. Thus both halves of the context-dependence conclusion are less secured in the manuscript than the abstract suggests. These are addressable reporting and robustness gaps rather than fatal flaws.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper is a large-scale conceptual replication of Inozemtseva et al. (ICSE 2014) and Papadakis et al. (ICSE 2018) for LLM-generated unit tests. Using 318 buggy focal methods from Defects4J and 11 LLMs (configured into 13 model settings), the authors generate 8,268 test suites containing 101,123 tests from both fixed and buggy versions of the focal methods. They measure statement, branch, and modified-condition coverage with CodeCover, mutation scores with PIT, and real-bug detection with a differential-testing oracle (a test is effective if it passes on the fixed version and fails on the buggy version). Correlations among test suite size, coverage, mutation score, and bug-detection ratio are computed at three granularities (combined, intra-model, inter-model) and with two aggregations (average, accumulated). The main findings are that coverage and mutation are only weakly correlated with effectiveness in the combined and intra-model views, but moderately to strongly correlated across models when tests are generated from bug-free code; that these cross-model correlations largely survive controlling for test suite size; and that coverage is not informative when tests are generated from buggy code. The authors conclude that proxy metrics are context-dependent and that suite size is not a dominant confounder for LLM-generated suites.","tokens_in":23833,"tokens_out":5766,"duration_ms":50956,"significance":"If the findings are robust, the paper makes a substantive contribution: it demonstrates that established negative results for human-written test suites do not transfer uniformly to LLM-generated tests, and it provides the community with a more nuanced interpretation of coverage- and mutation-based evaluations. Strengths include the large generated corpus, the use of real Defects4J bugs rather than mutants alone, focal-method-level suite construction, a differential-testing oracle, two aggregation schemes, and a publicly archived replication package with scripts and raw data. The manuscript is also commendably explicit about its limitations. However, the positive cross-model conclusion and the buggy-input conclusion are currently less secured in the paper than the abstract suggests: the former rests on n=13 inter-model correlations without robustness diagnostics, and the latter is deferred to the replication package. Both issues are addressable with additional analysis and reporting, so I view this as a major-revision rather than a rejection.","major_comments":[{"comment":"The central claim that coverage loses predictive power when the code-under-test is buggy rests entirely on a summary statement: the text says, 'Due to space constraints... we do not report the full set of coefficients here; complete statistics are available in our replication package.' No coefficient, p-value, or effect size is given in the manuscript for the buggy-input condition under any of the three analysis views. Because the abstract and Section 7 present the buggy-input result as half of the main context-dependence conclusion, this is a load-bearing reporting gap. Please add a compact table or appendix with the combined, intra-model, and inter-model correlations for the buggy-input suites, analogous to Tables 9–11 for the bug-free condition.","section":"§5.4 (RQ2, buggy-input setting)"},{"comment":"The cross-model conclusion that coverage and mutation are meaningful when comparing across models is based on correlations computed from 13 model-level data points. The paper acknowledges the small sample in §8, but it provides no leave-one-out analysis, bootstrap confidence intervals, or robust-regression diagnostics. At n=13, one or two models can move a Pearson coefficient by several tenths and change its significance, and the effective number of independent points is likely smaller than 13 because the settings include paired reasoning and non-reasoning configurations of the same hybrid models. Since the combined and intra-model views are weak, the inter-model view is the only supporting line of evidence for the positive half of the central claim. Please add robustness diagnostics (e.g., leave-one-out tables, bootstrap confidence intervals, or Spearman/robust fits) and state how the paired configurations are treated in the analysis.","section":"§3.7; Tables 6, 7, 10, 11, 13, 14"}],"minor_comments":[{"comment":"The paper says the evaluation yields 13 model settings, but Table 2 lists 11 model names without marking the hybrid configurations that produce the extra settings. Please enumerate the 13 settings explicitly or add a configuration column to the table.","section":"§3.3; Table 2"},{"comment":"The description of the sampling procedure says 'we construct 1,000 unique draws' but does not state whether sampling is with or without replacement, how uniqueness is determined, or how many suites are available per focal method and model. Please clarify these details, as they affect the interpretation of the combined- and intra-model correlations.","section":"§3.5"},{"comment":"Several p-values in Table 8 are reported as 0.0. Since a p-value cannot be exactly zero, please report these as inequalities or with a bound (e.g., p < 10^{-300}) rather than 0.0.","section":"§4.4; Table 8"},{"comment":"The Guilford scale is described for verbal labels of correlation strength, but the paper does not state whether these thresholds are applied only to Pearson coefficients or also to Kendall's tau. Please clarify, since some conclusions in Sections 4 and 5 rely on the threshold language.","section":"§3.7"}],"recommendation":"major_revision","confidential_remarks":"The paper is a good fit for an ISSTA-adjacent empirical venue and the artifact is a clear strength. The two major concerns are both about reporting robustness rather than about the design being wrong: the buggy-code statistics need to be in the paper, and the inter-model analysis needs diagnostics. I did not independently execute the artifact; my assessment is based on the manuscript text. If the authors add the missing coefficients and robustness checks, I would expect the revised version to be publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, well-scoped replication with one real robustness gap. It gives the LLM test-generation evaluation crowd something to argue about: the old Inozemtseva and Papadakis warnings about coverage and mutation proxies, and about size as a confounder, do not transfer cleanly to LLM-generated tests. The paper's contribution is showing that the transfer fails in a context-dependent way rather than simply present or absent.\n\nWhat is new: the study runs the analysis on 101,123 tests from 11 LLMs (13 model settings), with a differential-testing oracle on Defects4J focal methods, under both bug-free and buggy prompts, with two aggregation schemes and three analysis views. That is a genuinely different population from human-written tests and from EvoSuite-generated tests, so the divergence is interesting rather than trivially expected. The bug-free/buggy contrast is coherent, and the StringUtils.equals example in Figure 4 makes the mechanism concrete. Credit where due: there are no fitted parameters, coverage, mutation, and bug detection are measured independently, and the artifact ships code, data, and scripts. The framing as a conceptual replication is honest.\n\nSoft spots: the positive half of the central claim—coverage and mutation are meaningful when comparing across models—rests on 13 model-level data points. The paper acknowledges the small sample but gives no leave-one-out analysis, no bootstrap confidence intervals, and no robust regression diagnostics. With hybrid models contributing two settings each, the effective number of independent observations is smaller than 13, and one or two outlier models could plausibly move those correlations by several tenths. Separately, the buggy-code side of the context-dependence claim is asserted with \"the trend is consistent\" and the full coefficients are deferred to the replication package; that is exactly the part that needs to be in front of the reader. The many correlation tests also get no multiplicity correction. None of this is fatal: the combined and intra-model views are weak as claimed, and the pattern across Tables 6, 7, 10, 11, 13, and 14 is suggestive. But the headline inter-model claim is held up by a load-bearing n=13 and should be presented as such.\n\nWho this is for: anyone evaluating LLM-based test generation, anyone citing coverage or mutation as a proxy for effectiveness, and anyone interested in whether classic software-engineering empirical results survive changes in the test-generating population. It deserves a serious referee. I would send it out and ask for leave-one-out or bootstrap diagnostics on the inter-model correlations, in-paper buggy-code coefficients, and a modest acknowledgment of multiple testing. The result, if it holds up, changes how a fast-moving literature should read its own metrics.","headline":"A solid, well-scoped replication that gives the LLM test-gen evaluation crowd a genuine divergence from Inozemtseva and Papadakis, but the headline cross-model result rests on 13 aggregate points and needs robustness work before it should be treated as settled.","tokens_in":24467,"tokens_out":1563,"would_cite":true,"duration_ms":16210,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For LLM-generated tests, coverage and mutation scores track real-bug detection only when the code given to the model is bug-free and the comparison is across models.","keywords":["test effectiveness","code coverage","mutation score","LLM-generated unit tests","replication study","real-bug detection","test suite size","proxy metrics"],"falsifier":"Recompute the inter-model correlations with one model removed at a time; if any single deletion pushes the key coefficients (average branch coverage vs. bug detection, raw mutation score vs. bug detection, branch vs. raw mutation at $k=10$) below the moderate threshold or past $p=0.05$, the across-model claim fails to generalize. Adding a new set of LLMs and seeing whether the correlations reproduce would settle it just as directly.","tokens_in":23398,"feed_emoji":"🐛","tokens_out":10664,"duration_ms":87630,"temperature":0.7,"pith_summary":"This paper claims that coverage and mutation score — the standard proxies used to judge LLM-generated unit tests — do relate to real-bug detection, but only under specific conditions. It replicates the two earlier studies cited as [30] and [40] using 101,123 tests generated by 11 large language models (13 model configurations) for 318 real Java defects, with each defect's code fed to the model in both its fixed and its buggy form. The results diverge from the earlier conclusions about human-written tests: test suite size is not a strong confounder, and when the code can be assumed bug-free, coverage and mutation correlate moderately to strongly with bug detection when models are compared against each other. When the code-under-test is itself buggy, coverage ceases to predict whether generated tests will expose the bug, and mutation testing cannot be applied. The takeaway is a context-dependent verdict on proxy metrics plus practical guidance for how to evaluate LLM-based test generation.","feed_headline":"Coverage and mutation scores only signal bug-finding on clean code","feed_subtitle":"Across 11 LLMs, coverage and mutation track real-bug detection only when the code-under-test is bug-free.","key_machinery":"The argument is carried by a three-granularity correlation design paired with a two-condition prompt experiment. For each of the 318 defects, the same LLM is prompted with both the fixed and the buggy version of the focal method; coverage (statement, branch, modified condition), mutation score, suite size, and bug-detection ratio are measured per method and aggregated two ways (average over methods, and accumulated global ratio). Correlations are then computed at three levels: pooled across all models, separately within each model, and between models with each model reduced to a single data point, under both unconstrained suite size and fixed sizes $k\\in\\{3,5,10\\}$. The inter-model level is where weak pooled correlations become the moderate-to-strong signals that ground the main claim, and the buggy-input condition is what draws the boundary of that claim.","core_discovery":"The central discovery is that the relationship among coverage, mutation score, test suite size, and real-bug detection for LLM-generated tests is not the relationship reported for human-written tests in [30] and [40]. Pooling suites across models or looking within a single model gives mostly weak correlations, but collapsing each model into one aggregated data point reveals moderate-to-strong signal: average branch coverage versus raw mutation score at size $k=10$ gives $r=0.780$, average branch coverage versus bug-detection ratio gives $r=0.861$, and raw mutation score versus bug detection gives $r=0.863$, with most coefficients staying moderate to strong after controlling suite size at $k\\in\\{3,5,10\\}$. This across-model signal holds when the code provided to the LLM is the fixed, bug-free version. When the same pipeline is run on the buggy version, coverage–bug-detection correlations become uniformly weak, and mutation analysis is not applicable because it presupposes a passing suite. The paper also finds that statement, branch, and modified-condition coverage are not interchangeable for LLM-generated tests, and that suite size is only weakly correlated with mutation score and bug detection.","pith_inferences":["The across-model conclusion rests on 13 model-level data points; a leave-one-out analysis or bootstrap confidence intervals would show whether a single outlier model is carrying the strong coefficients. This check is not in the paper, which only acknowledges the small sample.","The buggy-input result points toward assertion quality as the operative factor: tests generated from buggy code often assert the faulty behavior, so a metric that scores whether a generated assertion matches the fixed version's behavior might predict bug detection where coverage cannot.","Because every suite comes from one prompt template, the context-dependence finding may itself be prompt-dependent; varying the prompt (adding documentation, removing context, changing style) would test whether the across-model signal persists.","The focal-method-level sampling likely explains why suite size is not a confounder; a project-level pool that mixes tests across methods could reintroduce a size effect, which would reconcile this result with earlier findings."],"forward_implications":["When the benchmark code can be assumed bug-free, ranking LLM test generators by average branch coverage or raw mutation score should roughly reproduce their real-bug detection ranking, even after controlling for the number of tests.","In bug-detection tasks where the code-under-test may already be buggy, coverage should not be used as a proxy; the paper argues for measuring bug detection directly.","Test suite size is not a dominant confounder for LLM-generated suites, so the size-control correction that prior work applied to human-written test pools does not carry over unchanged.","Statement, branch, and modified-condition coverage carry different information for LLM-generated tests; a single coverage number can mislead.","Compilation and pass rates are best read as cost-effectiveness measures, not as evidence of bug-detection ability."],"supporting_citations":[{"why":"Supplies the earlier coverage–size–mutation correlation analysis that this study replicates and finds does not transfer to LLM-generated tests.","marker":"[30]"},{"why":"Supplies the earlier mutation-score–real-fault correlation analysis whose conclusion this study re-examines.","marker":"[40]"},{"why":"Provides the real-bug benchmark with buggy and fixed focal methods used as prompt inputs, oracles, and effectiveness targets.","marker":"[34]"},{"why":"Establishes that LLMs generate fewer bug-detecting tests when prompted with buggy code, motivating the buggy-input condition and its interpretation.","marker":"[29]"},{"why":"Provides the prompt-construction and unit-test-extraction workflow adopted for all test generation.","marker":"[57]"},{"why":"Supplies the mutation testing tool used to compute raw and normalized mutation scores.","marker":"[13]"},{"why":"Supplies the coverage measurement tool for statement, branch, and modified condition coverage.","marker":"[1]"},{"why":"Defines the correlation-strength scale used to classify coefficients as weak, moderate, strong, or very strong.","marker":"[23]"}],"fun_headline_variants":["Coverage and mutation predict bugs only for bug-free code","Clean code needed for coverage to signal real bugs","LLM test metrics work only when code is bug-free","Coverage and mutation are unreliable on buggy code","Bug-free code required for coverage to track real bugs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 13 model-level data points, one per LLM configuration, are representative enough that the moderate-to-strong inter-model correlations reflect a real relationship between proxy metrics and bug detection rather than the pull of one or two outlier models.","fun_headline_variants_meta":{"raw":{"variants":["Coverage and mutation predict bugs only for bug-free code","Clean code needed for coverage to signal real bugs","LLM test metrics work only when code is bug-free","Coverage and mutation are unreliable on buggy code","Bug-free code required for coverage to track real bugs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000279,"raw_usage":{"total_tokens":1734,"prompt_tokens":1101,"completion_tokens":633,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":717,"completion_tokens_details":{"reasoning_tokens":556}},"tokens_in":717,"tokens_out":633,"duration_ms":5465,"temperature":1.0,"reasoning_tokens":556,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:27:45.211756+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the inter-model correlations with one model removed at a time; if any single deletion pushes the key coefficients (average branch coverage vs. bug detection, raw mutation score vs. bug detection, branch vs. raw mutation at $k=10$) below the moderate threshold or past $p=0.05$, the across-model claim fails to generalize. Adding a new set of LLMs and seeing whether the correlations reproduce would settle it just as directly.","supporting_citations":[{"cited_title":"CodeCover","cited_arxiv_id":null,"evidence_quote":"Supplies the coverage measurement tool for statement, branch, and modified condition coverage."},{"cited_title":"Guilford","cited_arxiv_id":null,"evidence_quote":"Defines the correlation-strength scale used to classify coefficients as weak, moderate, strong, or very strong."}],"review_version":2}