{"id":"33ad7be8-7d2d-4910-a022-e34f0e1d4cc4","arxiv_id":"2412.14137","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Coverage-guided LLM test generators systematically filter out bug-revealing tests and can end up validating bugs instead of finding them.","lead":"This study tests three AI coding tools that write unit tests automatically, feeding them real buggy code. It finds that tools designed to keep only passing tests often discard tests that reveal bugs and keep tests that lock in the buggy behavior.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 59.6%/68.1% bug-validating percentages are only as sound as the untested assumption that Refactory's reference solutions are the unique correct outputs for all 287 samples; with just four problem statements, ambiguous inputs could inflate the headline.","rationale":"The central claim has two layers. The first layer, that coverage-driven, pass-filtering generators discard all tests that fail on the buggy program, follows from the tools' own pipelines and is convincingly documented: 470 and 400 rejected OG-Failed/REF-Success tests for CoverAgent and CoverUp, respectively. The second layer, that final suites 'validate bugs' in 59.6%/68.1% of their tests, requires the Refactory reference implementations to be correct and, more specifically, to define the unique acceptable output for every generated-test input. Section VI explicitly acknowledges only sampled manual verification of these oracles. With only four problem statements in the filtered dataset, this assumption is both load-bearing and under-verified. If some 'OG Success, REF Failed' tests assert acceptable alternative behaviors, those tests are not bug-validating, and the headline percentages overstate the problem. The proposed check using Refactory's correct submissions as alternative oracles would settle this directly. Because the underlying mechanism (rejection of failing tests) is not in dispute, the paper's qualitative conclusion is sound, but the quantitative headline needs this oracle robustness check before full acceptance. CONDITIONAL remains the appropriate verdict, so no change from the reader's assessment.","tokens_in":7295,"tokens_out":15551,"duration_ms":148087,"concrete_test":"For each of the four problem statements, collect the correct student submissions for that assignment from Refactory and run them on the inputs of the 171+62 tests classified 'OG Success, REF Failed.' Any input for which correct implementations produce different outputs is semantically ambiguous; drop those tests and recompute the bug-validating percentages. If adjusted percentages remain above 50%, the oracle concern is resolved; if they fall materially, the headline numbers overstate how often the tools validate bugs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section VI concedes that the evaluation 'assumes that the reference solutions represent correct implementations,' with only a sampled manual verification. The classification 'OG Success, REF Failed' treats every test that passes the buggy program and fails the reference as a test that validates a bug. That inference requires the reference output to be the only valid output for that input. The filtered dataset contains only four problem statements, and intro-Python assignments can admit multiple acceptable behaviors (e.g., tie ordering, whitespace handling, representation choices). If even a small fraction of the 171 CoverAgent and 62 CoverUp tests in this category assert a legitimate alternative output, the headline percentages are inflated. The core qualitative finding survives: by construction, both tools discard every test that fails on the buggy code, so no final suite contains an OG-Failed/REF-Success test. But the quantitative claim that final suites 'validate bugs' in 59.6% and 68.1% of their tests, and the associated 'false sense of security' magnitude, rests on this oracle assumption. The paper's own limitation statement marks this as the least secure load-bearing step.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper empirically evaluates three LLM-based test generation tools (GitHub Copilot, Codium CoverAgent, and CoverUp) on 287 buggy Python programs drawn from the Refactory dataset. It reports that coverage-driven tools that filter failing tests produce final test suites in which a large fraction of tests pass on the buggy implementation but fail on a reference solution (59.6% for CoverAgent and 68.1% for CoverUp), while discarding many tests that would have exposed the bugs (470 and 400, respectively). The authors conclude that the design choices of using code coverage as the primary objective and systematically filtering failing tests can validate bugs and create a false sense of security.","tokens_in":7499,"tokens_out":6096,"duration_ms":54588,"significance":"If the reported magnitudes hold, this is a timely and important finding: it directly challenges the design of popular coverage-guided LLM test generators and gives developers a concrete reason to distrust suites generated against unverified code. The core qualitative insight is robust and supported by construction: because these tools discard any test that fails on the current implementation, their final suites cannot contain tests that fail on that implementation, and the discarded-test counts show that bug-revealing tests are systematically removed. The paper uses a public benchmark and provides a replication package, which supports reproducibility. The main weakness is that the headline percentages treat the Refactory reference solutions as unique oracles, an assumption that the paper itself only partially verifies. The directional claim is sound, but the quantitative strength of the conclusion needs additional support.","major_comments":[{"comment":"The headline percentages (59.6% for CoverAgent, 68.1% for CoverUp) classify every test that passes on the buggy program and fails on the reference solution as 'validating bugs.' This inference requires that the reference output is the only acceptable output for each input. The dataset contains only four problem statements from an introductory Python course, where alternative valid behaviors (e.g., output formatting, tie-breaking order, representation choices) are plausible. Section VI states that only a sample of reference solutions was manually verified, but it does not report the sample size, selection procedure, or which problem statements were covered. Please report these details and either manually inspect all tests in the 'OG Success, REF Failed' category or provide a sensitivity analysis that bounds the reported percentages under alternative oracles.","section":"Section VI and Table I"},{"comment":"The label 'OG Success, REF Failed' conflates two distinct interpretations: (a) a test asserts an output that is wrong according to the problem specification, and (b) a test fails on the reference implementation for reasons unrelated to the target bug, such as the reference containing its own flaw or the input admitting multiple valid outputs. Only interpretation (a) makes the test a genuine bug-validating test. The current analysis does not distinguish these cases, so the claim that final suites 'validate bugs' in 59.6% and 68.1% of their tests is stronger than what the data directly show.","section":"Section IV and Table I"},{"comment":"The conclusion attributes the observed behavior to 'the use of code coverage as a primary objective and the systematic filtering of failing tests,' but the empirical design does not isolate these two factors. CoverAgent and CoverUp both employ coverage targets and failing-test filtering, so the bug-validating rates cannot be causally attributed to either factor individually based on the presented data. An ablation study, or at minimum a careful discussion of the relative contribution of each factor, is needed to support the causal wording in the conclusion.","section":"Section VII"}],"minor_comments":[{"comment":"The phrase 'the test coverage, based only on passing tests, was less than 100%' is unclear; it should specify that coverage is measured on the buggy implementation when executed only with the tests that pass on that implementation, and explain why passing-test-only coverage was chosen as the filter.","section":"Section III-A, step 3"},{"comment":"The bullet list for CoverAgent contains overlapping items: 'Filter out tests that don’t compile or run' and 'Discard tests that don’t pass on the current code.' Merging these into a single filtering step would be easier to follow.","section":"Section III-B2"},{"comment":"The sentence 'in the first 2 cases (with OG Failed), CoverUp and CoverAgent do not generate test suites and reject all tests' is confusing, because the tools do generate suites containing the 'OG Success' tests. Rephrase to indicate that all tests falling into the first two categories are generated but rejected during filtering and therefore do not appear in the final suites.","section":"Section IV"},{"comment":"The asterisk note should explicitly state that the starred counts refer to tests that were generated but rejected during the filtering process, and thus are not part of the final test suites reported in the table.","section":"Table I"},{"comment":"The capitalization of 'CoverUp' is inconsistent (e.g., 'Coverup' appears in Sections II and III-B3); the tool name should be spelled uniformly.","section":"Throughout"},{"comment":"The replication package is mentioned as containing necessary modifications, but no URL or repository identifier is provided; please include the link.","section":"Section VI"},{"comment":"In the find_sum example, showing the correct expected outputs (without the +1 bug) alongside the buggy assertions would make the bug-validating nature of the generated test suite immediately clear to the reader.","section":"Section V"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of a software engineering venue and addresses a question of practical importance. The major concern is the oracle assumption underlying the quantitative headline, which is acknowledged in the threats section but not resolved with evidence. This is fixable with additional manual verification or a sensitivity analysis, so I recommend major revision rather than rejection. I also encourage the authors to add a clearer separation of the coverage objective and the failing-test filtering as causal factors. The self-citation [15] is relevant to the TDD recommendation and does not raise concerns about citation ethics."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper has a real finding: two popular LLM test generators, by design, throw out tests that would expose bugs and keep tests that lock in the buggy behavior. The mechanism is structural—any test that fails on the current code is discarded—so the final suite cannot contain a test that fails on the buggy implementation. The paper demonstrates this with concrete counts and by instrumenting CoverAgent and CoverUp to capture rejected tests. That's worth your time.\n\nWhat's new: prior work showed LLMs are misled by buggy source code. This paper goes further and shows that even when the LLM generates a useful, bug-revealing test, the tool's filtering removes it. The empirical demonstration on a shared benchmark (Refactory) makes the design flaw concrete. The paper also gives nice real-world illustrations from SWE-bench.\n\nWeak spots: the headline percentages (59.6% and 68.1% bug-validating tests) rely on the reference solution being the unique correct output for each input. The authors acknowledge this in Section VI and manually verified a sample, but not all 287 samples across only four problem statements. If some of the tests labeled \"OG Success, REF Failed\" are actually asserting a legitimate alternative behavior (whitespace, tie ordering, etc.), those percentages are inflated. The stress-test note is right to flag this as the least secure load-bearing step. It's a caveat, not a fatal flaw: the core qualitative finding—that these tools cannot produce bug-revealing tests by construction—holds regardless of the oracle. Two minor issues: the tools start from empty test suites, which is not their typical deployment, and there's no statistical analysis of variability across LLM runs, though the effect sizes are large enough that this may not matter. Also, I couldn't find a link to the replication package in the text.\n\nOverall, the central argument is sound. The quantitative magnitude needs an oracle audit—either verify all reference solutions or soften the headline claims. I'd send this to review with that request. It's a solid paper for the SE/testing community and would spark good reading-group discussion.\n\nRecommendation: accept for peer review, conditional on oracle validation.","headline":"Design-level bug in LLM test generators is real and well demonstrated, but the headline percentages rest on an unverified oracle assumption.","tokens_in":8029,"tokens_out":3326,"would_cite":true,"duration_ms":29430,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Filtering out failing tests makes LLM test generators lock in bugs instead of finding them.","keywords":["LLM-based test generation","software testing","test oracle","code coverage","bug detection","coverage-guided testing","test suite reliability"],"falsifier":"Audit all tests classified as 'OG Success, REF Failed' (171 from CoverAgent and 62 from CoverUp) against human-written oracles derived only from the original problem statements; if a substantial share of them confirm the buggy output as correct, the reference-solution assumption fails and the reported bug-validation rates are overstated.","tokens_in":7088,"feed_emoji":"🐛","tokens_out":10620,"duration_ms":85234,"temperature":0.7,"pith_summary":"The paper argues that the design of LLM-based test generators—maximizing code coverage and filtering out tests that fail on the code under test—can make the tools bless bugs instead of catching them. On 287 real human-written buggy Python programs, the two coverage-guided tools (CoverAgent and CoverUp) rejected the tests that failed on the buggy implementation, i.e., the only tests that could reveal bugs, and kept tests that assert the buggy outputs as expected. In their final test suites, 59.6% of CoverAgent's tests and 68.1% of CoverUp's tests pass on the buggy code but fail on the correct reference solution, while a pure-generation baseline produced far fewer such tests. The authors conclude that the design philosophy needs revision: tests should be derived from requirements, and relying on these tools during active development can create a false sense of security.","feed_headline":"Filtering failing tests makes AI testers validate bugs","feed_subtitle":"Up to 68% of their retained tests pass on faulty code and fail on the correct version, hiding defects from developers.","key_machinery":"The central machinery is a coverage-guided generate-and-filter loop. The tool measures which lines or branches are uncovered, prompts an LLM to write tests for those regions, runs the tests against the current implementation, discards every test that fails, and repeats until a coverage target is reached. Because \"fails on the code under test\" is treated as \"bad test,\" the loop deletes the only tests that could expose a bug and keeps assertions that match wrong outputs; coverage then pushes generation toward more of those bug-engrained tests.","core_discovery":"The central discovery is that the bug-validation effect comes from the pipeline design, not from the LLM's ability to write tests. Both coverage-guided tools treat a test that fails on the current implementation as a bad test to discard, and they treat a test that passes and increases coverage as good; the result is that assertions matching the incorrect behavior are retained while potentially bug-revealing tests are thrown away. Evaluated against the reference solution, the retained suites show the damage: 171 of 287 CoverAgent tests (59.6%) and 62 of 91 CoverUp tests (68.1%) pass on the original buggy implementation and fail on the reference, meaning they validate bugs. CoverUp failed to produce any suite at all for 196 of 287 samples, and its successful suites had the highest bug-validation rate. The paper's manual inspection of a few real-world issues from open-source projects showed the same pattern of coverage targets being met while the critical edge case was ignored.","pith_inferences":["If the mechanism is the filter-and-cover loop, then any test generator that uses the same keep-only-passing-tests policy should show the same bias, regardless of which LLM is behind it; a direct test would be to add a keep-failing-tests mode to one of these tools and measure whether mutation scores improve.","The false sense of security may be self-reinforcing: once a bug-validating suite is committed, future code changes are checked against assertions that already encode the bug, so the bug becomes harder to remove without breaking the 'passing' suite.","A testable extension: run the same pipeline on code with seeded faults and compute the mutation score of generated suites; the prediction is that coverage-driven filtering lowers the score, i.e., produces weaker tests.","The critique likely extends to mutation-based LLM test generators that use the same failing-test filter, because they inherit the same assumption that a failing test is a mistaken test."],"forward_implications":["A developer using one of these tools during active development can end up with a green test suite whose assertions encode current bugs, so the suite hides regressions instead of catching them.","The effect is large enough to matter in practice: in the final suites of the two coverage-guided tools, a majority of retained tests validate bugs rather than detect them.","Coverage targets act as an early stopping condition that can leave critical edge cases untested, as the paper's timezone-crash example illustrated.","The paper recommends shifting to requirement-driven test development, using LLMs to help write tests rather than to infer the intended behavior from possibly buggy code."],"supporting_citations":[{"why":"Describes CoverUp, the coverage-guided iterative generator whose final suites are measured; its 91-test retained suite had 68.1% bug-validating tests.","marker":"[2]"},{"why":"Describes CoverAgent (qodo-cover), the tool whose generate-filter-retain design is analyzed; its retained suite had 59.6% bug-validating tests.","marker":"[5]"},{"why":"Provides the dataset of 287 real human-written buggy Python implementations and reference solutions used across the evaluation.","marker":"[11]"},{"why":"Shows that incorrect source code misleads LLM test generation; this paper extends that effect to the pipeline's filtering decisions.","marker":"[10]"},{"why":"Documents the 'assured LLM' approach of keeping only passing, coverage-improving tests, the design principle behind CoverAgent that the paper implicates.","marker":"[12]"},{"why":"Supplies the empirical baseline for GitHub Copilot test generation, used to represent the pure-generation approach.","marker":"[3]"}],"fun_headline_variants":["Discarding failing tests makes AI testers validate bugs","Coverage-driven test generators retain tests that validate bugs","When AI testers discard failures, they hide real bugs","Test pipeline design causes LLM testers to validate defects"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The study assumes that every reference solution is a correct implementation, so a test that passes on the buggy code but fails on the reference is counted as validating a bug; if some reference solutions are wrong or some inputs admit more than one valid output, those percentages would be inflated.","fun_headline_variants_meta":{"raw":{"variants":["Discarding failing tests makes AI testers validate bugs","Coverage-driven test generators retain tests that validate bugs","When AI testers discard failures, they hide real bugs","Test pipeline design causes LLM testers to validate defects"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00028,"raw_usage":{"total_tokens":1632,"prompt_tokens":890,"completion_tokens":742,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":677}},"tokens_in":506,"tokens_out":742,"duration_ms":6117,"temperature":1.0,"reasoning_tokens":677,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:26:10.880457+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Audit all tests classified as 'OG Success, REF Failed' (171 from CoverAgent and 62 from CoverUp) against human-written oracles derived only from the original problem statements; if a substantial share of them confirm the buggy output as correct, the reference-solution assumption fails and the reported bug-validation rates are overstated.","supporting_citations":[{"cited_title":"GitHub - qodo-ai/qodo-cover: Qodo-Cover: An AI-Powered Tool for Automated Test Generation and Code Coverage Enhancement! — github.com,","cited_arxiv_id":null,"evidence_quote":"Describes CoverAgent (qodo-cover), the tool whose generate-filter-retain design is analyzed; its retained suite had 59.6% bug-validating tests."},{"cited_title":"Re- factoring based program repair applied to programming assignments,","cited_arxiv_id":null,"evidence_quote":"Provides the dataset of 287 real human-written buggy Python implementations and reference solutions used across the evaluation."},{"cited_title":"Automated unit test improvement using large language models at meta,","cited_arxiv_id":null,"evidence_quote":"Documents the 'assured LLM' approach of keeping only passing, coverage-improving tests, the design principle behind CoverAgent that the paper implicates."},{"cited_title":"Using github copilot for test generation in python: An empirical study,","cited_arxiv_id":null,"evidence_quote":"Supplies the empirical baseline for GitHub Copilot test generation, used to represent the pure-generation approach."}],"review_version":1}