{"id":"2ce325d0-c5a9-4d68-a466-b07407b15355","arxiv_id":"2607.22883","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Buggy code in prompts steers LLMs to write tests that validate the bug, and generating tests from an LLM-written spec docstring instead of the code reduces such misguided tests and increases bug-finding ones.","lead":"The paper measures how much buggy code misleads large language models when they write unit tests, and shows that replacing the buggy code with an AI-written specification docstring produces more bug-finding tests. The finding gives test engineers a simple prompt change that improves AI-generated tests without needing the fixed code.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The core 'misguided test' metric assumes the fixed Defects4J revision is the unique oracle (Section 2.6); tests passing on buggy but failing on fixed may include legitimate assertions unrelated to the bug, so the headline effect could be overstated.","rationale":"This is the most load-bearing concern because the metric definition is the foundation of both research questions. The RQ1 claim (buggy code increases misguided tests and suppresses effective tests) and the RQ2 claim (docstring replacement reduces misguided and increases effective tests) are both counts of tests whose labels are defined relative to the fixed version's behavior. If the fixed version is not the unique intended behavior, some labeled 'misguided' tests are actually correct tests of unspecified or alternative behavior, and the measured misguidance effect would be an artifact of the patch's behavioral delta rather than evidence that the LLM is misguided. The paper itself shows awareness of oracle sensitivity when critiquing Huang et al. (Section 2.6, Table 3), so validating its own oracle is essential. The paired buggy/fixed design, the consistently large effect across 13 configurations, the model-internal sequence-score evidence, and the manual docstring inspection (Section 4.4) all provide real support, but those lines of evidence do not remove the need to check the labels themselves. The lack of confidence intervals is a secondary weakness, and the benchmark selection criterion (Section 2.2) may inflate effect sizes, but neither is as fundamental as the oracle. I therefore agree with the reader that CONDITIONAL is appropriate, pending an oracle validation check.","tokens_in":28384,"tokens_out":6590,"duration_ms":58101,"concrete_test":"Randomly sample 100 'misguided' tests from the buggy-condition runs. Two Java experts, blind to the paper's labels, independently classify each as (a) asserting the exact behavior the patch fixes, (b) asserting a buggy/fixed behavioral difference unrelated to the fix, or (c) asserting plausibly correct or unspecified behavior. Compute Cohen's kappa and the fraction in (b)+(c). If that fraction exceeds about 15%, the metric is materially contaminated and the headline effect sizes need re-estimation; if below 5%, the oracle assumption is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claims rest on the classification in Table 2: a 'misguided' test is one that passes on the buggy version and fails on the fixed version. Section 2.6 states this assumption explicitly: 'we treat the fixed Defects4J version as the gold standard for intended program behavior.' This oracle is load-bearing because both the RQ1 misguidance effect and the RQ2 mitigation gains are measured as counts of tests in this category. If a Defects4J patch includes changes beyond the minimal bug fix—refactoring, altered edge-case semantics, or new behavior—then a test that passes on the buggy version and fails on the fixed version need not assert the buggy behavior; it may assert a behavior that is merely different from the patch's final state. The motivating Lang-14 example (StringUtils.equals) is clean because the fixed implementation clearly matches the method's intent, but the paper does not check whether all 318 focal methods have such unambiguously intent-matching patches. The authors manually inspect docstring quality (Section 4.4) but never manually validate the test labels that drive the headline numbers. The benchmark filter in Section 2.2 (only methods with an existing human-written test that triggers the bug) could make buggy-fixed behavioral differences more salient than in the general population, which would inflate effect sizes, but this is secondary; the oracle definition is a correctness issue, not just a generalizability one.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a new metric for quantifying the 'misguidance effect' of buggy code on LLM-generated unit tests: a test is 'misguided' if it passes on the buggy version of a focal method and fails on its fixed version. Using 318 focal methods from Defects4J and 11 LLMs (13 configurations), the authors report that prompting with buggy code increases misguided tests (average 137.69 vs 16.46 for fixed code) and suppresses effective tests (104.15 vs 304.08). They then propose a specification-based approach that replaces the code under test with an LLM-generated docstring, reporting a reduction in misguided tests to 113.00 and an increase in effective tests to 186.77. The paper also analyzes sequence scores, multi-round feedback, and manual docstring annotations, and evaluates the approach on bug-free code.","tokens_in":28652,"tokens_out":9791,"duration_ms":78127,"significance":"If the results hold, this is a valuable contribution to LLM-based test generation: it sharpens the definition of misguidance relative to prior work (Huang et al.), provides a practical mitigation strategy, and ships a public replication package with manual annotations. The study's strengths include a large and diverse model set, a consistent experimental pipeline, cross-scoring sequence-score analysis, and good inter-annotator agreement. The main concerns are the unvalidated oracle assumption, the biased benchmark filter, and the lack of inferential statistics for the headline comparisons.","major_comments":[{"comment":"The classification in Table 2 labels a test as 'misguided' if it passes on the buggy version and fails on the fixed version, assuming the fixed Defects4J version is the unique gold standard for intended behavior. Because Defects4J patches can include changes beyond the minimal bug fix, a test in this category may assert behavior that merely differs from the post-patch state rather than the buggy behavior itself, which would overstate the misguidance effect. This is load-bearing for the central claims of RQ1 (Tables 4 and 5) and RQ2 (Tables 6 and 8). Please manually or automatically validate a sample of 'misguided' tests against the patch diff to estimate the rate of non-bug-related behavioral changes, and report how the reported effect sizes change if those tests are excluded.","section":"Section 2.6, Table 2"},{"comment":"The benchmark filter restricts the focal methods to those that trigger at least one existing human-written test in Defects4J, which ensures the patch is a bug fix but also limits the study to bugs that are already detectable by the provided test suite. Such bugs may have more clearly identifiable buggy behavior, potentially inflating the measured misguidance effect relative to the general population of defects. The paper does not discuss this selection bias in Section 6. Please add either a sensitivity analysis on a sample of unfiltered methods or a discussion of how the filter affects the interpretation and generalizability of the effect sizes.","section":"Section 2.2, criterion (3)"},{"comment":"The paper repeatedly uses 'significantly' and 'substantially' for the main comparisons (e.g., Section 4.1 states that the approach 'significantly improves test quality'), but no p-values, confidence intervals, or effect sizes are reported for these contrasts; the only inferential statistics are the Pearson correlations in Section 3.1. Since the results are based on 13 model configurations, paired tests (e.g., Wilcoxon signed-rank) and effect sizes are feasible. Please report these for the key comparisons in Tables 4, 6, and 8.","section":"Sections 3.1, 4.1, and 4.2"}],"minor_comments":[{"comment":"The paper does not discuss flaky tests, which could affect the pass/fail labels in Table 2 and therefore the classification of tests; please state how flakiness was handled or mitigated.","section":"Section 6"},{"comment":"The multi-round setup is described as following ChatTester, but the exact number of rounds, the feedback format, and the stopping criterion are not specified; please provide these details for reproducibility.","section":"Section 4.3"},{"comment":"The paper should define the denominators for CFR and FAR (e.g., percentage of all generated tests versus percentage of compiled tests) explicitly in the text.","section":"Table 12"},{"comment":"The manual inspection selects the two models with the largest and smallest reductions in misguided test suites, which is a selection for extremes that may overstate the relationship between docstring quality and test outcomes; please clarify the implications or consider a random sample.","section":"Section 4.4"},{"comment":"The sequence score uses log probabilities but does not specify the base of the logarithm; please state the base for reproducibility.","section":"Equation (1)"}],"recommendation":"major_revision","confidential_remarks":"The paper is a good fit for ISSTA and includes a replication package, which is commendable. The main risks to acceptance are the unvalidated oracle assumption and the missing inferential statistics on the headline comparisons; both are addressable with additional analyses. The benchmark filter bias, if not addressed, could also be seen as a generalizability threat by reviewers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Zhao, Zhou, and Cohen have done a careful, useful study. The new metric—misguided tests defined as pass-on-buggy/fail-on-fixed—is a real refinement over Huang et al.'s aggregate pass-on-buggy count, and the finding that over 90% of buggy-passing tests are true negatives is a clean rebuttal. The core empirical claim holds: buggy code roughly halves effective tests (104 vs. 304 per model on average) and increases misguided tests (138 vs. 16), and replacing the code with an LLM-generated spec docstring flips the trend (113 misguided, 187 effective). The effect is consistent across 12 of 13 configurations, the multi-round experiment is useful, and the manual docstring inspection has good inter-annotator agreement.\n\nSoft spots: the paper says \"significantly\" without reporting tests or confidence intervals for the main comparisons. One model, Qwen3-Coder-Plus, slightly increases misguided tests under the docstring approach (80 to 91), so the mitigation is not universal. The benchmark filter (only focal methods with a triggering human test) may overstate effect sizes, and the fixed-version-as-gold-standard oracle is an assumption worth stating more carefully—though for Defects4J it is a reasonable one, and the stress-test worry about non-minimal patches is not supported by evidence in the paper. The sequence-score corroboration is suggestive but the differences are small and untested. Fix the Zenodo DOI inconsistency (10.5281/zenodo.21428153 vs .21428156).\n\nThe paper is a solid empirical contribution, not a breakthrough. It deserves a serious referee and would benefit from a revision that tightens the statistics and discusses the oracle assumption. I would bring it to a reading group and would cite the metric.","headline":"A solid empirical study with a genuinely better metric for the misguidance effect; the main results hold, but the statistics need tightening before acceptance.","tokens_in":29181,"tokens_out":2375,"would_cite":true,"duration_ms":21417,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Prompting LLMs with buggy code measurably steers generated unit tests toward validating the bug, while replacing the code with an LLM-generated specification docstring cuts 'misguided tests' and nearly doubles bug-finding tests.","keywords":["LLM unit test generation","misguidance effect","specification-based testing","buggy code prompting","misguided tests","effective tests","bug detection effectiveness"],"falsifier":"Sample the tests the metric labels 'misguided' and have independent human judges decide, without seeing the fixed code, whether each assertion contradicts the method's documented or intended behavior; if a substantial fraction are judged acceptable, the measured misguidance effect would shrink or disappear.","tokens_in":28183,"feed_emoji":"🐛","tokens_out":9690,"duration_ms":74248,"temperature":0.7,"pith_summary":"This paper claims that showing an LLM buggy code has a measurable and harmful 'misguidance effect' on the unit tests it writes: the model starts treating the bug as intended behavior. The authors quantify the effect with a new metric that runs every generated test against both the buggy code and its fixed counterpart; tests that pass on the buggy version but fail on the fixed one are 'misguided,' and tests that fail on the buggy version but pass on the fixed one are 'effective' at finding the bug. Across 11 large language models and hundreds of real Java methods, buggy input produced on average 137.69 misguided tests and only 104.15 effective tests, and model-internal probability scores confirm that the buggy code biases the model's preferences. The paper's proposed fix is to replace the code in the prompt with an LLM-generated specification docstring, which lowers misguided tests to 113.00 and raises effective tests to 186.77. If the effect is real, LLM-based test generation should be built around recovered specifications rather than the code under test.","feed_headline":"Buggy code steers LLM tests toward the bug","feed_subtitle":"Replacing the code with an LLM-generated docstring nearly doubles the count of bug-finding tests.","key_machinery":"The mechanism is a dual-execution test classifier: every generated test is compiled and run against both the buggy version and its fixed counterpart, and the four outcome combinations define the labels (true negative, effective, misguided, false positive). This classifier is what makes the misguidance effect measurable rather than anecdotal. The mitigation machinery is a two-step specification pipeline: the model first writes a docstring that describes intended behavior while avoiding code quotation, optionally after an explicit audit for logical mistakes and robustness gaps, and then that docstring replaces the code entirely in the test-generation prompt. The paper's ablations show that both halves are necessary: removing the code without providing a spec loses too many effective tests, and adding the docstring without removing the code barely helps.","core_discovery":"The paper's central discovery is that the same LLM produces a very different test suite depending on whether the prompt contains buggy code or its fixed counterpart: relative to fixed-code prompting, buggy-code prompting increases tests that assert the buggy behavior and suppresses tests that expose it, averaging 137.69 misguided versus 104.15 effective tests per model, compared with 16.46 and 304.08 for fixed-code input. This pattern is not a measurement artifact of simply counting tests that pass on buggy code, since over 90% of such tests pass on the fixed version too; the new 'misguided test' definition isolates the minority that actively assert the bug. The paper then shows that a two-step specification-based prompt, generate a behavioral docstring from the buggy code and then feed only that docstring to the test-writing model, reduces misguided tests to 113.00 and increases effective tests to 186.77, with a stronger analysis-driven docstring prompt improving both further. Sequence-score evidence shows that conditioned on buggy code, the model assigns higher likelihood to misguided tests, and conditioned on fixed code, to effective tests, so the behavioral change tracks the model's internal preference.","pith_inferences":["Beyond the paper: the same spec-replacement design could be applied to other bug-inducing feedback loops, such as LLM-based program repair and code generation with test feedback, where erroneous code repeatedly re-enters the prompt.","Beyond the paper: the paper's oracle assumption can be stress-tested by re-labelling a sample of 'misguided' tests with human judges; if many are acceptable alternative behaviours, the reported effect sizes would shrink.","Beyond the paper: because the advanced docstring prompt slightly raises hallucinated-behaviour tests, roughly from 16% to 18%, investing in specification quality, for example by cross-checking generated specs against execution traces, should be a high-leverage next step.","Beyond the paper: a direct comparison against human-written specifications would bracket the upper bound of the mitigation, since the paper shows only what LLM-recovered specifications can achieve."],"forward_implications":["Existing benchmarks that feed only bug-free code to LLMs likely overestimate real bug-detection ability, because the same models produce roughly three times fewer effective tests when the code is buggy.","Practical test-generation tools should treat the code under test as suspect and derive tests from a separately recovered specification, and the code must be removed entirely rather than supplemented with documentation.","Models with stronger code comprehension are also more susceptible to misguidance, so choosing a more capable model does not by itself mitigate the effect.","Multi-round feedback-driven test generation inherits and accumulates the misguidance effect, while starting from a specification docstring keeps misguided-test growth lower across refinement rounds.","The mitigation adds only one extra LLM call and, on bug-free code, keeps compilation-failure, false-alarm, and coverage metrics comparable to code-based prompting."],"supporting_citations":[{"why":"Supplies the paired buggy/fixed programs and human-written tests that define the benchmark, the gold-standard oracle, and the focal-method selection.","marker":"[34]"},{"why":"Defines the prompt-construction, context, and test-extraction workflow that all test-generation runs adapt.","marker":"[73]"},{"why":"Provides the multi-round feedback pipeline and the docstring-with-code supplementary baseline that the proposed method must outperform.","marker":"[77]"},{"why":"Introduces the earlier aggregate misguidance metric that the paper shows is inflated by true-negative tests.","marker":"[31]"},{"why":"Prior observation that buggy code can prevent detection of its own bug, motivating the quantitative evaluation.","marker":"[43]"},{"why":"Defines the sequence score used to measure model-internal preference for misguided versus effective tests.","marker":"[11]"},{"why":"Another multi-round test-generation pipeline used to test the misguidance effect in iterative settings.","marker":"[15]"}],"fun_headline_variants":["Buggy code steers LLM tests to assert the bug","Spec docstrings fix LLM tests that chase bugs","Swap buggy code for a docstring in LLM test prompts","Docstring prompting halves misguided LLM unit tests","LLM tests with buggy code validate the bug instead of finding it"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the fixed version uniquely defines intended behavior, so a test that passes on the buggy version and fails on the fixed version is automatically 'misguided'; if some such tests assert acceptable alternative behaviors, the effect sizes and the reported improvements are overstated.","fun_headline_variants_meta":{"raw":{"variants":["Buggy code steers LLM tests to assert the bug","Spec docstrings fix LLM tests that chase bugs","Swap buggy code for a docstring in LLM test prompts","Docstring prompting halves misguided LLM unit tests","LLM tests with buggy code validate the bug instead of finding it"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000245,"raw_usage":{"total_tokens":1572,"prompt_tokens":1016,"completion_tokens":556,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":632,"completion_tokens_details":{"reasoning_tokens":472}},"tokens_in":632,"tokens_out":556,"duration_ms":4735,"temperature":1.0,"reasoning_tokens":472,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:27:23.633054+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Sample the tests the metric labels 'misguided' and have independent human judges decide, without seeing the fixed code, whether each assertion contradicts the method's documented or intended behavior; if a substantial fraction are judged acceptable, the measured misguidance effect would shrink or disappear.","supporting_citations":[],"review_version":2}