{"id":"6cbc495f-7b96-4570-92ff-3c930d673bb3","arxiv_id":"2607.09345","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.5,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"After removing fix-commit shortcuts and enforcing project-disjoint evaluation, CodeBERT flakiness detectors collapse to majority baselines on developer-confirmed flaky tests with rerun-confirmed non-flaky labels.","lead":"Code-based flaky-test detectors that report high F1 scores mostly exploit benchmark shortcuts and evaluation protocols, not flakiness signals in the test source. The work shows the field should reframe the problem around classifying observed failures and environment-conditioned failure risk.","discovery_kind":"replication","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified that overturns the central claim.","rationale":"The strongest claim is tightly scoped: after removing the fix-commit shortcut and holding projects out, the reimplementations do not beat constant baselines on developer-confirmed flakiness with rerun non-flaky labels, and the published high scores rested on the shortcut plus the CV protocol. That claim is directly supported by the counterfactual square (Table VII), the leakage-fixed reproductions, and the FlakeBench contrast under identical conventions. The 500-run limitation is correctly flagged by the reader and by the authors (Section VI), but it cannot manufacture the observed near-zero recall on the positive class. Therefore no adjustment to ACCEPT / HIGH is warranted. The concrete multi-environment re-labeling check is still worth running as a robustness verification, not as a condition for acceptance.","tokens_in":24968,"tokens_out":530,"duration_ms":12023,"concrete_test":"Re-evaluate the two CodeBERT variants on C-IDoFT under the same project-disjoint folds after re-labeling the non-flaky class with a multi-environment campaign (e.g., 100 runs each under two OS images and two resource-pressure settings). If flaky-class recall on the developer-confirmed positives rises above ~0.3 and F1 meaningfully exceeds the always-flaky baseline, the oracle concern would land; if recall stays near zero, the central claim is reinforced.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader's weakest assumption (500-run single-config non-flaky oracle) is real but not load-bearing for the strongest claim. The claim rests on near-zero flaky-class recall over developer-confirmed flaky tests under project-disjoint evaluation (Table V: Flakify+ 0.021, FlakyLens+ 0.120 vs always-flaky F1 0.054). Mislabeled rare non-flaky tests can only create false positives and thus depress precision; they do not enter the recall denominator. The paper already separates the protocol effect on identical data (CV flaky-class F1 0.746/0.686 vs Disjoint 0.035/0.070) and shows success on FlakeBench under the same protocol, so the collapse is not an artifact of an under-powered non-flaky oracle. Residual threats (four folds, single annotator on FlakeCI) are ordinary and do not reverse the direction of the results.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper argues that code-based flaky-test detection is often ill-posed because flakiness is not a static property of test source code. It reproduces three published LLM/CodeBERT detectors (Flakify, FlakyQ, FlakyXBert), shows that reported high F1 scores on IDoFT rest on data leakage, fix-commit non-flaky labeling, and non-project-disjoint evaluation, and constructs C-IDoFT as a controlled counterfactual that keeps developer-confirmed IDoFT flaky tests while rebuilding non-flaky labels from 500 reruns. Under project-disjoint evaluation on C-IDoFT, CodeBERT reimplementations collapse to constant majority baselines (flaky-class F1 0.035/0.070 vs always-flaky 0.054), while the same models recover nearly all flaky tests on FlakeBench. A second corpus, FlakeCI (86 same-commit flaky E2E tests), finds that test code plus CI logs attribute a cause for only 42% of cases. The authors reframe the task toward failure-level classification and environment-conditioned failure probability, and release datasets and a replication package.","tokens_in":25161,"tokens_out":1355,"duration_ms":28144,"significance":"If the results hold, the paper substantially revises how the SE community should evaluate and pursue flaky-test prediction: much of the recent LLM-based progress on standard benchmarks is an artifact of labeling shortcuts and evaluation protocol rather than a generalizable code signal. Strengths include a carefully designed counterfactual (C-IDoFT holds the flaky population fixed while changing only non-flaky construction), an explicit counterfactual square separating labeling from protocol (Table VII), project-disjoint evaluation with constant baselines, reproduction of three published detectors including leakage fixes, and a released replication package with curation pipelines. The FlakeBench contrast and FlakeCI diagnosis study usefully bound when code is and is not informative. This is a high-value critical empirical contribution for flaky-test research and for SE-ML evaluation practice more broadly.","major_comments":[{"comment":"Table V and §IV-B: C-IDoFT project-disjoint results rest on only four folds, and the flaky-class F1 standard deviations (0.035±0.058; 0.070±0.078) are comparable to or larger than the means. The direction of collapse relative to the always-flaky baseline (0.054) and the CV-vs-Disjoint contrast on identical data are still clear, but the paper should either increase the number of project-disjoint folds (or use leave-one-project-out where feasible) or state the claim more carefully as a consistent near-baseline collapse rather than a precisely estimated F1. This is the main quantitative support for the central negative result and deserves tighter uncertainty reporting.","section":"§IV-B, Table V"},{"comment":"§IV-C and Table VI: The 42%/58% attribution split on FlakeCI is used to support the reframe away from static code. Root-cause labels were assigned by a single author over 86 tests, with category concentration (15 of 23 Network cases from two projects). The threats section correctly treats 42% as a lower bound, but if this split remains a headline result, the manuscript should add at least a second independent coder on a substantial subset and report agreement (or restrict the claim more tightly to “in our manual analysis, evidence was insufficient without speculation”). This is secondary to the C-IDoFT collapse but load-bearing for the diagnosis/reframe half of the contribution.","section":"§IV-C, Table VI"}],"minor_comments":[{"comment":"Footnote 1 and Table I: The IDoFT label-count discrepancies (3813 vs 3862; 3195/618 vs 3226/587) are carefully documented but still easy to miss. A short explicit “label inventory” table in the appendix would help readers reconcile paper, package, and your filtered subsets.","section":"§II-B, Footnote 1, Table I"},{"comment":"§III RQ1 metrics: The always-flaky flaky-class F1 formula F1_f = 2p/(1+p) is useful; stating it once in a numbered equation would make later baseline comparisons easier to audit.","section":"§III"},{"comment":"Table VII is excellent for the counterfactual square; consider also reporting flaky-class F1 (not only weighted F1) in every cell so readers do not have to cross-reference Tables II–V.","section":"§V, Table VII"},{"comment":"§IV-A FlakyXBert: The four notebooks shipping another project’s outputs are a serious artifact issue; the corrected 0.83 figure is clear, but a one-line note on whether authors updated the public package after notification would help reproducibility.","section":"§IV-A, Table III"},{"comment":"Collection windows “between January and April 2026” (§IV-C) will look odd relative to a 2026 arXiv stamp depending on final publication date; confirm consistency of dates in the camera-ready text.","section":"§IV-C"},{"comment":"Minor prose: “C-IDoFT is a controlled counterfactual, not a benchmark for reuse” is stated well in the abstract and §II-D; repeating that caveat once more near Table V would reduce the risk of others adopting it as a leaderboard set.","section":"§IV-B"}],"recommendation":"minor_revision","confidential_remarks":"This is a strong critical paper that will be uncomfortable for authors of the reproduced detectors, but the methodology is careful and the central negative result is well supported. I would not treat community pushback as a scientific reason to delay. Fit for a top SE journal is good: evaluation validity plus a constructive reframe. The two major comments are fixable without new theory; if the authors strengthen fold uncertainty and FlakeCI annotation, accept is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The punchline is clean: once you keep the same developer-confirmed flaky tests and rebuild only the non-flaky class from reruns, and once you hold projects out, the CodeBERT reimplementations of Flakify and FlakyLens fall to constant baselines (flaky-class F1 0.035/0.070 vs always-flaky 0.054). The high published numbers lived in the fix-commit shortcut and the non-disjoint protocol. That is the result that matters.\n\nWhat is new is the controlled counterfactual itself. C-IDoFT holds the flaky population fixed while changing only the non-flaky construction, so the performance drop can be attributed to labeling rather than to a wholly different dataset. The counterfactual square (Table VII) separates labeling from protocol on identical conventions; the protocol change alone moves flaky-class F1 from ~0.75 under CV to near zero under project-disjoint folds. Recovery on FlakeBench under the same protocol shows the models can still separate lexically obvious categories, which strengthens rather than weakens the argument. FlakeCI’s 42/58 split is a useful secondary observation about where diagnosis evidence actually sits for E2E failures. The leakage fixes and the notebook-output mismatches on FlakyXBert are carefully documented. Artifact and baselines are present.\n\nSoft spots are ordinary and the authors flag them. Four project-disjoint folds are thin for variance claims. The 500-run single-config non-flaky oracle can mislabel rare environment-dependent failures, but that only creates false positives and does not enter the near-zero recall over developer-confirmed flaky tests that carries the main claim. FlakeCI is small, single-annotator, and concentrated; it supports the diagnosis question, not a training benchmark. None of this reverses the direction of the results.\n\nThis is for anyone working on flakiness detection, SE-ML evaluation hygiene, or industrial CI reliability. It deserves a serious referee. I would engage with it and cite the counterfactual and the protocol separation.","headline":"Controlled counterfactual shows published code-based flaky-test F1 scores rest on labeling shortcuts and CV protocol, not a generalizable signal in test code; collapse to baseline is real and well-isolated.","tokens_in":25854,"tokens_out":515,"would_cite":true,"duration_ms":7068,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Code-based flaky-test detectors collapse to majority baselines once labeling shortcuts and project overlap are removed.","keywords":["flaky tests","code-based detection","benchmark shortcuts","project-disjoint evaluation","CI log mining","failure-level classification","CodeBERT"],"falsifier":"A project-disjoint evaluation on a larger set of developer-confirmed flaky tests whose non-flaky counterparts survive deep multi-environment reruns in which a code-only model clearly and stably exceeds the always-majority baseline on flaky-class F1.","tokens_in":25822,"feed_emoji":"🧪","tokens_out":857,"duration_ms":8708,"temperature":0.7,"pith_summary":"The paper argues that asking whether a test is flaky from its source code alone is often the wrong problem: flakiness typically arises at runtime from interaction with the system and environment, not from static syntax. Reproducing three published code-model detectors, the authors show that high F1 scores on the popular IDoFT benchmark rest on two artifacts—a fix-commit labeling shortcut that pairs nearly identical flaky and fixed versions, and evaluation protocols that leak information across folds or projects. They rebuild only the non-flaky class of that benchmark from 500 repeated executions (C-IDoFT) while keeping developer-confirmed flaky tests; under project-disjoint evaluation the same models fall to constant majority baselines. On a curated subset of flakiness types that are visible in code they still succeed, and a CI-mined set of 86 end-to-end flaky tests shows that test code plus logs explain only 42% of causes. The practical upshot is a reframing: classify whether an observed failure is flaky, or estimate how likely a test is to fail under its actual execution environment.","feed_headline":"Code-only flaky-test detectors fall to chance once shortcuts go","feed_subtitle":"High F1 scores rested on near-duplicate labels and project overlap, not flakiness signal in the source.","key_machinery":"C-IDoFT, a controlled counterfactual that keeps developer-confirmed IDoFT flaky tests and rebuilds only the non-flaky class from 500 module-level reruns, evaluated under project-disjoint folds against majority-class baselines.","core_discovery":"Once the fix-commit labeling shortcut is removed and projects are held out, CodeBERT reimplementations of published code-based detectors do not meaningfully exceed constant majority baselines on developer-confirmed flaky tests with rerun-confirmed non-flaky labels; the previously reported high F1 scores rested on that labeling shortcut and the cross-validation protocol, not on a generalizable signal in the test code.","pith_inferences":["The same critique—task ill-posedness plus labeling shortcuts—likely applies to other software-engineering ML tasks that pair near-duplicate fixed/unfixed code fragments.","Industrial CI systems already hold the decisive signal (pass/fail histories and logs); the bottleneck is labeling genuine versus flaky failures at scale rather than inventing better static models.","Language-model agents that actively perturb the environment (latency, clock, resource pressure) to test a hypothesized cause would close the loop that pure static detection lacks."],"forward_implications":["Detector papers should report flaky-class precision/recall/F1 under project-disjoint splits and against majority baselines; matching the baseline is not detection.","Non-flaky labels should come from repeated execution or CI history, not from later fixed versions of the same tests.","Benchmarks restricted to code-visible flakiness types should be reported as such, not as general flaky-test detection.","Progress is more likely by classifying individual observed failures from logs and runtime evidence, or by estimating environment-conditioned failure probability.","Static code models remain useful only for the narrow subset of flakiness that is lexically obvious in the test method."],"fun_headline_variants":["Code-based flaky detectors drop to chance without label shortcuts","Project holdout wipes out claimed gains for code-only flaky detectors","Flaky-test detectors beat baselines only via near-duplicate labels","Code signal alone fails for flakiness once shortcuts and overlap vanish","Reported flaky-detector F1 rested on protocol not generalizable code"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"That 500 repeated runs on a single CI configuration are enough to label a test non-flaky for the counterfactual; rare environment-dependent failures can still be mislabeled.","fun_headline_variants_meta":{"raw":{"variants":["Code-based flaky detectors drop to chance without label shortcuts","Project holdout wipes out claimed gains for code-only flaky detectors","Flaky-test detectors beat baselines only via near-duplicate labels","Code signal alone fails for flakiness once shortcuts and overlap vanish","Reported flaky-detector F1 rested on protocol not generalizable code"]},"model":"grok-4.5","effort":"low","cost_usd":0.004044,"raw_usage":{"total_tokens":1326,"prompt_tokens":930,"num_sources_used":0,"completion_tokens":95,"cost_in_usd_ticks":40440000,"prompt_tokens_details":{"text_tokens":930,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":301,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":930,"tokens_out":95,"duration_ms":4112,"temperature":1.0,"reasoning_tokens":301,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-13T03:42:59.360768+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"A project-disjoint evaluation on a larger set of developer-confirmed flaky tests whose non-flaky counterparts survive deep multi-environment reruns in which a code-only model clearly and stably exceeds the always-majority baseline on flaky-class F1.","supporting_citations":[],"review_version":1}