{"id":"1d044d9a-24f2-4eb4-9f3f-e457e330e259","arxiv_id":"2502.01825","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A flaky-test classifier scores 8% higher on augmented copies of its training data than on independent test cases, but the comparison conflates augmentation with train-test overlap.","lead":"This paper tests whether using computer-generated variations of training examples in test sets inflates a machine learning model's apparent accuracy. It reports an 8% gap between performance on such variations and on genuinely new examples, but the experimental setup makes that gap almost inevitable.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 8% gap in Experiment 2 conflates augmentation with train-test overlap, so the central claim of augmentation-induced bias is not established.","rationale":"The reader's rejection is based on the same load-bearing concern: the comparison in Experiment 2 cannot separate the augmentation effect from the effect of testing on variants of training data. My stress-test confirms this. The paper's own description of Testing Set 2 as 'augmented versions (v1 and v2) of training cases' makes the confound explicit. A model trained on v0 originals will naturally perform better on close mutations of those exact inputs, especially for mutated identifiers and constants that preserve the original flakiness label; this is a memorization/overlap effect, not necessarily an augmentation artifact. The proposed extra test set directly controls for this by evaluating augmented versions of held-out originals, which are unseen during training. If the gap disappears, the paper's central empirical claim collapses to a statement about test-set leakage, which is a much weaker and more obvious point. If the gap persists, the claim gains real support. Because the current text presents the confounded result as 'systematic bias' and draws the main practical guidelines from it, the central claim remains unsupported as written. The paper does provide code and a reproducible setup, which is valuable, but reproducibility does not cure the design confound. This does not require changing the reader's REJECT verdict; it reinforces it.","tokens_in":5402,"tokens_out":3144,"duration_ms":32882,"concrete_test":"Reproduce Experiment 2 with an additional Testing Set 3: take the same held-out originals used in Testing Set 1 (Async 26, UC 7, Conc 9, Time 8, TOD 24) and form Testing Set 3 from their v1 and v2 augmented variants, preserving per-category counts. Train exactly as in Table II, then compare F1 on Testing Set 1, Testing Set 2, and Testing Set 3. If F1(Set3) is statistically indistinguishable from F1(Set1), the 8% gap is caused by train-test overlap, not augmentation. If F1(Set3) is approximately F1(Set2), then augmentation independently explains the gap. Use paired bootstrap or McNemar's test across the same originals to assess significance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim rests on Experiment 2 (Section III-D, Table III, Figure 2), where the model is trained on original v0 cases and then compared on Testing Set 1 (v0 of held-out originals) versus Testing Set 2 (v1/v2 of training cases). These two test sets differ in two variables simultaneously: whether the test samples are augmented, and whether they are copies/variants of training samples. Any classifier with even mild memorization of training inputs will tend to score higher on mutated copies of those inputs than on genuinely unseen originals. Thus the observed average F1 gap of +8% is fully explainable by train-test overlap alone; it does not isolate a bias caused by the augmentation process. The TOD category's negative difference (-5%) further shows the gap is not uniform, but it does not resolve the confound. The paper's practical recommendation to keep separate non-augmented validation sets is sensible, but the measurement offered as evidence does not support the stated conclusion that augmentation itself introduces systematic bias. This is a design confound rather than an internal inconsistency: the claim could be true, but this experiment cannot decide it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses whether data augmentation introduces bias in machine learning models for software engineering tasks, using flaky test classification as a case study. The authors use the FlakyCat dataset, where each original test (v0) has two augmented versions (v1, v2), and run two experiments. Experiment 1 compares a model trained on original cases only with a model trained on original and augmented cases. Experiment 2 trains only on original cases and evaluates on two test sets: original versions of unseen cases and augmented versions of training cases. The paper reports an average F1-score gap of 8% between these two test sets and interprets this as augmentation-induced bias, recommending that practitioners keep separate non-augmented validation sets. The code is made available in a GitHub repository.","tokens_in":5584,"tokens_out":3280,"duration_ms":35203,"significance":"The question is timely and important: if data augmentation in training and testing sets inflates reported performance, many published evaluations in software engineering could be overly optimistic. The paper has clear strengths: it uses a third-party augmented dataset (reducing self-confirmatory bias), it targets a concrete SE task, and it makes a practical, falsifiable recommendation about validation set design. However, the two experiments as designed do not support the central claim. Experiment 1 confounds augmentation with training-set size, and Experiment 2 confounds augmentation with train-test overlap. The reported 8% gap is therefore expected even in the absence of any augmentation-specific bias. The manuscript would need a substantial redesign of its experiments before the central conclusion can be evaluated.","major_comments":[{"comment":"The design description is misleading: Testing Set 2 is called 'augmented training data' in the experiment title, but the results section says the model is evaluated on 'entirely new test cases.' Testing Set 2 is not new; it consists of augmented versions of the training cases. This wording obscures the train-test overlap confound and should be corrected if the experiment is redesigned and resubmitted.","section":"Section III-D"}],"minor_comments":[{"comment":"Reference [1] is given as 'arXiv preprint:2401.15422' but should follow the standard arXiv citation format. Please also check that all references are complete and consistent.","section":"References"}],"recommendation":"reject","confidential_remarks":"The practical recommendation to maintain separate non-augmented validation sets is sensible and likely worth stating as a methodological caution. However, the experimental evidence presented does not support the paper's central claim of augmentation-induced bias, because both experiments contain a design confound that fully explains the observed differences. This is not a matter of missing details or presentation; the reported comparisons cannot be salvaged without new experiments that match training-set size and eliminate train-test overlap. I recommend rejection, though I would encourage the authors to consider a resubmission with a properly controlled design."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague —\n\nQuick take: the paper reports an 8% F1 gap between augmented training variants and unseen originals, but the experiment that produces it compares unseen originals against augmented copies of training cases. That gap is exactly what you'd expect from train-test overlap alone. So the central claim—that augmentation itself inflates test performance—is not supported by the measurement.\n\nWhat's genuinely useful: the paper is a concrete case study on FlakyCat with code on GitHub, and the authors did the right thing by using a third-party augmented dataset to avoid self-confirmatory bias. The practical advice—hold out a non-augmented validation set and label augmented data—is sensible and worth stating. The split tables are clear.\n\nThe soft spots are load-bearing. Experiment 2's Testing Set 2 is built from v1/v2 versions of training cases. Any classifier with mild memorization will score higher on mutated copies of inputs it trained on than on genuinely new originals. The paper even describes the design as comparing 'augmented variants of familiar cases versus new cases,' which names the confound, but then interprets the difference as augmentation-induced bias. The TOD category's negative difference (-5%) shows the gap isn't uniform, but it doesn't rescue the interpretation.\n\nExperiment 1 has a separate confound: Phase B trains on three times more data than Phase A (originals plus two augmented versions), so the 12% improvement can't be attributed to augmentation per se. The paper also reports no error bars, significance tests, or multiple runs, and the threats-to-validity section doesn't mention either design confound.\n\nNone of this means the paper is worthless. The guidelines in Section III-F are reasonable, and the study is a useful starting point for a harder question. But as it stands, the empirical support for 'augmentation-induced bias' is not there. To make the claim stick, Experiment 2 would need a test set of augmented versions of held-out originals, or the claim would need to be reframed as an effect of test-set overlap with training data.\n\nWho this is for: SE researchers working with augmented datasets, especially on test classification. They might read it as a cautionary example and a source of evaluation best practices. I'd encourage a serious editor to send it to review, but the expected outcome should be major revision or reject-and-resubmit, not acceptance in current form.","headline":"The 8% augmentation-bias gap is confounded with train-test overlap, so the paper's central claim is unsupported, though the practical guidelines and code make it worth a revision attempt.","tokens_in":6090,"tokens_out":3040,"would_cite":false,"duration_ms":27244,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Augmented test data inflates machine-learning scores: a flaky-test classifier scores on average 8% higher on augmented copies of training cases than on genuinely new test cases.","keywords":["data augmentation","augmentation bias","flaky test classification","model evaluation","SMOTE","CodeBERT","software testing","machine learning"],"falsifier":"Run the same model on a Testing Set 3 containing augmented copies of held-out original test cases that never appear in training; if the F1 gap relative to Testing Set 1 drops to near zero, the 8-point effect comes from training-set overlap rather than augmentation artifacts.","tokens_in":81,"feed_emoji":"📈","tokens_out":7200,"duration_ms":121447,"temperature":0.7,"pith_summary":"The paper sets out to establish that data augmentation can bias machine-learning evaluation when augmented samples appear in both training and testing sets. In a flaky-test classification case study, a model scores about 8% higher on average (F1) when tested on augmented copies of training cases than on genuinely new test cases. The authors interpret this gap as evidence of augmentation-induced artifacts rather than improved generalization, and they recommend a separate non-augmented validation set. If correct, many reported performance numbers for models trained with augmented software-engineering data may be overstated.","feed_headline":"Augmented test data inflates flaky-test F1 scores by about 8%","feed_subtitle":"Models score 8% higher on augmented copies of training cases, masking true skill","key_machinery":"The key machinery is a paired testing protocol. The model is trained only on original flaky-test code, then evaluated on two test sets: Testing Set 1, original versions of unseen test cases, and Testing Set 2, augmented v1/v2 copies of the training cases. The F1 gap between the two sets is the paper's operational measure of augmentation-induced bias. The FlakyCat dataset provides the labeled flaky-test categories, and an adapted SMOTE method (mutating variable names, constants, method names, and unused variable declarations) creates the augmented copies.","core_discovery":"The central claim is that augmentation-induced bias is systematic and measurable. A flaky-test category classifier built on CodeBERT representations and a Siamese contrastive training setup performs better on augmented variants (v1 and v2) of test cases seen in training than on original unseen test cases: the average F1 gap is 8 percentage points, ranging from 13 points for Unordered Collections to -5 points for Test Order Dependency. The same model trained with augmented data beats the original-only baseline by 12 average F1 points, so augmentation helps learning but also injects patterns that make augmented examples artificially easy. The paper concludes that evaluation on augmented test data can inflate performance and recommends keeping a separate original-data validation set when assessing models trained on augmented data.","pith_inferences":["A stronger experimental design would also test augmented copies of held-out original cases; the current comparison cannot fully separate augmentation artifacts from memorization of near-duplicates of training data.","The 8-point gap probably mixes two effects: augmentation-specific patterns and the intrinsic advantage of re-seeing examples similar to the training set; separating them would give a cleaner bias estimate.","A practical standard for ML benchmarks could be to require a 'pristine test gap' against a fully original, never-augmented test set alongside any augmented-data result."],"forward_implications":["Reported F1 scores for augmented-data models may overstate real performance by about 8 points on average unless validation uses non-augmented held-out data.","Dataset providers should label augmented samples so downstream users can exclude them from validation and testing.","Augmentation strategies may need to be category-specific, since the bias varies from 13 points for Unordered Collections to -5 points for Test Order Dependency.","Augmentation still provides a real training benefit (12 F1 points in Experiment 1), so the remedy is controlled evaluation rather than abandoning augmentation.","The same evaluation bias likely affects other software-engineering tasks such as defect prediction and fault localization, as the paper itself suggests."],"supporting_citations":[{"why":"Supplies the FlakyCat dataset and the third-party adapted SMOTE augmentation used in both experiments.","marker":"[9]"},{"why":"Supplies the flaky-test classifier (Siamese network with CodeBERT encoder) that the experiments train and evaluate.","marker":"[10]"},{"why":"Supplies CodeBERT, the base encoder that produces the test-code representations used by the model.","marker":"[11]"},{"why":"Defines SMOTE, the over-sampling method whose adaptation generates the augmented versions.","marker":"[3]"}],"fun_headline_variants":["Augmented test data inflates flaky-test F1 by 8%","Evaluating on augmented test data inflates flaky-test scores","Augmentation bias in test sets masks true flaky-test accuracy","Flaky-test F1 scores inflated 8% by augmented test data"],"cache_read_input_tokens":8320,"weakest_assumption_plain":"The load-bearing premise is that the F1 gap between Testing Set 1 and Testing Set 2 isolates augmentation bias; Testing Set 2 is built from augmented copies of training cases, so the gap also includes the effect of testing on near-duplicates of training examples.","fun_headline_variants_meta":{"raw":{"variants":["Augmented test data inflates flaky-test F1 by 8%","Evaluating on augmented test data inflates flaky-test scores","Augmentation bias in test sets masks true flaky-test accuracy","Flaky-test F1 scores inflated 8% by augmented test data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000709,"raw_usage":{"total_tokens":3119,"prompt_tokens":795,"completion_tokens":2324,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":411,"completion_tokens_details":{"reasoning_tokens":2248}},"tokens_in":411,"tokens_out":2324,"duration_ms":14540,"temperature":1.0,"reasoning_tokens":2248,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T14:19:11.802304+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same model on a Testing Set 3 containing augmented copies of held-out original test cases that never appear in training; if the F1 gap relative to Testing Set 1 drops to near zero, the 8-point effect comes from training-set overlap rather than augmentation artifacts.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the FlakyCat dataset and the third-party adapted SMOTE augmentation used in both experiments."},{"cited_title":"More and J","cited_arxiv_id":null,"evidence_quote":"Supplies the flaky-test classifier (Siamese network with CodeBERT encoder) that the experiments train and evaluate."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies CodeBERT, the base encoder that produces the test-code representations used by the model."},{"cited_title":"Chawla, K","cited_arxiv_id":null,"evidence_quote":"Defines SMOTE, the over-sampling method whose adaptation generates the augmented versions."}],"review_version":1}