{"id":"3c94ca7d-465f-4b7d-809e-25af62b4c4d0","arxiv_id":"2411.12137","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Deep learning models trained on buggy software engineering data show characteristic gradient, weight, and bias symptoms that can flag data quality problems.","lead":"This paper shows that deep learning models trained on low-quality software engineering data display distinctive internal warning signs in their gradients, weights, and biases. A monitoring tool that watches these signs could help engineers catch data bugs during training.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never documents how individual bug types were isolated; it uses pre-existing dirty datasets that themselves contain multiple intertwined quality issues, so the symptom-to-bug attributions in Tables 4-15 are confounded.","rationale":"I read the paper as an empirical mapping from data-quality and preprocessing defects to training-time observables, with validation on six extra datasets and a monitoring prototype. The strongest independent support is the multirun design (15 runs per configuration), the validation datasets, and DataGuardian's moderate detection rates. I looked for the condition that must hold for the central claim: the buggy variants must isolate the named bug. That condition is least secure. Section 3.3.2 obtains buggy datasets from prior studies, and Section 3.2.4 documents multiple co-occurring quality issues in those same datasets, yet Section 4 reports per-bug-type percentages and Section 3.5 calls the design controlled. No injection protocol is given anywhere. This is not a disagreement with consensus; it is an internal gap between the method as described and the causal attributions in the results. I also note secondary issues: Tables 4-15 lack variance, and the abstract's metric-data sentence attributes exploding gradients to quality issues when Table 14 ties exploding gradients to missing preprocessing. These are less load-bearing than the confounding problem. The proposed audit and controlled injection check would settle it. If single-bug injection reproduces the symptoms, the paper's fingerprint idea survives; if not, the verdict should become REJECT or UNVERDICTED for the causal claims. Since the reader already reached CONDITIONAL for exactly this reason, I recommend no change to the verdict.","tokens_in":34572,"tokens_out":5199,"duration_ms":51375,"concrete_test":"Audit the actual dataset variants from the replication package or the cited sources [10, 104, 102]. For each variant used in Tables 4-15, measure the per-instance label flip rate relative to the clean version, the class-imbalance ratio, and the temporal or feature-distribution shift; then check whether each variant differs from clean only in the intended dimension. Next, run one controlled injection experiment per bug type: start from the clean data, inject only label noise at the measured rate (or only undersample to the measured imbalance, or only split by time to create drift), keep all preprocessing identical, retrain the same baselines, and test whether the same symptom (e.g., near-zero biases for label noise in code data) appears.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that specific data bugs produce identifiable, generalizable training-time symptoms, and that these symptoms can serve as fingerprints for monitoring. That claim requires that each 'buggy' dataset variant differ from its clean counterpart only in the named bug. The methodology never establishes this. Section 3.3.2 says only that buggy datasets were 'obtained from the same existing studies [10, 104, 102]', with no description of how label noise, class imbalance, or concept drift were injected into Devign, BigVul, Eclipse, Hadoop, OpenStack, or QT. Section 3.2.4, however, states that these datasets contain multiple simultaneous quality problems: Devign and BigVul have label noise and temporal relevance issues; Eclipse and Hadoop have duplicate-report issues, concept drift, and label noise; OpenStack and QT have labelling errors, class imbalance, and concept drift. Yet Tables 4, 8, and 12 treat label noise, class imbalance, and concept drift as separate experimental conditions, and Section 3.5 refers to 'controlled experiments' and to introducing 'a bug like class imbalance'. If the same pre-existing dirty dataset was labelled as more than one bug type, or if a 'label-noise' variant also contains class imbalance and drift, the observed symptoms cannot be attributed to a single bug. Because the abstract and DataGuardian convert these attributions into causal fingerprints, this is the load-bearing link of the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an empirical study of how data-quality and preprocessing bugs affect the training behaviour of deep learning models for software engineering tasks. Using three data types (code, text, metric), two datasets per type, and state-of-the-art baselines (LineVul, CodeBERT, DCCNN, BERT-MLP, DeepJIT, CodeBERTJIT), the authors train clean and buggy variants, log gradients/weights/biases with Weights & Biases, identify statistical symptoms, and triangulate with attention, t-SNE, and GradCAM analyses. They then validate their findings on six additional datasets and present a prototype monitoring tool, DataGuardian. The headline claim is that specific data bugs (label noise, class imbalance, concept drift) and missing preprocessing produce distinct, generalizable training-time symptoms that can serve as fingerprints for detection.","tokens_in":34819,"tokens_out":4880,"duration_ms":48711,"significance":"If the central claim holds, this is a useful contribution: training-time gradient, weight, and bias statistics would provide a practical, low-cost signal for monitoring data quality in SE deep-learning pipelines, complementing existing performance-based evaluations. The study has clear strengths: it covers multiple data types and tasks, runs each configuration fifteen times to address stochasticity, triangulates quantitative metrics with three XAI techniques, validates on six held-out datasets, and provides a replication package and a working prototype. These design choices make the empirical skeleton credible and the results potentially actionable for MLOps-oriented work.","major_comments":[{"comment":"The central claim that specific data bugs produce identifiable fingerprints requires that each buggy variant differ from its clean counterpart only in the named bug. Section 3.3.2 states only that buggy datasets were 'obtained from the same existing studies [10, 104, 102]' and never describes how label noise, class imbalance, or concept drift were injected into Devign, BigVul, Eclipse, Hadoop, OpenStack, or QT. Section 3.2.4, however, reports that each of these datasets contains multiple simultaneous quality issues (e.g., Devign and BigVul have label noise and temporal relevance issues; OpenStack and QT have labelling errors, class imbalance, and concept drift). If the same pre-existing dirty dataset is labelled as more than one bug type, or if a 'label-noise' variant also contains drift and imbalance, the symptom-to-bug attributions in Tables 4, 8, and 12 are confounded. Section 3.5's reference to 'controlled experiments' and to introducing 'a bug like class imbalance' does not resolve this because no injection protocol appears anywhere in the manuscript. The authors must specify the construction of each clean/buggy pair and, ideally, verify that the induced perturbation is the only difference.","section":"§3.3.2, §3.2.4, Tables 4/8/12"},{"comment":"The symptom-determination procedure is partly circular. Section 3.5 derives the statistical thresholds (Z-score > 3, IQR fences, skewness > 1, |kurtosis| > 3) by comparing the buggy models against the bug-free baseline, and then uses those same thresholds to declare which symptoms are present in the buggy models and to attribute them to data-quality issues. Because the thresholds are fit to the very comparisons they are used to explain, the prevalence percentages in Tables 4-15 and the odds ratios in Section 4.4.2 are not independent evidence. The authors should either fix the thresholds a priori (e.g., from literature or from a separate calibration run) or clearly present the validation-dataset results as the only non-circular check, with thresholds locked before that validation.","section":"§3.5"},{"comment":"The generalizability evaluation is reported at too coarse a granularity to support the claim. Section 4.4 gives aggregate percentages across the six held-out datasets but no per-dataset tables, per-symptom counts, or dataset-level statistical results, so the assertion that findings align 'within a 1-5% margin' cannot be checked. Likewise, Section 5.4 reports DataGuardian's success rates (12/15, 11/15, etc.) on Juliet, Mozilla, and JDT without defining what counts as a correct root-cause identification or providing a comparison against clean-data runs; since the thresholds that define the symptoms are the same thresholds DataGuardian uses to infer root causes, this evaluation does not independently validate the bug-type attribution.","section":"§4.4 and §5.4"}],"minor_comments":[{"comment":"The text says 'As shown in Table 3, DeepJIT witnessed a 4.45-9.24% performance drop', but the metric-based performance results appear in Table 13, not Table 3; please correct the cross-reference.","section":"§4.3.1(a)"},{"comment":"In the buggy-data paragraph, 'the models trained on LineVul (with label noise)' appears to refer to a dataset (Devign) rather than the model; please clarify the naming.","section":"§4.1.3"},{"comment":"There are small formatting slips, including 'T ext-Based' in Table 1 and 'F rameworks' in Section 3.3.1; please proofread the manuscript.","section":"Table 1 and §3.3.1"},{"comment":"The heading 'Signifiance Tests' is misspelled, and the odds-ratio range reported (15.53 to 806.10) should be accompanied by confidence intervals, since McNemar's test alone does not quantify uncertainty in the odds ratio.","section":"§4.4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a strong empirical skeleton, but the bug-type-specific conclusions are currently overclaimed relative to the methodology. If the authors can document controlled perturbation or reframe the paper as dataset-level clean-versus-dirty comparisons without bug-type attribution, the contribution would be publishable; in current form the central fingerprint claim is not yet supported. The validation and DataGuardian evaluations also need more detail before the practical monitoring claims can be assessed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a useful, big empirical map from dirty training data to training-time symptoms in DL4SE models, but the specific bug-to-symptom attributions are weaker than the abstract claims because the “buggy” datasets are pre-existing dirty corpora, not controlled injections. The central idea is worth engaging; the load-bearing link needs repair.\n\nWhat’s genuinely new: the paper is the first to look at internal training dynamics—gradients, weights, biases—rather than just end-task performance, for data bugs in software engineering. It does this across three data types (code, text, metric) and three representative tasks, with 900 models, 15 runs each, and W&B logging. The triangulation with attention weights, t-SNE, and GradCAM adds qualitative texture. The validation on six held-out datasets and the DataGuardian prototype are constructive, and the Zenodo replication package is a real asset.\n\nThe soft spots are real. Section 3.3.2 says buggy datasets were “obtained from the same existing studies,” with no description of how label noise, class imbalance, or concept drift were isolated. But Section 3.2.4 lists multiple simultaneous quality issues in those same datasets (label noise plus temporal drift in Devign and BigVul; duplicates, drift, and label noise in Eclipse and Hadoop; labelling errors, imbalance, and drift in OpenStack and QT). Tables 4, 8, and 12 treat these as separate conditions, and Section 3.5 calls them “controlled experiments.” That is a mismatch. If a “label-noise” variant also contains class imbalance and concept drift, the symptom-to-bug mapping in those tables is confounded. The DataGuardian evaluation on Juliet, Mozilla, and JDT helps, but those datasets come from the same sources, so it doesn’t fully break the circularity. Also, symptom thresholds are defined from the same clean-vs-buggy comparisons they are later used to explain; the held-out validation mitigates this but doesn’t eliminate it. And the performance tables report no variance, though the text mentions 15 runs.\n\nNone of this sinks the paper. The broad finding—dirty data degrades training dynamics—is unsurprising but now documented at scale. The specific fingerprint claims need either controlled bug-injection experiments or substantially softened language. The causal phrasing in the abstract (“cause biased learning”) overstates what the design can show.\n\nWho should read this: practitioners building data-quality monitors for DL4SE, and researchers cataloguing data-bug symptoms. It deserves a serious referee, but that referee should ask for controlled injections or cautious rewording, error bars, and a clearer account of dataset provenance.","headline":"A large-scale, useful map of training-time symptoms for dirty data in DL4SE, but the specific bug-to-symptom attributions are confounded because the 'buggy' datasets are pre-existing dirty corpora with multiple intertwined quality issues, not controlled injections.","tokens_in":35363,"tokens_out":2330,"would_cite":true,"duration_ms":23511,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training-time gradients, weights, and biases can fingerprint which data bug is poisoning a deep learning model.","keywords":["data bugs","training data quality","deep learning","software engineering","gradient analysis","label noise","class imbalance","concept drift"],"falsifier":"Audit the actual label error rates, class imbalance ratios, and distribution shifts in the clean and buggy variants of Devign, BigVul, Eclipse, Hadoop, OpenStack, and QT; if a buggy variant differs from its clean pair in multiple quality dimensions at once, the claimed fingerprint for the named issue would not be supported.","tokens_in":1525,"feed_emoji":"🐛","tokens_out":1721,"duration_ms":63508,"temperature":0.7,"pith_summary":"This paper tries to show that data bugs in software engineering datasets leave observable fingerprints in the internal state of deep learning models during training. It compares models trained on clean data with models trained on buggy code-based, text-based, and metric-based data, and with models whose preprocessing steps were omitted. Its central claim is that code-data problems show up as biased learning and gradient instability, text-data problems as overfitting and poor generalization, and metric-data problems as vanishing or exploding gradients and overfitting, and that these patterns repeat across new datasets. If true, practitioners could detect dirty training data by watching gradients, weights, and biases rather than waiting for end-task failures. The paper also builds a monitoring prototype that uses the derived thresholds to flag training problems and their root causes.","feed_headline":"Training data bugs leave distinct fingerprints in model weights","feed_subtitle":"Code, text, and metric bugs each show different training-time symptoms, and cleaning removes them.","key_machinery":"The central machinery is the layer-wise monitoring of gradients, weights, and biases during training, aggregated into statistics such as mean, median, variance, skewness, kurtosis, and sparsity, with Z-scores, interquartile range outliers, and distribution-shape thresholds used to flag deviations. These aggregated statistics are compared across clean, buggy, and missing-preprocessing variants to define recognizable 'symptoms'; post-hoc analyses with attention weights, t-SNE embeddings, and Grad-CAM maps are used to confirm that the affected models lose focus on meaningful input features. The comparison of parameter statistics across variants is what turns training dynamics into diagnostic fingerprints.","core_discovery":"The paper claims that each data type has a distinct symptom signature. Quality issues in code data produce near-zero biases, smaller weights, and gradient instability; quality issues in text data produce abnormal weight distributions, gradient skewness, and overfitting to noisy patterns; quality issues in metric data produce sparse parameter updates, vanishing gradients, and higher training loss. Missing preprocessing produces its own set of signatures: slow convergence, extreme bias values, and skewed parameter distributions for code; extreme weights and skewed bias distributions for text; and exploding gradients with high weight variance for metrics. The paper further claims that these signatures generalize across six additional datasets and that cleaning data and restoring preprocessing removes nearly all symptoms, with occurrence rates dropping from 70-85% of runs to below 12%.","pith_inferences":["If training-time parameter statistics really fingerprint data bugs, the same approach could be applied beyond software engineering to any domain with paired clean and buggy datasets, and the paper's own code-summarization prototype suggests this extension is plausible.","A stronger test of the claimed signatures would train on datasets with a single artificially injected bug at controlled severity levels, producing dose-response curves for each symptom rather than relying on pre-existing buggy variants.","The symptom-to-bug attributions assume each buggy variant differs from its clean pair only in the named issue; measuring the actual label error rates, class ratios, and distribution shifts in the variants would decide whether the signatures are specific or conflated.","Because the prototype monitor detects the correct root cause in only 10 to 12 of 15 runs, the remaining runs hint at a false-negative tail, and a practical system would likely need to combine training-dynamics thresholds with direct data-side checks."],"forward_implications":["Monitoring systems can flag near-zero biases, smaller weights, or gradient instability during training as early warnings of label noise or concept drift in code-based datasets.","Text-based models that show abnormal weight distributions, gradient skewness, or unusually high bias values are candidates for a data-quality audit targeting concept drift or class imbalance.","Metric-based models exhibiting vanishing gradients, sparse parameter updates, or stagnant loss can be traced to class imbalance, while exploding gradients and high weight variance point to missing feature normalization.","Omitting preprocessing is itself a detectable bug: slow convergence, extreme or skewed parameter distributions, and exploding gradients appear within a few epochs and disappear after restoring preprocessing.","Cleaning data and restoring preprocessing reduces the occurrence of these symptoms from the 70-85% range to below 12%, suggesting that monitoring can double as a validation step before deployment."],"supporting_citations":[{"why":"It supplies the buggy and clean code-based datasets for vulnerability detection and the label-quality measurements used to justify the quality issues.","marker":"[10]"},{"why":"It supplies the text-based duplicate-bug-report datasets with known concept drift and duplicate rates, and the clean and buggy variants used in RQ2.","marker":"[104]"},{"why":"It supplies the metric-based just-in-time defect prediction datasets and the clean and buggy variants used in RQ3.","marker":"[102]"},{"why":"It is the code-based baseline model and the source of the attention-weight analysis procedure used to interpret what the model learns.","marker":"[19]"},{"why":"It is the metric-based baseline model and the source of the OpenStack and QT datasets that exhibit class imbalance.","marker":"[29]"},{"why":"It is the text-based baseline model for duplicate bug report detection and the model whose learned representations are visualized with t-SNE.","marker":"[28]"},{"why":"It provides the taxonomy of deep learning faults used to define the missing-preprocessing conditions and to choose which preprocessing operations to omit.","marker":"[31]"}],"fun_headline_variants":["Code, text, and metric bugs each leave distinct training signatures","Cleaning data and preprocessing removes most bug symptoms","Data bug signatures generalize across six new datasets","Training bugs in code, text, and metric data show unique fingerprints"],"cache_read_input_tokens":37504,"weakest_assumption_plain":"The study assumes that each buggy dataset contains only the named quality problem, and that its paired clean version isolates that problem; if a buggy dataset mixes several quality issues, the symptom-to-bug attributions do not hold.","fun_headline_variants_meta":{"raw":{"variants":["Code, text, and metric bugs each leave distinct training signatures","Cleaning data and preprocessing removes most bug symptoms","Data bug signatures generalize across six new datasets","Training bugs in code, text, and metric data show unique fingerprints"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000251,"raw_usage":{"total_tokens":1559,"prompt_tokens":950,"completion_tokens":609,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":544}},"tokens_in":566,"tokens_out":609,"duration_ms":7283,"temperature":1.0,"reasoning_tokens":544,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:52:09.120872+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Audit the actual label error rates, class imbalance ratios, and distribution shifts in the clean and buggy variants of Devign, BigVul, Eclipse, Hadoop, OpenStack, and QT; if a buggy variant differs from its clean pair in multiple quality dimensions at once, the claimed fingerprint for the named issue would not be supported.","supporting_citations":[{"cited_title":"Duplicate bug report detection: How far are we? ACM Transactions on Software Engineering and Methodology, 32(4):1–32, 2023","cited_arxiv_id":null,"evidence_quote":"It supplies the text-based duplicate-bug-report datasets with known concept drift and duplicate rates, and the clean and buggy variants used in RQ2."}],"review_version":1}