{"id":"2ce5bf0c-d5bf-483c-87e3-c18aebc68a58","arxiv_id":"2412.16336","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Only 58 of 314 eligible deep learning benchmark faults meet all four realism conditions, and only 86 of 165 reproduction attempts succeed, suggesting most 'real' DL fault benchmarks are not faithful to their sources.","lead":"Researchers manually audited 490 bugs across five deep learning fault benchmarks and found that only 18.5% of eligible bugs satisfy their four realism conditions, with only 52% of attempted bugs reproducing. The finding matters because the software engineering community uses these benchmarks to evaluate testing, localization, and repair tools, so unrealistic bugs can distort those evaluations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 18.5% realism rate is inflated by counting missing source information as a mismatch and by applying training-data conditions to tensor-shape faults for which the paper itself deems them irrelevant.","rationale":"The reader's weakest assumption identifies the source-completeness problem, and the paper's own text supports it: 84% of faults come from SO, and Section 7.1.2 admits that SO posts often lack full code and dataset information. My concern goes one step further by tying this to the specific arithmetic of the 18.5% figure. The realism definition requires all four conditions, but condition 3 cannot be verified when the source omits the dataset, and the paper counts that omission as a mismatch. Section 4.2.5 shows the authors themselves believe training-data realism is unnecessary for tensor-shape faults, yet the headline still includes those conditions for SFData. This makes the compound metric internally inconsistent: the same conditions are considered irrelevant for reproduction but decisive for realism. A simple arithmetic recomputation shows the headline could nearly triple if the conditions are applied consistently with the reproduction protocol. This does not mean the paper's qualitative message is wrong; there are documented cases where benchmarks changed epochs, datasets, or fixes relative to what the source actually stated. The per-dataset tables (Table 2) are a strength because they allow readers to separate source-code correspondence from training-data issues. But the abstract's unqualified 'only 18.5%' is misleading without a sensitivity analysis. The paper should report realism rates both under the strict all-four-condition definition and under the relaxed definition it uses for reproduction, and it should explicitly treat absent dataset information as 'unknown' rather than 'mismatched.' The verdict remains CONDITIONAL because the central phenomenon, that a sizable fraction of benchmark faults deviate from their sources and are hard to reproduce, is plausible and independently supported in part by prior work (e.g., Kim et al. found only 9 of 58 DeepFD faults reproducible), but the headline number needs qualification.","tokens_in":17003,"tokens_out":4387,"duration_ms":41710,"concrete_test":"Recompute the realism rates reported in Table 2 under two alternative scorings: (a) mark training-data correspondence as 'N/A' (not a failure) when the SO post or GitHub commit does not specify the training dataset, and (b) drop the two training-data conditions for tensor-shape faults exactly as the paper does for reproduction in Section 4.2.5. If the overall all-conditions rate moves from 18.5% to substantially above 50%, or if the per-dataset rates change materially for SFData, then the headline is an artifact of the scoring rule rather than a measure of how faithfully the benchmarks preserve real faults.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that only 18.5% of the 314 eligible faults are 'real' depends on a compound realism criterion whose third condition, 'training data matches the source,' is unsatisfiable whenever the source does not mention the dataset. The paper's own results show this is not a rare case: Section 5.2 states that the main cause of training-data mismatches is 'the lack of training dataset information in SO posts,' and Section 7.1.2 acknowledges that SO posts 'very often do not contain the full code and information on the used training dataset.' For such faults, the benchmark authors necessarily fill gaps, and a mismatch between the filled-in data and an absent source specification is not evidence that the fault is fake. Additionally, the paper relaxes the two training-data conditions for SFData tensor-shape faults when attempting reproduction (Section 4.2.5), yet keeps them in the headline realism count. This is inconsistent: for 143 SFData bugs, 123 have buggy and fixed code matching the source, but only 16 satisfy all four conditions; the gap is almost entirely due to training-data conditions that the paper itself treats as irrelevant for reproducing this class of faults. Recomputing with training-data conditions dropped for SFData raises the all-four rate from 58/314 (18.5%) to roughly 165/314 (52.5%) before accounting for other source-incompleteness cases. Thus the 18.5% headline conflates 'the benchmark deviates from a complete, authoritative source' with 'the source was too incomplete to permit a full reconstruction,' and it overstates the fraction of faults that are unfaithful to their real-world origins.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports a manual audit of 490 faults from five deep learning fault benchmarks (TFBugs2018, DeepLocalize, DeepFD, Defects4ML, SFData). The authors define four realism conditions: the buggy code must match the code in the source (a Stack Overflow post or GitHub commit), the fix must match the source's fix, the training data must match the dataset mentioned in the source, and the training data must be realistic. After excluding 176 faults that lack usable sources or are not genuine DL faults, 314 faults are analyzed. Only 58 (18.5%) satisfy all four conditions. For the 165 faults that passed the realism conditions (with the two training-data conditions relaxed for SFData tensor-shape faults), reproduction attempts succeeded for 86 (52%), with 79 stable across 20 runs. The paper also analyzes the distribution of fault types using the taxonomy by Humbatova et al. and compares fault components with the mutation operators of DeepCrime, finding that only 22% of faults can be fully simulated by existing mutation operators.","tokens_in":17274,"tokens_out":6917,"duration_ms":52887,"significance":"If the 18.5% realism rate were taken at face value, this would be a major indictment of the five benchmarks and of all evaluations built on them, making the paper a landmark in the field. The paper makes several strong contributions: a systematic literature search with snowballing, a detailed manual procedure with pilot and consensus phases, a replication package with the analysis spreadsheet, and a set of reproducibility experiments with stability checks. The comparison with DeepCrime mutation operators is a useful addition to the mutation-testing literature. However, the headline rate is load-bearing and is currently overstated, because it counts 'source does not specify the dataset' as a failure of the benchmark and because it applies training-data conditions to tensor-shape faults that the paper itself excludes when reproducing. With suitable sensitivity analyses the paper could still support a strong qualitative conclusion, but the specific number 18.5% needs to be re-derived or heavily qualified. The contribution is nevertheless valuable and likely to influence how the community interprets DL fault benchmarks.","major_comments":[{"comment":"The 'training data matches source' condition is unsatisfiable when the source post does not state which dataset was used; the paper's own analysis indicates this is the dominant cause of training-data mismatches (Section 5.2) and that Stack Overflow posts 'very often do not contain the full code and information on the used training dataset' (Section 7.1.2). For such faults, any training data chosen by the benchmark authors will by construction fail condition (3), so the mismatch measure conflates 'source omits information' with 'benchmark is unrealistic.' The abstract's 18.5% figure therefore needs to be accompanied by a sensitivity analysis that treats missing source dataset information as 'unknown' rather than 'non-matching,' or that reports the realism rate on the subset of faults whose sources actually specify a dataset.","section":"Section 5.2 and Section 7.1.2"},{"comment":"The paper relaxes the two training-data conditions for SFData when attempting reproduction, arguing that 'the usage of realistic training data is not relevant to the reproduction of such faults,' yet these conditions are retained in the headline realism count. For SFData, 123 of 143 faults have buggy and fixed code matching the source, but only 16 satisfy all four conditions, with the gap driven by training-data conditions that the paper itself deems irrelevant for this class. Dropping the training-data conditions for SFData, as the paper does for reproduction, yields exactly 165 of 314 (52.5%) faults satisfying the remaining conditions, not 18.5%. The paper should recompute the all-conditions realism rate under the same relaxation used for reproduction (or at least report both rates) and should present the resulting number as the primary result if the qualitative conclusion changes materially.","section":"Section 4.2.5 and Table 5"},{"comment":"No inter-rater reliability statistic is reported for the manual classification, although the pilot and consensus meetings are described. Because the central findings depend on subjective judgments (e.g., whether a textual fix suggestion matches the applied fix, whether a dataset is 'real,' whether a code difference is 'syntactic' or changes model structure), the absence of a quantitative agreement measure (e.g., Cohen's kappa) weakens the replicability of the audit. The authors should provide such a statistic computed on the pilot or on a random sample, or justify why it is not applicable.","section":"Section 4.3"}],"minor_comments":[{"comment":"In the TFBugs2018 row, the 'RSB R.' column reports '0/5 (60%)', which is internally inconsistent; the percentage should be 0%.","section":"Table 4"},{"comment":"The RQ1 summary states that 'bugs and fixes in the benchmarks match the sources 81.2% and 78% of the time,' but Table 2 shows an overall 'Buggy m.' of 254/314 = 80.9%, not 81.2%; the 81.2% figure appears to be a typo.","section":"Section 5.1"},{"comment":"The statement that reproduction attempts were 'successful only in 52% of cases' refers to the 165 bugs selected for reproduction attempts, not to all 314 eligible faults; this should be stated explicitly to avoid misinterpretation.","section":"Abstract and Section 5.5"},{"comment":"The sentence 'As there is a low rate of match between the buggy version and the initial source code in SO, our assumption that the information in the posts is very often not sufficient to reproduce the reported problem' is grammatically incomplete; it should be rephrased, for example by splitting the thought into two sentences.","section":"Section 7.1.2"},{"comment":"The date on which the Scopus search was performed is not reported; adding the search date would improve the reproducibility of the systematic literature search.","section":"Section 2.2"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The paper is within scope and the underlying dataset is valuable, but the abstract's central claim (18.5% realism) is sensitive to the authors' own definitional choices, particularly the treatment of missing training-data information and the inconsistent handling of SFData's training-data conditions. The paper would benefit from a sensitivity analysis and a more cautious framing of the headline result. The manual analysis lacks an inter-rater reliability statistic, which is worth requesting in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nRead this if you care about how DL testing/repair evaluations are built. The paper manually audits 490 faults from five DL fault benchmarks, checks whether buggy/fixed code and training data match the stated SO/GitHub source, and tries to reproduce the faults. The headline is that only 18.5% of 314 eligible faults pass all four realism conditions; only 52% of 165 attempted reproductions succeed.\n\nWhat's genuinely new: this is the first multi-benchmark audit with explicit correspondence conditions. Prior work (Kim et al.) only reproduced DeepFD faults. The systematic search, the pilot, and the consensus process are real effort, and the per-benchmark breakdowns are useful. The reproducibility part is the strongest: independent of the realism definitions, 86 of 165 bugs reproduced, and only 79 stable across 20 runs—that's a concrete, credible finding.\n\nThe soft spot is the headline 18.5% number. The stress-test note is right: condition 3, 'training data matches source,' fails whenever the SO post doesn't mention the dataset. The paper's own Section 5.2 says the main cause of mismatch is lack of dataset information in posts, and Section 7.1.2 admits SO posts often don't contain full code or dataset info. For SFData specifically, the paper relaxes the training-data conditions for reproduction but keeps them in the realism count. Dropping only those two conditions for SFData lifts the all-four rate from 58/314 to roughly 165/314. So the 18.5% conflates 'benchmark deviates from an incomplete source' with 'benchmark is unfaithful to a real fault.' That's a real overstatement.\n\nThe paper still shows genuine problems: DeepLocalize and DeepFD alter epochs/dataset sizes in ways that create new root causes, some fixes were invented by benchmark authors, and many faults aren't reproducible. So the central message survives, but the precise 18.5% should not be quoted without the caveat.\n\nMissing: an inter-rater reliability statistic for the manual coding. They did a pilot and consensus meetings, but a kappa/alpha would help trust the categories.\n\nWho it's for: anyone using these benchmarks for evaluation, and anyone building mutation tools—RQ4's finding that only 22% of faults can be simulated by DeepCrime is directly actionable. It deserves serious peer review; the flaws are fixable by re-reporting per-condition and per-dataset rates, and the reproducibility data is worth publishing on its own.\n\nRecommendation: engage with it, and push for a revised version that separates 'source incomplete' from 'benchmark unfaithful.'","headline":"A useful, well-executed audit of DL fault benchmarks, but the headline 18.5% realism rate is inflated by counting missing source information as a mismatch—read it for the reproducibility data and per-benchmark numbers, not the headline.","tokens_in":17846,"tokens_out":2277,"would_cite":true,"duration_ms":19055,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A manual audit of 314 deep-learning faults finds that only 18.5% match the source they were mined from.","keywords":["deep learning faults","fault benchmarks","realism audit","reproducibility","fault types","mutation testing","empirical study","tool evaluation"],"falsifier":"Take the faults that failed the realism audit and compare each against an archived version of its source, separating cases where the source lacks information from cases where the benchmark changed the code. If most mismatches vanish once incomplete sources are set aside, the 18.5% figure overstates the problem; if most mismatches are benchmark-side alterations, the figure stands.","tokens_in":16801,"feed_emoji":"🐛","tokens_out":6036,"duration_ms":50429,"temperature":0.7,"pith_summary":"This paper tries to establish whether faults in widely used deep-learning fault benchmarks are really the real faults they claim to be. The authors manually checked 314 eligible faults from five benchmarks against the forum posts and code commits those faults were supposedly extracted from. They report that only 58 faults, 18.5%, satisfy all four of their realism conditions, and only 86 of 165 faults eligible for reproduction could be reproduced on current environments. If the audit is right, evaluations of deep-learning testing, fault-localisation, and repair tools on these benchmarks may not reflect how those tools perform on genuine real-world faults.","feed_headline":"Only 18.5% of 'real fault' deep-learning benchmarks pass an audit","feed_subtitle":"That means evaluations of DL testing, fault-localisation, and repair tools may not reflect performance on real faults.","key_machinery":"The audit instrument is a set of four realism conditions applied by manual inspection: the buggy code must match the source, the fix must match the source, the training data must match the source, and the training data must be realistic rather than random filler. Faults that pass are then executed repeatedly, with crashes required to appear in the buggy version and disappear in the fixed version, and low-performance symptoms required to show a statistically significant improvement across twenty runs as judged by a generalized linear model and an effect-size threshold. This combination of source-correspondence checks and repeated execution is what supports the headline percentages.","core_discovery":"The paper's central discovery is that the label 'real fault' attached to benchmark entries is largely unsupported by the evidence. After manually comparing the buggy code, the fix, and the training data of each fault against its stated source, only 58 of 314 faults met all four realism conditions, and reproduction attempts succeeded for only 86 of 165 eligible faults, with 79 of those stable across twenty runs. The mismatches are not rare exceptions: for two benchmarks, nearly half of the buggy versions or fixes do not match their sources, and most analysed faults come from forum posts that often omit the full code and dataset information needed for faithful reproduction.","pith_inferences":["If the sources themselves are often incomplete, then 'does not match source' may partly reflect missing information in the original post rather than alteration by the benchmark authors; separating those two cases could change the 18.5% figure.","Even if many benchmark pairs are not faithful to their sources, they may still be usable as curated buggy-fixed pairs for tool evaluation, effectively serving as realistic mutants rather than true real faults.","A practical extension would be to turn the four-condition audit into a reusable checklist and a publicly maintained registry of verified faults, so new benchmarks can be certified before widespread adoption.","Reproduction under original pinned environments, for example using containers, would separate environment drift from fault realism; the reported 52% reproduction rate is an upper bound under current-version conditions."],"forward_implications":["Benchmark-based evaluations of deep-learning testing, localisation, and repair tools likely overstate how well those tools handle real faults.","The pool of genuinely real and reproducible faults across the five benchmarks is small, so benchmark diversity is narrower than the total bug count suggests.","Fault types are heavily skewed, with only 36 fault types represented among the realistic faults and three fault types accounting for a large share of all fault components.","Future fault extraction should apply stricter inclusion criteria, because many forum posts lack the full code and dataset information needed for faithful reproduction.","Higher-order faults, with an average order of 1.9 and some reaching order 12, are common, so mutation tools will need higher-order operators to approximate real faults."],"supporting_citations":[{"why":"Supplies one of the five analysed benchmarks, with faults drawn from an early deep-learning framework.","marker":"[16]"},{"why":"Supplies one analysed benchmark whose buggy and fixed versions were compared with their stated sources.","marker":"[24]"},{"why":"Supplies one analysed benchmark and contributes source posts that overlap with another benchmark.","marker":"[4]"},{"why":"Supplies the largest independent benchmark and was used for the pilot study that calibrated the manual analysis.","marker":"[7]"},{"why":"Supplies the largest analysed fault set, focused on tensor-shape faults, with training-data conditions relaxed for reproduction.","marker":"[13, 12]"},{"why":"Supplies the taxonomy of deep-learning fault types used to label each diff in the fault-type analysis.","marker":"[10]"},{"why":"Supplies the mutation operators used to test whether benchmark faults could be generated artificially.","marker":"[26, 27]"},{"why":"Supplies a prior reproduction study of one benchmark whose low reproduction rate motivates this broader audit.","marker":"[14]"}],"fun_headline_variants":["Only 18.5% of DL fault benchmarks are truly real","Benchmark real bugs: only 18.5% hold up to scrutiny","Most real DL faults fail reality check: only 18.5% pass","Only 1 in 5 DL benchmark faults are actually real","Just 18.5% of DL benchmark faults pass realism test"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The audit treats the source post or commit as complete and authoritative ground truth for the fault and its fix, even though such posts often omit code and dataset details.","fun_headline_variants_meta":{"raw":{"variants":["Only 18.5% of DL fault benchmarks are truly real","Benchmark real bugs: only 18.5% hold up to scrutiny","Most real DL faults fail reality check: only 18.5% pass","Only 1 in 5 DL benchmark faults are actually real","Just 18.5% of DL benchmark faults pass realism test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000701,"raw_usage":{"total_tokens":3105,"prompt_tokens":829,"completion_tokens":2276,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":445,"completion_tokens_details":{"reasoning_tokens":2196}},"tokens_in":445,"tokens_out":2276,"duration_ms":13703,"temperature":1.0,"reasoning_tokens":2196,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:40:42.530126+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the faults that failed the realism audit and compare each against an archived version of its source, separating cases where the source lacks information from cases where the benchmark changed the code. If most mismatches vanish once incomplete sources are set aside, the 18.5% figure overstates the problem; if most mismatches are benchmark-side alterations, the figure stands.","supporting_citations":[{"cited_title":"An empirical study on tensorflow program bugs,","cited_arxiv_id":null,"evidence_quote":"Supplies one of the five analysed benchmarks, with faults drawn from an early deep-learning framework."},{"cited_title":"Deeplocalize: Fault localization for deep neural networks,","cited_arxiv_id":null,"evidence_quote":"Supplies one analysed benchmark whose buggy and fixed versions were compared with their stated sources."},{"cited_title":"Bugs in machine learning-based systems: a faultload benchmark,","cited_arxiv_id":null,"evidence_quote":"Supplies the largest independent benchmark and was used for the pilot study that calibrated the manual analysis."},{"cited_title":"Repairing dnn architecture: Are we there yet?","cited_arxiv_id":null,"evidence_quote":"Supplies a prior reproduction study of one benchmark whose low reproduction rate motivates this broader audit."}],"review_version":1}