Pith. sign in

REVIEW 3 major objections 5 minor 32 references

Real Faults in Deep Learning Fault Benchmarks: How Real Are They?

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A manual audit of 314 deep-learning faults finds that only 18.5% match the source they were mined from.

desk verdict A useful, well-executed audit of DL fault benchmarks, but the headline 18.5% realism rate is inflated by counting missing source information as a mismatch—read it for the reproducibility data and per-benchmark numbers, not the headline. read the letter →

arxiv 2412.16336 v1 pith:237PV7YZ submitted 2024-12-20 cs.SE cs.AIcs.LG

Gunel Jahangirova , Nargiz Humbatova , Jinhan Kim , Shin Yoo , Paolo Tonella This is my paper · ORCID
classification cs.SEcs.AIcs.LG
keywords deeplearningfaultsfaultbenchmarksrealismauditreproducibilitytypesmutationtestingempiricalstudytoolevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish whether faults in widely used deep-learning fault benchmarks are really the real faults they claim to be. The authors manually checked 314 eligible faults from five benchmarks against the forum posts and code commits those faults were supposedly extracted from. They report that only 58 faults, 18.5%, satisfy all four of their realism conditions, and only 86 of 165 faults eligible for reproduction could be reproduced on current environments. If the audit is right, evaluations of deep-learning testing, fault-localisation, and repair tools on these benchmarks may not reflect how those tools perform on genuine real-world faults.

What carries the argument

The audit instrument is a set of four realism conditions applied by manual inspection: the buggy code must match the source, the fix must match the source, the training data must match the source, and the training data must be realistic rather than random filler. Faults that pass are then executed repeatedly, with crashes required to appear in the buggy version and disappear in the fixed version, and low-performance symptoms required to show a statistically significant improvement across twenty runs as judged by a generalized linear model and an effect-size threshold. This combination of source-correspondence checks and repeated execution is what supports the headline percentages.

What would settle it

Take the faults that failed the realism audit and compare each against an archived version of its source, separating cases where the source lacks information from cases where the benchmark changed the code. If most mismatches vanish once incomplete sources are set aside, the 18.5% figure overstates the problem; if most mismatches are benchmark-side alterations, the figure stands.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the label 'real fault' attached to benchmark entries is largely unsupported by the evidence. After manually comparing the buggy code, the fix, and the training data of each fault against its stated source, only 58 of 314 faults met all four realism conditions, and reproduction attempts succeeded for only 86 of 165 eligible faults, with 79 of those stable across twenty runs. The mismatches are not rare exceptions: for two benchmarks, nearly half of the buggy versions or fixes do not match their sources, and most analysed faults come from forum posts that often omit the full code and dataset information needed for faithful reproduction.

Load-bearing premise

The audit treats the source post or commit as complete and authoritative ground truth for the fault and its fix, even though such posts often omit code and dataset details.

Editorial extensions

If this is right

  • Benchmark-based evaluations of deep-learning testing, localisation, and repair tools likely overstate how well those tools handle real faults.
  • The pool of genuinely real and reproducible faults across the five benchmarks is small, so benchmark diversity is narrower than the total bug count suggests.
  • Fault types are heavily skewed, with only 36 fault types represented among the realistic faults and three fault types accounting for a large share of all fault components.
  • Future fault extraction should apply stricter inclusion criteria, because many forum posts lack the full code and dataset information needed for faithful reproduction.
  • Higher-order faults, with an average order of 1.9 and some reaching order 12, are common, so mutation tools will need higher-order operators to approximate real faults.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the sources themselves are often incomplete, then 'does not match source' may partly reflect missing information in the original post rather than alteration by the benchmark authors; separating those two cases could change the 18.5% figure.
  • Even if many benchmark pairs are not faithful to their sources, they may still be usable as curated buggy-fixed pairs for tool evaluation, effectively serving as realistic mutants rather than true real faults.
  • A practical extension would be to turn the four-condition audit into a reusable checklist and a publicly maintained registry of verified faults, so new benchmarks can be certified before widespread adoption.
  • Reproduction under original pinned environments, for example using containers, would separate environment drift from fault realism; the reported 52% reproduction rate is an upper bound under current-version conditions.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper reports a manual audit of 490 faults from five deep learning fault benchmarks (TFBugs2018, DeepLocalize, DeepFD, Defects4ML, SFData). The authors define four realism conditions: the buggy code must match the code in the source (a Stack Overflow post or GitHub commit), the fix must match the source's fix, the training data must match the dataset mentioned in the source, and the training data must be realistic. After excluding 176 faults that lack usable sources or are not genuine DL faults, 314 faults are analyzed. Only 58 (18.5%) satisfy all four conditions. For the 165 faults that passed the realism conditions (with the two training-data conditions relaxed for SFData tensor-shape faults), reproduction attempts succeeded for 86 (52%), with 79 stable across 20 runs. The paper also analyzes the distribution of fault types using the taxonomy by Humbatova et al. and compares fault components with the mutation operators of DeepCrime, finding that only 22% of faults can be fully simulated by existing mutation operators.

Significance. If the 18.5% realism rate were taken at face value, this would be a major indictment of the five benchmarks and of all evaluations built on them, making the paper a landmark in the field. The paper makes several strong contributions: a systematic literature search with snowballing, a detailed manual procedure with pilot and consensus phases, a replication package with the analysis spreadsheet, and a set of reproducibility experiments with stability checks. The comparison with DeepCrime mutation operators is a useful addition to the mutation-testing literature. However, the headline rate is load-bearing and is currently overstated, because it counts 'source does not specify the dataset' as a failure of the benchmark and because it applies training-data conditions to tensor-shape faults that the paper itself excludes when reproducing. With suitable sensitivity analyses the paper could still support a strong qualitative conclusion, but the specific number 18.5% needs to be re-derived or heavily qualified. The contribution is nevertheless valuable and likely to influence how the community interprets DL fault benchmarks.

major comments (3)
  1. [Section 5.2 and Section 7.1.2] The 'training data matches source' condition is unsatisfiable when the source post does not state which dataset was used; the paper's own analysis indicates this is the dominant cause of training-data mismatches (Section 5.2) and that Stack Overflow posts 'very often do not contain the full code and information on the used training dataset' (Section 7.1.2). For such faults, any training data chosen by the benchmark authors will by construction fail condition (3), so the mismatch measure conflates 'source omits information' with 'benchmark is unrealistic.' The abstract's 18.5% figure therefore needs to be accompanied by a sensitivity analysis that treats missing source dataset information as 'unknown' rather than 'non-matching,' or that reports the realism rate on the subset of faults whose sources actually specify a dataset.
  2. [Section 4.2.5 and Table 5] The paper relaxes the two training-data conditions for SFData when attempting reproduction, arguing that 'the usage of realistic training data is not relevant to the reproduction of such faults,' yet these conditions are retained in the headline realism count. For SFData, 123 of 143 faults have buggy and fixed code matching the source, but only 16 satisfy all four conditions, with the gap driven by training-data conditions that the paper itself deems irrelevant for this class. Dropping the training-data conditions for SFData, as the paper does for reproduction, yields exactly 165 of 314 (52.5%) faults satisfying the remaining conditions, not 18.5%. The paper should recompute the all-conditions realism rate under the same relaxation used for reproduction (or at least report both rates) and should present the resulting number as the primary result if the qualitative conclusion changes materially.
  3. [Section 4.3] No inter-rater reliability statistic is reported for the manual classification, although the pilot and consensus meetings are described. Because the central findings depend on subjective judgments (e.g., whether a textual fix suggestion matches the applied fix, whether a dataset is 'real,' whether a code difference is 'syntactic' or changes model structure), the absence of a quantitative agreement measure (e.g., Cohen's kappa) weakens the replicability of the audit. The authors should provide such a statistic computed on the pilot or on a random sample, or justify why it is not applicable.
minor comments (5)
  1. [Table 4] In the TFBugs2018 row, the 'RSB R.' column reports '0/5 (60%)', which is internally inconsistent; the percentage should be 0%.
  2. [Section 5.1] The RQ1 summary states that 'bugs and fixes in the benchmarks match the sources 81.2% and 78% of the time,' but Table 2 shows an overall 'Buggy m.' of 254/314 = 80.9%, not 81.2%; the 81.2% figure appears to be a typo.
  3. [Abstract and Section 5.5] The statement that reproduction attempts were 'successful only in 52% of cases' refers to the 165 bugs selected for reproduction attempts, not to all 314 eligible faults; this should be stated explicitly to avoid misinterpretation.
  4. [Section 7.1.2] The sentence 'As there is a low rate of match between the buggy version and the initial source code in SO, our assumption that the information in the posts is very often not sufficient to reproduce the reported problem' is grammatically incomplete; it should be rephrased, for example by splitting the thought into two sentences.
  5. [Section 2.2] The date on which the Scopus search was performed is not reported; adding the search date would improve the reproducibility of the systematic literature search.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the 18.5% realism rate is a transparent operational count; only minor self-citations in RQ3/RQ4, not load-bearing.

full rationale

This paper is an empirical audit of external DL fault benchmarks, not a derivation. The 18.5% figure is the count of 314 eligible faults that satisfy four explicitly stated realism conditions (Section 1 and Sections 4.2.1-4.2.2); it is a direct operationalization, and the paper reports per-condition numbers in Table 2, so readers can recompute and disaggregate. The manual comparisons to SO posts and GitHub commits are external evidence, and the reproduction attempts are independent executions. The only self-references that appear in the analysis are the DL fault taxonomy [10] and the DeepCrime mutation tool [26,27] used for RQ3/RQ4 (fault-type distribution and simulability by mutants). These are prior published artifacts, and they do not feed into the headline realism count; RQ3/RQ4 are auxiliary descriptive questions. The skeptical concern that training-data conditions are unsatisfiable when SO posts omit dataset information, and that SFData tensor-shape faults are counted against conditions the paper itself relaxes for reproduction, is a validity/correctness challenge to the operationalization, not a circular derivation: the paper does not hide the relaxation (Section 4.2.5) and reports enough data to recompute the rate without those conditions. Consequently, there are no load-bearing circular steps; score 2 reflects the minor self-citation in RQ3/RQ4, not a circular chain.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The study is empirical, not derivational; there are no fitted free parameters or invented entities. The 18.5% statistic rests on the domain assumptions listed above: the conditions operationalize realism, external sources are authoritative ground truth, and the taxonomy/DeepCrime tool represent the space of DL faults and mutations.

assumptions (3)
  • domain assumption The four realism conditions (source-code match, fix match, training-data match, training-data realism) are an appropriate operationalization of fault realism.
    Section 4.2 defines these conditions; the central 18.5% result is measured against them, so the statistic is conditional on this definition.
  • domain assumption SO posts and GitHub commits are a faithful and complete record of the original fault and fix.
    Section 4.2.1 and 4.2.2 compare benchmark versions to these sources; if sources are incomplete, mismatches may not indicate reduced realism.
  • domain assumption The Humbatova et al. taxonomy and DeepCrime mutation operators represent the relevant space of DL faults and mutations.
    Section 4.2.3 uses the taxonomy and Section 4.2.4 uses DeepCrime, both from the authors' prior work; RQ3/RQ4 results depend on this.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Real Faults in Deep Learning Fault Benchmarks: How Real Are They?." pith.science (2026). https://pith.science/paper/237PV7YZ

@misc{pith2026241216336,
  author       = {Pith},
  title        = {Pith review of: Real Faults in Deep Learning Fault Benchmarks: How Real Are They?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/237PV7YZ}},
  note         = {Machine review of arXiv:2412.16336}
}
read the original abstract

As the adoption of Deep Learning (DL) systems continues to rise, an increasing number of approaches are being proposed to test these systems, localise faults within them, and repair those faults. The best attestation of effectiveness for such techniques is an evaluation that showcases their capability to detect, localise and fix real faults. To facilitate these evaluations, the research community has collected multiple benchmarks of real faults in DL systems. In this work, we perform a manual analysis of 490 faults from five different benchmarks and identify that 314 of them are eligible for our study. Our investigation focuses specifically on how well the bugs correspond to the sources they were extracted from, which fault types are represented, and whether the bugs are reproducible. Our findings indicate that only 18.5% of the faults satisfy our realism conditions. Our attempts to reproduce these faults were successful only in 52% of cases.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 26 canonical work pages

  1. [1]

    Supporting controlled experimentation with testing techniques: An infrastructure and its potential impact,

    H. Do, S. Elbaum, and G. Rothermel, “Supporting controlled experimentation with testing techniques: An infrastructure and its potential impact,” Empirical Software Engineering, vol. 10, pp. 405–435, 2005

    work page 2005

  2. [2]

    Defects4j: A database of existing faults to enable controlled testing studies for java programs,

    R. Just, D. Jalali, and M. D. Ernst, “Defects4j: A database of existing faults to enable controlled testing studies for java programs,” in Proceedings of the 2014 international symposium on software testing and analysis, 2014, pp. 437–440

    2014

  3. [3]

    Bugsjs: a benchmark of javascript bugs,

    P. Gyimesi, B. Vancsics, A. Stocco, D. Mazinanian, A. Beszédes, R. Ferenc, and A. Mesbah, “Bugsjs: a benchmark of javascript bugs,” in2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST). IEEE, 2019, pp. 90–101

    work page 2019

  4. [4]

    Deepfd: automated fault diagnosis and localization for deep learning programs,

    J. Cao, M. Li, X. Chen, M. Wen, Y . Tian, B. Wu, and S.-C. Cheung, “Deepfd: automated fault diagnosis and localization for deep learning programs,” ser. ICSE ’22, IEEE. New York, NY , USA: Association for Computing Machinery, 2022, p. 573–585. [Online]. Available: https://doi.org/10.1145/3510003.3510099

    arXiv 2022

  5. [5]

    Automatic fault detection for deep learning programs using graph transformations,

    A. Nikanjam, H. B. Braiek, M. M. Morovati, and F. Khomh, “Automatic fault detection for deep learning programs using graph transformations,” ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 31, no. 1, pp. 1–27, 2021

    work page 2021

  6. [6]

    Deepdiagnosis: automatically diagnosing faults and recommending actionable fixes in deep learning programs,

    M. Wardat, B. D. Cruz, W. Le, and H. Rajan, “Deepdiagnosis: automatically diagnosing faults and recommending actionable fixes in deep learning programs,” in Proceedings of the 44th international conference on software engineering, 2022, pp. 561–572

    work page 2022

  7. [7]

    Bugs in machine learning-based systems: a faultload benchmark,

    M. M. Morovati, A. Nikanjam, F. Khomh, and Z. M. Jiang, “Bugs in machine learning-based systems: a faultload benchmark,” Empirical Software Engineering, vol. 28, no. 3, p. 62, 2023

    work page 2023

  8. [8]

    gdefects4dl: a dataset of general real-world deep learning program defects,

    Y . Liang, Y . Lin, X. Song, J. Sun, Z. Feng, and J. S. Dong, “gdefects4dl: a dataset of general real-world deep learning program defects,” in Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings, 2022, pp. 90–94

    work page 2022

Show all 32 references
  1. [9]

    Iso/pas 21448:2019 road vehicles — safety of the intended functionality

    I. O. for Standardization, “Iso/pas 21448:2019 road vehicles — safety of the intended functionality.” https: //www.iso.org/standard/70939.html, 2019

    2019

  2. [10]

    Taxonomy of real faults in deep learning systems,

    N. Humbatova, G. Jahangirova, G. Bavota, V . Riccio, A. Stocco, and P. Tonella, “Taxonomy of real faults in deep learning systems,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, ser. ICSE ’20. New York, NY , USA: Association for Computin...

    2020

  3. [11]

    Guidelines for snowballing in systematic literature studies and a replication in software engineering,

    C. Wohlin, “Guidelines for snowballing in systematic literature studies and a replication in software engineering,” in Proceedings of the 18th international conference on evaluation and assessment in software engineering, 2014, pp. 1–10

    2014

  4. [12]

    Automatically repairing tensor shape faults in deep learning programs,

    D. Wu, B. Shen, Y . Chen, H. Jiang, and L. Qiao, “Automatically repairing tensor shape faults in deep learning programs,” Information and Software Technology, vol. 151, p. 107027, 2022. 14 Real Faults in Deep Learning Fault Benchmarks: How Real Are They? A PREPRINT

    2022

  5. [13]

    Tensfa: detecting and repairing tensor shape faults in deep learning systems,

    ——, “Tensfa: detecting and repairing tensor shape faults in deep learning systems,” in 2021 IEEE 32nd International Symposium on Software Reliability Engineering (ISSRE). IEEE, 2021, pp. 11–21

    2021

  6. [14]

    Repairing dnn architecture: Are we there yet?

    J. Kim, N. Humbatova, G. Jahangirova, P. Tonella, and S. Yoo, “Repairing dnn architecture: Are we there yet?” in 2023 IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE, 2023, pp. 234–245

    2023

  7. [15]

    Detecting numerical bugs in neural network architectures,

    Y . Zhang, L. Ren, L. Chen, Y . Xiong, S.-C. Cheung, and T. Xie, “Detecting numerical bugs in neural network architectures,” in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2020,...

    2020

  8. [16]

    An empirical study on tensorflow program bugs,

    Y . Zhang, Y . Chen, S.-C. Cheung, Y . Xiong, and L. Zhang, “An empirical study on tensorflow program bugs,” in Proceedings of the 27th ACM SIGSOFT international symposium on software testing and analysis, 2018, pp. 129–140

    2018

  9. [17]

    Tensorfuzz: Debugging neural networks with coverage- guided fuzzing,

    A. Odena, C. Olsson, D. Andersen, and I. Goodfellow, “Tensorfuzz: Debugging neural networks with coverage- guided fuzzing,” in International Conference on Machine Learning. PMLR, 2019, pp. 4901–4911

    2019

  10. [18]

    TensorFlow Model Garden,

    H. Yu, C. Chen, X. Du, Y . Li, A. Rashwan, L. Hou, P. Jin, F. Yang, F. Liu, J. Kim, and J. Li, “TensorFlow Model Garden,” https://github.com/tensorflow/models, 2020

    2020

  11. [19]

    Exposing numerical bugs in deep learning via gradient back-propagation,

    M. Yan, J. Chen, X. Zhang, L. Tan, G. Wang, and Z. Wang, “Exposing numerical bugs in deep learning via gradient back-propagation,” in Proceedings of the 29th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering,...

    2021

  12. [20]

    Reliability assurance for deep neural network architectures against numerical defects,

    L. Li, Y . Zhang, L. Ren, Y . Xiong, and T. Xie, “Reliability assurance for deep neural network architectures against numerical defects,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023, pp. 1827–1839

    2023

  13. [21]

    Detecting tensorflow program bugs in real-world industrial environment,

    C. Liu, J. Lu, G. Li, T. Yuan, L. Li, F. Tan, J. Yang, L. You, and J. Xue, “Detecting tensorflow program bugs in real-world industrial environment,” in2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2021, pp. 55–66

    2021

  14. [22]

    Autotrainer: An automatic dnn training problem detection and repair system,

    X. Zhang, J. Zhai, S. Ma, and C. Shen, “Autotrainer: An automatic dnn training problem detection and repair system,” in 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 2021, pp. 359–371

    2021

  15. [23]

    Cleanml: A study for evaluating the impact of data cleaning on ml classification tasks,

    P. Li, X. Rao, J. Blase, Y . Zhang, X. Chu, and C. Zhang, “Cleanml: A study for evaluating the impact of data cleaning on ml classification tasks,” in 2021 IEEE 37th International Conference on Data Engineering (ICDE). IEEE, 2021, pp. 13–24

    2021

  16. [24]

    Deeplocalize: Fault localization for deep neural networks,

    M. Wardat, W. Le, and H. Rajan, “Deeplocalize: Fault localization for deep neural networks,” in2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 2021, pp. 251–262

    2021

  17. [25]

    Repairing deep neural networks: Fix patterns and challenges,

    M. J. Islam, R. Pan, G. Nguyen, and H. Rajan, “Repairing deep neural networks: Fix patterns and challenges,” in Proceedings of the ACM/IEEE 42nd international conference on software engineering, 2020, pp. 1135–1146

    2020

  18. [26]

    Deepcrime: mutation testing of deep learning systems based on real faults,

    N. Humbatova, G. Jahangirova, and P. Tonella, “Deepcrime: mutation testing of deep learning systems based on real faults,” in ISSTA ’21: 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, Denmark, July 11-17, 2021, 2021, pp. 67–78

    2021

  19. [27]

    Deepcrime: from real faults to mutation testing tool for deep learning,

    ——, “Deepcrime: from real faults to mutation testing tool for deep learning,” in 2023 IEEE/ACM 45th Inter- national Conference on Software Engineering: Companion Proceedings (ICSE-Companion). IEEE, 2023, pp. 68–72

    2023

  20. [28]

    DeepMutation++: A Mutation Testing Framework for Deep Learning Systems,

    Q. Hu, L. Ma, X. Xie, B. Yu, Y . Liu, and J. Zhao, “DeepMutation++: A Mutation Testing Framework for Deep Learning Systems,” in 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2019, pp. 1158–1161. [Online]. Available: https://doi.org/...

    2019

  21. [29]

    Generalized linear models,

    J. A. Nelder and R. W. Wedderburn, “Generalized linear models,”Journal of the Royal Statistical Society: Series A (General), vol. 135, no. 3, pp. 370–384, 1972. [Online]. Available: https://doi.org/10.2307/2344614

    1972 doi

  22. [30]

    On effect size

    K. Kelley and K. J. Preacher, “On effect size.” Psychological methods, vol. 17, no. 2, p. 137, 2012. [Online]. Available: https://doi.org/10.1037/a0028086

    2012 doi

  23. [31]

    Replication package,

    G. Jahangirova, N. Humbatova, J. Kim, S. Yoo, and T. Paolo, “Replication package,” https://anonymous.4open. science/r/HowRealAreThey-D68E, 2024

    2024

  24. [32]

    Towards enhancing the reproducibility of deep learning bugs: an empirical study,

    M. B. Shah, M. M. Rahman, and F. Khomh, “Towards enhancing the reproducibility of deep learning bugs: an empirical study,”arXiv preprint arXiv:2401.03069, 2024. 15

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.