Pith. sign in

REVIEW 4 major objections 4 minor 77 references

Improved Detection and Diagnosis of Faults in Deep Neural Networks Using Hierarchical and Explainable Classification

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DEFault reports 94.30% accuracy at detecting faulty DNN programs on a real-world benchmark, and 63.46% accuracy at diagnosing their fault categories, surpassing the prior state of the art on both tasks.

desk verdict Solid engineering contribution with a large mutation dataset and strong internal results, but the real-world benchmark numbers rest on an unexamined overlap assumption that needs an explicit check. read the letter →

arxiv 2501.12560 v1 pith:AMB7LLMN submitted 2025-01-22 cs.SE

Sigma Jahan , Mehil B Shah , Parvez Mahbub , Mohammad Masudur Rahman This is my paper · ORCID
classification cs.SE
keywords DNNfaultdetectionhierarchicalclassificationRandomForestSHAPmutationtestingdynamicanalysisstaticdeeplearningbugs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DEFault is a two-stage fault-analysis technique for deep neural network programs. It claims that runtime ('dynamic') features captured during model training, fed through a hierarchy of Random Forest classifiers, can first decide whether a program is faulty and then assign one or more of seven fault categories from the literature. A second stage uses SHAP on static source-code features (layer counts, neuron counts, shapes) to explain the root cause of layer faults. Trained on 14,652 mutated DNN programs and tested on a 52-program real-world benchmark, it reports 94.30% fault-detection accuracy and 63.46% diagnosis accuracy, beating the previously best tool, DeepFD, by 3.92 and 11.54 percentage points respectively. If those numbers hold, DEFault is the first tested tool to cover every major DNN fault category, including weights and regularization, in one automated pipeline.

What carries the argument

The load-bearing machinery is a three-level hierarchy of Random Forest classifiers trained on 23 dynamic features (six newly proposed: activation saturation, adjusted learning rate, CPU, GPU and memory utilization, plus related runtime signals) captured per epoch during training. Level 1 detects fault presence; Level 2 runs seven binary classifiers to flag each of the seven fault categories, since a program can have several faults; Level 3 drills into hyperparameter root causes with four more binary classifiers. A separate explainer module trains a Random Forest on five families of static features and uses SHAP to rank which architectural property drives the layer-fault verdict. The training set itself is produced by an extended DeepCrime mutation framework that adds ten layer-focused mutation operators and RNN support, then labels mutants by the isKilled() accuracy-comparison criterion.

What would settle it

A reader could test the benchmark overlap by checking the 52 benchmark program URLs or code against the 60 reproduced StackOverflow programs; if even a handful overlap or are near-duplicates, the comparison is contaminated. A stronger test is to collect a new set of faulty DNN programs not mutated from the 60 bases, run DEFault's already-trained classifiers on them, and see whether detection accuracy stays near 94% and diagnosis near 63%.

Watch

Extended reading notes

Core claim

DEFault's central discovery is that dynamic and static information do different jobs in DNN fault analysis and are best kept in separate stages. Dynamic features alone (97% accuracy on the test set) identify whether training is going wrong and which category it belongs to; static features alone are poor detectors (62.52%) but are exactly what is needed to say which layer property is at fault. The paper shows that a hierarchy of binary Random Forest classifiers—one for fault detection, seven for the fault categories, four for hyperparameter sub-causes—handles multi-fault programs without the data-imbalance problems of a single multilabel model, and that a SHAP explainer over static features returns the correct layer root cause in the top five ranked features for six of seven benchmark layer faults.

Load-bearing premise

The reported accuracies assume that the 14,652 mutants generated from 60 StackOverflow programs and labelled by comparing each mutant's accuracy to the original model's are representative of real-world DNN faults, and that the 52-program benchmark does not substantially overlap with those 60 base programs; if either assumption fails, the 94% and 63% numbers are inflated.

Editorial extensions

If this is right

  • Developers get a single tool that detects faults and names the category for all seven categories in the standard DNN-fault taxonomy, including weights and regularization faults that DeepFD does not support.
  • The six new dynamic features (adjusted learning rate, activation saturation, hardware utilization, and related signals) contribute about a 4-point accuracy gain on detection (97% versus 93% in the ablation).
  • The hierarchical design means programs with multiple simultaneous faults are handled by running seven independent binary classifiers rather than a multilabel model, avoiding severe class imbalance.
  • For layer faults, the SHAP-based explainer gives developers a ranked list of static architectural features to inspect, with the correct root cause in the top five for six of the seven benchmark layer faults.
  • Static features alone are too weak for detection (62.52% accuracy) but essential for diagnosis, which argues that future tools should keep detection and explanation roles separate.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the mutant-based training labels generalize, this points toward a general recipe: learn detection from runtime traces and explanation from code structure, rather than trying to make one representation do both.
  • A natural next test is whether the same hierarchy transfers to attention-based architectures (the paper lists this as future work), since the static feature set currently targets layer counts and shapes common to FFNN, CNN, and RNN models.
  • The SHAP explainer is evaluated only for layer faults; extending it to weights and regularization faults, or to hyperparameter root causes, would test whether static features can explain those categories too.
  • Because static features are cheap to extract, the explainer could in principle be bolted onto any existing fault detector, decoupling 'is it broken?' from 'why is it broken?'.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents DEFault, a hierarchical random-forest technique that uses dynamic training features to detect and categorize faults in DNN programs and a SHAP-based explainer on static features to analyze layer-fault root causes. The authors construct a training set of 14,652 mutants from 60 StackOverflow programs, report internal detection accuracy of 97% and categorization accuracy of 92% on a random split, and report 94.30% detection and 63.46% categorization on DeepFD's 52-program real-world benchmark, outperforming four baselines. A PixelCNN case study illustrates the method.

Significance. If the benchmark numbers are reliable, DEFault is a meaningful advance: it is the only evaluated tool covering seven fault categories including layer and weight faults, it adds RNN support, and it combines static and dynamic analysis with an interpretable explanation step. The replication package, manual validation with Cohen's kappa, and explicit falsifiable performance claims are assets. The significance is conditional, however, because the central external-validity question—whether training and benchmark programs are truly disjoint—is not addressed in the paper.

major comments (4)
  1. [V-A, IV-B] The 94.30% detection and 63.46% categorization results on the 52-program benchmark (Table VII) require that the benchmark is effectively disjoint from the 60 base programs used to generate the 14,652 training mutants. Section IV-B describes the 60 base programs as reproduced from StackOverflow, while Section V-A identifies the benchmark only as DeepFD's 52 real-world faulty models 'obtained from StackOverflow and GitHub.' The paper reports no check for exact or near-duplicate programs between these two sets, and Section VII's claim that the representativeness threat is minimal does not cover this overlap. If any benchmark program or a close variant appears among the 60 bases, the Level-1 and Level-2 classifiers could memorize base-specific dynamic signatures, inflating the benchmark numbers. Please report a program-level overlap analysis (e.g., code similarity or manual inspection) and rerun the benchmark after excluding any overlapping bases.
  2. [V-A, Tables IV-VI] The internal evaluation uses a random 70-15-15 split of the 14,652 mutants (Section V-A), not a split grouped by base program. Because all mutants from the same 60 base programs can appear in both training and testing partitions, the high internal accuracy in Tables IV-VI and the RQ2 ablation (detection accuracy falling from 97% to 93% when the five novel features are removed) may partly reflect leakage of base-program-specific training dynamics rather than generalizable fault detection. Please repeat the evaluation with a grouped split, e.g., leave-one-base-program-out or a cluster split by base program, and report whether the internal and feature-ablation results are stable.
  3. [V-D4, Table VIII] The real-world root-cause evaluation for layer faults is based on only seven benchmark programs. The reported Top@1 accuracy of 57.10% is 4 out of 7, and Top@5 accuracy of 85.71% is 6 out of 7; with this sample size the root-cause claim is fragile and the 63.46% benchmark categorization figure in Table VII subsumes a much wider assertion than the evidence supports. Please either expand the layer-fault benchmark or explicitly present the RCA results as preliminary, with confidence intervals.
  4. [VII, Tables IV-VIII] The paper states that experiments were repeated 15 times to mitigate randomness (Section VII), but all reported tables give point estimates without standard deviations, confidence intervals, or significance tests against baselines. Given the stochastic nature of DNN training and the use of random forest and mutation sampling, the lack of variance information makes it difficult to assess whether the reported improvements over DeepFD (Table VII) are reliable. Please report distributions or confidence intervals, and where applicable statistical tests for the comparisons.
minor comments (4)
  1. [VI-C] Section VI-C contains a typo, 'DEEault's loss and optimization classifiers,' which should be 'DEFault's.'
  2. [VI-D] Section VI-D uses 'PixcelCNN' instead of 'PixelCNN'; please correct this typo.
  3. [I, V-D2] The contribution list in Section I states that the five novel features improve fault prediction 'by ≈ 5.6% in terms of recall,' but RQ2 (Section V-D2) reports an accuracy drop from 97% to 93% and does not report recall; please align the claimed metric with the reported experiment.
  4. [III] Section III says DEFault supports 'seven main categories out of eight from the DL fault categories,' while Table I and the text list seven categories; this wording should be clarified.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DEFault's central empirical claims rest on an external benchmark and supervised learning with independent features, not on a self-referential derivation.

full rationale

The paper's central claims—94.30% fault-detection and 63.46% fault-categorization on the 52-program benchmark—are empirical supervised-learning results, not derived equivalences. The training labels are produced by the isKilled() criterion applied to mutants, while the benchmark is DeepFD's external 52 real-world faulty programs ([27]); the comparison baselines are run from their official replication packages. No fitted parameter is later renamed as a prediction, and no equation or construction makes the benchmark outcome equal to the training labels. The self-citations present ([10], [38], [30]) support motivation, StackOverflow filtering criteria, and the replication package; they are not load-bearing for the claimed detection/diagnosis numbers. I flag as a validity concern, not as circularity, that the paper does not report an overlap analysis between the 60 StackOverflow base programs used for mutant generation and the 52 StackOverflow/GitHub benchmark programs; if such overlap existed, it would inflate the benchmark numbers, but the paper nowhere establishes that equivalence by construction. The self-reported limitation in Section VI-C (misclassification of an optimization fault due to overlapping features) is an honest empirical limitation, not a circular step. Accordingly, the derivation chain is self-contained against external evidence and the circularity score is 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The causal chain from mutant labels to real-world fault claims rests on assumptions about representativeness of mutation operators, killability labels, and benchmark disjointness. These are domain assumptions, not mathematically derived facts.

free parameters (4)
  • Random Forest hyperparameters = not reported
    Grid search with 5-fold cross-validation over parameters such as max depth and max features is used for each level classifier; the fitted values are not listed in the paper.
  • Activation saturation thresholds = predefined but unspecified
    Saturated activation is computed against thresholds chosen by the authors; the numeric thresholds are not given in Section IV-E2 or Table III.
  • isKilled statistical thresholds = not specified
    Generalized Linear Model and Cohen's d thresholds determine whether a mutant is labelled faulty; these thresholds are not reported in Section IV-D.
  • Mutation parameter bounds = e.g., CNN kernel sizes 1-7, average of 3 faults per mutant
    Hand-chosen bounds used to generate the training mutants; these choices shape the distribution of faults the classifier sees and are described in Section IV-D.
assumptions (5)
  • domain assumption DeepCrime and the extended mutation operators generate faults that match the real-world DNN fault taxonomy.
    Section IV-D relies on this taxonomy to justify the mutation operators and the seven fault categories used for training labels.
  • domain assumption The isKilled() accuracy-comparison criterion correctly separates faulty from correct mutants.
    Section IV-D labels mutants as faulty only when their accuracy distribution is statistically significantly worse, but this proxy is not independently validated.
  • domain assumption The 60 reproduced StackOverflow programs provide a representative base population for generating diverse mutants.
    Section IV-A and IV-B describe the manual collection and reproduction process; the threats section only asserts rigor and does not demonstrate representativeness.
  • domain assumption The 52-program benchmark is disjoint from the 60 base programs used for mutation, or any overlap is negligible.
    Section V-A introduces the benchmark from DeepFD without listing program IDs or providing an overlap analysis with the training base programs.
  • domain assumption SHAP feature rankings on static features correspond to true root causes for layer faults.
    Section IV-G and V-D4 use SHAP to rank static features and compare against ground truth on only 7 layer faults, which is a weak validation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improved Detection and Diagnosis of Faults in Deep Neural Networks Using Hierarchical and Explainable Classification." pith.science (2026). https://pith.science/paper/AMB7LLMN

@misc{pith2026250112560,
  author       = {Pith},
  title        = {Pith review of: Improved Detection and Diagnosis of Faults in Deep Neural Networks Using Hierarchical and Explainable Classification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AMB7LLMN}},
  note         = {Machine review of arXiv:2501.12560}
}
read the original abstract

Deep Neural Networks (DNN) have found numerous applications in various domains, including fraud detection, medical diagnosis, facial recognition, and autonomous driving. However, DNN-based systems often suffer from reliability issues due to their inherent complexity and the stochastic nature of their underlying models. Unfortunately, existing techniques to detect faults in DNN programs are either limited by the types of faults (e.g., hyperparameter or layer) they support or the kind of information (e.g., dynamic or static) they use. As a result, they might fall short of comprehensively detecting and diagnosing the faults. In this paper, we present DEFault (Detect and Explain Fault) -- a novel technique to detect and diagnose faults in DNN programs. It first captures dynamic (i.e., runtime) features during model training and leverages a hierarchical classification approach to detect all major fault categories from the literature. Then, it captures static features (e.g., layer types) from DNN programs and leverages explainable AI methods (e.g., SHAP) to narrow down the root cause of the fault. We train and evaluate DEFault on a large, diverse dataset of ~14.5K DNN programs and further validate our technique using a benchmark dataset of 52 real-life faulty DNN programs. Our approach achieves ~94% recall in detecting real-world faulty DNN programs and ~63% recall in diagnosing the root causes of the faults, demonstrating 3.92% - 11.54% higher performance than that of state-of-the-art techniques. Thus, DEFault has the potential to significantly improve the reliability of DNN programs by effectively detecting and diagnosing the faults.

Figures

Figures reproduced from arXiv: 2501.12560 by the authors.

Figure 1
Figure 1. Schematic diagram of dataset preparation and classifier model training [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Distribution of fault types in our datasets [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Types of DNN faults and their root causes [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Workflow of DEFault: (a) Fault detection, (b) Fault categorization, and (c) Root cause analysis [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Feature Impact on Faulty DNN Program Detection using SHAP [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: (a) Loss-Accuracy (b) Gradient Norms during PixelCNN training [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

77 extracted references · 74 canonical work pages

  1. [1]

    Methods for interpreting and understanding deep neural networks,

    G. Montavon, W. Samek, and K.-R. M ¨uller, “Methods for interpreting and understanding deep neural networks,” Digital signal processing, vol. 73, pp. 1–15, 2018

    work page 2018

  2. [2]

    Deep learning approach for intelligent financial fraud detection system,

    A. M. Mubalaike and E. Adali, “Deep learning approach for intelligent financial fraud detection system,” in UBMK, IEEE, 2018, pp. 598–603

    work page 2018

  3. [3]

    Explaining software bugs leveraging code structures in neural machine translation,

    P. Mahbub, O. Shuvo, and M. M. Rahman, “Explaining software bugs leveraging code structures in neural machine translation,” in Proceedings of 2023 IEEE/ACM 45th Inter- national Conference on Software Engineering (ICSE) , 2023, pp. 640–652

    work page 2023

  4. [4]

    Predicting line-level defects by capturing code contexts with hierarchical transformers,

    P. Mahbub and M. M. Rahman, “Predicting line-level defects by capturing code contexts with hierarchical transformers,” in 2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) , 2024, pp. 308–319

    work page 2024

  5. [5]

    A guide to deep learning in healthcare,

    A. Esteva, A. Robicquet, B. Ramsundar, et al. , “A guide to deep learning in healthcare,” Nature medicine, vol. 25, no. 1, pp. 24–29, 2019

    work page 2019

  6. [6]

    Sphereface: Deep hypersphere embedding for face recogni- tion,

    W. Liu, Y . Wen, Z. Yu, M. Li, B. Raj, and L. Song, “Sphereface: Deep hypersphere embedding for face recogni- tion,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 212–220

    work page 2017

  7. [7]

    A survey of deep learning techniques for autonomous driving,

    S. Grigorescu, B. Trasnea, T. Cocias, and G. Macesanu, “A survey of deep learning techniques for autonomous driving,” Journal of field robotics , vol. 37, no. 3, pp. 362–386, 2020

    work page 2020

  8. [8]

    Understanding software- 2.0: A study of machine learning library usage and evolution,

    M. Dilhara, A. Ketkar, and D. Dig, “Understanding software- 2.0: A study of machine learning library usage and evolution,” TOSEM, vol. 30, no. 4, pp. 1–42, 2021

    work page 2021

Show all 77 references
  1. [9]

    Deep learning & software engineering: State of research and future directions,

    P. Devanbu, M. Dwyer, S. Elbaum, et al., “Deep learning & software engineering: State of research and future directions,” arXiv preprint arXiv:2009.08525 , 2020

    2009 arXiv

  2. [10]

    Towards under- standing the challenges of bug localization in deep learning systems,

    S. Jahan, M. B. Shah, and M. M. Rahman, “Towards under- standing the challenges of bug localization in deep learning systems,” arXiv preprint arXiv:2402.01021 , 2024

    2024 arXiv

  3. [11]

    Jaguar: A spectrum-based fault localization tool for real-world software,

    H. L. Ribeiro, R. P. de Araujo, M. L. Chaim, H. A. de Souza, and F. Kon, “Jaguar: A spectrum-based fault localization tool for real-world software,” in ICS, IEEE, 2018, pp. 404–409

    2018

  4. [12]

    Mahbub, Comprehending software bugs leveraging code structures with neural language models

    P. Mahbub, Comprehending software bugs leveraging code structures with neural language models. Master’s thesis, 2023

    2023

  5. [13]

    Artificial neural networks based optimization techniques: A review,

    M. G. Abdolrasol, S. S. Hussain, T. S. Ustun, et al., “Artificial neural networks based optimization techniques: A review,” Electronics, vol. 10, no. 21, p. 2689, 2021

    2021

  6. [14]

    An empirical study on tensorflow program bugs,

    Y . Zhang, Y . Chen, S.-C. Cheung, Y . Xiong, and L. Zhang, “An empirical study on tensorflow program bugs,” in ISSTA, 2018, pp. 129–140

    2018

  7. [15]

    A comprehen- sive study on deep learning bug characteristics,

    M. J. Islam, G. Nguyen, R. Pan, and H. Rajan, “A comprehen- sive study on deep learning bug characteristics,” inESEC/FSE, 2019, pp. 510–520

    2019

  8. [16]

    Taxonomy of real faults in deep learning systems,

    N. Humbatova, G. Jahangirova, G. Bavota, V . Riccio, A. Stocco, and P. Tonella, “Taxonomy of real faults in deep learning systems,” in ICSE, 2020, pp. 1110–1121

    2020

  9. [17]

    Testing feedforward neural networks training programs,

    H. Ben Braiek and F. Khomh, “Testing feedforward neural networks training programs,” TOSEM, vol. 32, no. 4, pp. 1–61, 2023

    2023

  10. [18]

    Bugs in machine learning-based systems: A faultload bench- mark,

    M. M. Morovati, A. Nikanjam, F. Khomh, and Z. M. Jiang, “Bugs in machine learning-based systems: A faultload bench- mark,” EMSE, vol. 28, no. 3, p. 62, 2023

    2023

  11. [19]

    Toward un- derstanding deep learning framework bugs,

    J. Chen, Y . Liang, Q. Shen, J. Jiang, and S. Li, “Toward un- derstanding deep learning framework bugs,” TOSEM, vol. 32, no. 6, pp. 1–31, 2023

    2023

  12. [20]

    Detecting numerical bugs in neural network archi- tectures,

    Y . Zhang, L. Ren, L. Chen, Y . Xiong, S.-C. Cheung, and T. Xie, “Detecting numerical bugs in neural network archi- tectures,” in ESEC/FSE, 2020, pp. 826–837

    2020

  13. [21]

    Deeplocalize: Fault lo- calization for deep neural networks,

    M. Wardat, W. Le, and H. Rajan, “Deeplocalize: Fault lo- calization for deep neural networks,” in ICSE, IEEE, 2021, pp. 251–262

    2021

  14. [22]

    Automatic fault detection for deep learning programs using graph transformations,

    A. Nikanjam, H. B. Braiek, M. M. Morovati, and F. Khomh, “Automatic fault detection for deep learning programs using graph transformations,” TOSEM, vol. 31, no. 1, pp. 1–27, 2021

    2021

  15. [23]

    Umlaut: Debugging deep learning programs using program structure and model behavior,

    E. Schoop, F. Huang, and B. Hartmann, “Umlaut: Debugging deep learning programs using program structure and model behavior,” in Proceedings of the 2021 CHI conference on human factors in computing systems , 2021, pp. 1–16

    2021

  16. [24]

    Autotrainer: An automatic dnn training problem detection and repair system,

    X. Zhang, J. Zhai, S. Ma, and C. Shen, “Autotrainer: An automatic dnn training problem detection and repair system,” in ICSE, IEEE, 2021, pp. 359–371

    2021

  17. [25]

    Deepdiagnosis: Automatically diagnosing faults and recommending actionable fixes in deep learning programs,

    M. Wardat, B. D. Cruz, W. Le, and H. Rajan, “Deepdiagnosis: Automatically diagnosing faults and recommending actionable fixes in deep learning programs,” in ICSE, 2022, pp. 561–572

    2022

  18. [26]

    Deepcrime: Mutation testing of deep learning systems based on real faults,

    N. Humbatova, G. Jahangirova, and P. Tonella, “Deepcrime: Mutation testing of deep learning systems based on real faults,” in ISSTA, 2021, pp. 67–78

    2021

  19. [27]

    Deepfd: Automated fault diagnosis and localization for deep learning programs,

    J. Cao, M. Li, X. Chen, et al. , “Deepfd: Automated fault diagnosis and localization for deep learning programs,” in ICSE, 2022, pp. 573–585

    2022

  20. [28]

    Cradle: Cross- backend validation to detect and localize bugs in deep learning libraries,

    H. V . Pham, T. Lutellier, W. Qi, and L. Tan, “Cradle: Cross- backend validation to detect and localize bugs in deep learning libraries,” in ICSE, IEEE, 2019, pp. 1027–1038

    2019

  21. [29]

    Deepmutation: Mutation testing of deep learning systems,

    L. Ma, F. Zhang, J. Sun, et al. , “Deepmutation: Mutation testing of deep learning systems,” in ISSRE, IEEE, 2018, pp. 100–111

    2018

  22. [30]

    science/r/ICSE 2025-F42C, Accessed: 2024-11-29, 2025

    Anonymous, Icse 2025 repository , https://anonymous.4open. science/r/ICSE 2025-F42C, Accessed: 2024-11-29, 2025

    2025

  23. [31]

    Nerdbug: Automated bug detection in neural networks,

    F. Jafarinejad, K. Narasimhan, and M. Mezini, “Nerdbug: Automated bug detection in neural networks,” in Proceedings of the 1st ACM International Workshop on AI and Software Testing/Analysis, 2021, pp. 13–16

    2021

  24. [32]

    Tensfa: Detecting and repairing tensor shape faults in deep learning systems,

    D. Wu, B. Shen, Y . Chen, H. Jiang, and L. Qiao, “Tensfa: Detecting and repairing tensor shape faults in deep learning systems,” in ISSRE, IEEE, 2021, pp. 11–21

    2021

  25. [33]

    Mode: Automated neural network model debugging via state dif- ferential analysis and input selection,

    S. Ma, Y . Liu, W.-C. Lee, X. Zhang, and A. Grama, “Mode: Automated neural network model debugging via state dif- ferential analysis and input selection,” in ESEC/FSE, 2018, pp. 175–186

    2018

  26. [34]

    Exposing numerical bugs in deep learning via gradient back- propagation,

    M. Yan, J. Chen, X. Zhang, L. Tan, G. Wang, and Z. Wang, “Exposing numerical bugs in deep learning via gradient back- propagation,” in ESEC/FSE, 2021, pp. 627–638

    2021

  27. [35]

    Mutation-based fault localization of deep neural networks,

    A. Ghanbari, D.-G. Thomas, M. A. Arshad, and H. Rajan, “Mutation-based fault localization of deep neural networks,” in ASE, Luxembourg, Luxembourg, 2023, pp. 1301–1313. DOI: 10.1109/ASE56229.2023.00171

    2023

  28. [36]

    Deepfault: Fault localization for deep neural networks,

    H. F. Eniser, S. Gerasimou, and A. Sen, “Deepfault: Fault localization for deep neural networks,” in International Con- ference on Fundamental Approaches to Software Engineering, Cham: Springer International Publishing, 2019, pp. 171–189

    2019

  29. [37]

    An effective data-driven approach for localizing deep learning faults,

    M. Wardat, B. D. Cruz, W. Le, and H. Rajan, “An effective data-driven approach for localizing deep learning faults,” arXiv preprint arXiv:2307.08947 , 2023

    2023 arXiv

  30. [38]

    Towards enhanc- ing the reproducibility of deep learning bugs: An empirical study,

    M. B. Shah, M. M. Rahman, and F. Khomh, “Towards enhanc- ing the reproducibility of deep learning bugs: An empirical study,” Empirical Software Engineering, vol. 30, no. 1, p. 23, 2025

    2025

  31. [39]

    Meta-analysis of cohen’s kappa,

    S. Sun, “Meta-analysis of cohen’s kappa,” Health Services and Outcomes Research Methodology , vol. 11, pp. 145–163, 2011

    2011

  32. [40]

    Goodfellow, Y

    I. Goodfellow, Y . Bengio, and A. Courville, Deep Learn- ing. MIT Press, 2016. [Online]. Available: http : / / www . deeplearningbook.org

    2016

  33. [41]

    Johnson, Convolutional Neural Networks for Visual Recog- nition

    J. Johnson, Convolutional Neural Networks for Visual Recog- nition. Stanford University, Lecture Notes, 2017. [Online]. Available: http://cs231n.stanford.edu/

    2017

  34. [42]

    Understanding the effect size and its measures.,

    C. Ialongo, “Understanding the effect size and its measures.,” Biochemia medica, vol. 26, no. 2, pp. 150–163, 2016

    2016

  35. [43]

    Alternatives to p value: Confidence interval and effect size,

    D. K. Lee, “Alternatives to p value: Confidence interval and effect size,” Korean journal of anesthesiology , vol. 69, no. 6, pp. 555–562, 2016

    2016

  36. [44]

    Deep- mutation++: A mutation testing framework for deep learning systems,

    Q. Hu, L. Ma, X. Xie, B. Yu, Y . Liu, and J. Zhao, “Deep- mutation++: A mutation testing framework for deep learning systems,” in ASE, IEEE, 2019, pp. 1158–1161

    2019

  37. [45]

    How to kill them all: An exploratory study on the impact of code observability on mutation testing,

    Q. Zhu, A. Zaidman, and A. Panichella, “How to kill them all: An exploratory study on the impact of code observability on mutation testing,” JSS, vol. 173, p. 110 864, 2021

    2021

  38. [46]

    Automatic mlp weight regularization on min- eralization prediction tasks,

    A. Skabar, “Automatic mlp weight regularization on min- eralization prediction tasks,” in Knowledge-Based Intelligent Information and Engineering Systems: 9th International Con- ference, KES 2005, Melbourne, Australia, September 14-16, 2005, Proceedings, Part III , vol. 9, Spr...

    2005 doi

  39. [47]

    High- dimensional dynamics of generalization error in neural net- works,

    M. S. Advani, A. M. Saxe, and H. Sompolinsky, “High- dimensional dynamics of generalization error in neural net- works,” Neural Networks, vol. 132, pp. 428–446, 2020. DOI: 10.1016/j.neunet.2020.09.010

    2020 doi

  40. [48]

    Deepsniffer: A dnn model extraction framework based on learning architectural hints,

    X. Hu, L. Liang, S. Li, et al. , “Deepsniffer: A dnn model extraction framework based on learning architectural hints,” in Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, 2020, pp. 385–399

    2020

  41. [49]

    A comprehensive study of deep learning compiler bugs,

    Q. Shen, H. Ma, J. Chen, Y . Tian, S.-C. Cheung, and X. Chen, “A comprehensive study of deep learning compiler bugs,” in ESEC/FSE, 2021, pp. 968–980

    2021

  42. [50]

    Understanding deep learning (still) requires rethinking gen- eralization,

    C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, “Understanding deep learning (still) requires rethinking gen- eralization,” Communications of the ACM , vol. 64, no. 3, pp. 107–115, 2021

    2021

  43. [51]

    Understanding neural networks via feature visualization: A survey,

    A. Nguyen, J. Yosinski, and J. Clune, “Understanding neural networks via feature visualization: A survey,” Explainable AI: interpreting, explaining and visualizing deep learning, pp. 55– 76, 2019

    2019

  44. [52]

    Practical recommendations for gradient-based training of deep architectures,

    Y . Bengio, “Practical recommendations for gradient-based training of deep architectures,” in Neural networks: Tricks of the trade: Second edition , Springer, 2012, pp. 437–478

    2012

  45. [53]

    Cyclical learning rates for training neural networks,

    L. N. Smith, “Cyclical learning rates for training neural networks,” in WACV, IEEE, 2017, pp. 464–472

    2017

  46. [54]

    A high- throughput screening approach to discovering good forms of biologically inspired visual representation,

    N. Pinto, D. Doukhan, J. J. DiCarlo, and D. D. Cox, “A high- throughput screening approach to discovering good forms of biologically inspired visual representation,” PLoS computa- tional biology, vol. 5, no. 11, e1000579, 2009

    2009

  47. [55]

    Understanding the difficulty of training deep feedforward neural networks,

    X. Glorot and Y . Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in Proceedings of the thirteenth international conference on artificial in- telligence and statistics , JMLR Workshop and Conference Proceedings, 2010, pp. 249–256

    2010

  48. [56]

    Hierar- chical classification,

    P. Arabie, L. Hubert, G. De Soete, and A. Gordon, “Hierar- chical classification,” P . Arabie, L. Hubert, G. De Soete, & A. Gordon, Clustering and classification , pp. 65–121, 1996

    1996

  49. [57]

    Random forest,

    S. J. Rigatti, “Random forest,” Journal of Insurance Medicine, vol. 47, no. 1, pp. 31–39, 2017

    2017

  50. [58]

    Random forests,

    L. Breiman, “Random forests,” Machine learning , vol. 45, pp. 5–32, 2001

    2001

  51. [59]

    Performance comparison of grid search and random search methods for hyperparameter tuning in extreme gradient boosting algorithm to predict chronic kidney failure.,

    D. A. Anggoro and S. S. Mukti, “Performance comparison of grid search and random search methods for hyperparameter tuning in extreme gradient boosting algorithm to predict chronic kidney failure.,” International Journal of Intelligent Engineering & Systems , vol. 14, no. 6, 2021

    2021

  52. [60]

    Random search for hyper- parameter optimization.,

    J. Bergstra and Y . Bengio, “Random search for hyper- parameter optimization.,” JMLR, vol. 13, no. 2, 2012

    2012

  53. [61]

    Explainable artificial intelligence: An analytical review,

    P. P. Angelov, E. A. Soares, R. Jiang, N. I. Arnold, and P. M. Atkinson, “Explainable artificial intelligence: An analytical review,” Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, vol. 11, no. 5, e1424, 2021

    2021

  54. [62]

    Failure mode and effects analysis of rc members based on machine-learning- based shapley additive explanations (shap) approach,

    S. Mangalathu, S.-H. Hwang, and J.-S. Jeon, “Failure mode and effects analysis of rc members based on machine-learning- based shapley additive explanations (shap) approach,” Engi- neering Structures, vol. 219, p. 110 927, 2020

    2020

  55. [63]

    Cross-validation metrics for evaluating classification performance on imbalanced data,

    N. W. S. Wardhani, M. Y . Rochayani, A. Iriany, A. D. Sulistyono, and P. Lestantyo, “Cross-validation metrics for evaluating classification performance on imbalanced data,” in IC3INA, IEEE, 2019, pp. 14–18

    2019

  56. [64]

    Layer-wise learning based stochastic gradient descent method for the optimization of deep convolutional neural network,

    Q. Zheng, X. Tian, N. Jiang, and M. Yang, “Layer-wise learning based stochastic gradient descent method for the optimization of deep convolutional neural network,” Journal of Intelligent & Fuzzy Systems , vol. 37, no. 4, pp. 5641–5654, 2019

    2019

  57. [65]

    Reltanh: An activation function with vanishing gradient resistance for sae-based dnns and its application to rotating machinery fault diagnosis,

    X. Wang, Y . Qin, Y . Wang, S. Xiang, and H. Chen, “Reltanh: An activation function with vanishing gradient resistance for sae-based dnns and its application to rotating machinery fault diagnosis,” Neurocomputing, vol. 363, pp. 88–98, 2019

    2019

  58. [66]

    G. H. Dunteman and M.-H. R. Ho, An introduction to gener- alized linear models . Sage Publications, 2005

    2005

  59. [67]

    From local explanations to global understanding with explainable ai for trees,

    S. M. Lundberg, G. Erion, H. Chen, et al. , “From local explanations to global understanding with explainable ai for trees,” Nature machine intelligence , vol. 2, no. 1, pp. 56–67, 2020

    2020

  60. [68]

    A unified optimization approach for sparse tensor operations on gpus,

    B. Liu, C. Wen, A. D. Sarwate, and M. M. Dehnavi, “A unified optimization approach for sparse tensor operations on gpus,” in CLUSTER, IEEE, 2017, pp. 47–57

    2017

  61. [69]

    The general inefficiency of batch training for gradient descent learning,

    D. R. Wilson and T. R. Martinez, “The general inefficiency of batch training for gradient descent learning,” Neural networks, vol. 16, no. 10, pp. 1429–1451, 2003

    2003

  62. [70]

    On locality of local explanation models,

    S. Ghalebikesabi, L. Ter-Minassian, K. DiazOrdaz, and C. C. Holmes, “On locality of local explanation models,” Advances in neural information processing systems, vol. 34, pp. 18 395– 18 407, 2021

    2021

  63. [71]

    Technologies, Tensorflow 2 published models , https : / / github.com/sarus-tech/tf2-published-models, Accessed: 2024- 11-21, 2023

    S. Technologies, Tensorflow 2 published models , https : / / github.com/sarus-tech/tf2-published-models, Accessed: 2024- 11-21, 2023

    2024

  64. [72]

    Gradient- based learning applied to document recognition,

    Y . LeCun, L. Bottou, Y . Bengio, and P. Haffner, “Gradient- based learning applied to document recognition,” Proceedings of the IEEE , vol. 86, no. 11, pp. 2278–2324, 1998

    1998

  65. [73]

    Cnn model for image classification on mnist and fashion-mnist dataset,

    S. S. Kadam, A. C. Adamuthe, and A. B. Patil, “Cnn model for image classification on mnist and fashion-mnist dataset,” Journal of scientific research , vol. 64, no. 2, pp. 374–384, 2020

    2020

  66. [74]

    An analysis of the softmax cross entropy loss for learning-to- rank with binary relevance,

    S. Bruch, X. Wang, M. Bendersky, and M. Najork, “An analysis of the softmax cross entropy loss for learning-to- rank with binary relevance,” in Proceedings of the 2019 ACM SIGIR international conference on theory of information retrieval, 2019, pp. 75–78

    2019

  67. [75]

    A hitchhiker’s guide to statis- tical tests for assessing randomized algorithms in software engineering,

    A. Arcuri and L. Briand, “A hitchhiker’s guide to statis- tical tests for assessing randomized algorithms in software engineering,” Software Testing, Verification and Reliability , vol. 24, no. 3, pp. 219–250, 2014

    2014

  68. [76]

    On construct validity: Issues of method and measurement.,

    G. T. Smith, “On construct validity: Issues of method and measurement.,” Psychological assessment , vol. 17, no. 4, p. 396, 2005

    2005

  69. [77]

    External validity,

    M. G. Findley, K. Kikuta, and M. Denly, “External validity,” Annual Review of Political Science , vol. 24, no. 1, pp. 365– 393, 2021

    2021

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.