Pith. sign in

REVIEW 6 major objections 6 minor 1 cited by

An Empirical Study of Fault Localisation Techniques for Deep Learning

T0 review · 6 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Single-ground-truth evaluation undervalues DL fault localisation tools; counting equivalent fixes lifts best recall from 0.31 to 0.61.

desk verdict Useful first third-party comparison of DL fault-localisation tools, but the headline recall gain from neutrality analysis is an optimistic max-over-alternatives bound, not a typical performance figure. read the letter →

arxiv 2412.11304 v2 pith:XQK5HOTE submitted 2024-12-15 cs.SE cs.AIcs.LG

Nargiz Humbatova , Jinhan Kim , Gunel Jahangirova , Shin Yoo , Paolo Tonella This is my paper · ORCID
classification cs.SEcs.AIcs.LG
keywords deeplearningfaultlocalisationempiricalstudyneutralityanalysisalternativegroundtruthsDNNdebuggingmutationtestingFD
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the standard way of judging deep-learning fault localisation tools, checking whether the tool's reported fault matches one human-defined fix, undervalues the tools. On a benchmark of 22 faulty models, 9 real and 13 mutant-generated, the four state-of-the-art tools reach at most 0.31 average recall and 0.23 precision against that single ground truth. When the authors run a neutrality analysis that searches for many alternative fixes with equal or better model accuracy, the picture changes: the best tool, DeepFD, reaches 0.61 average recall and 0.41 precision. The paper's central claim is that evaluation protocols should count all equivalent patches, not just the one a human happened to write.

What carries the argument

The neutrality analysis is the mechanism responsible for the central result. It is a breadth-first search that starts from a faulty model, mutates one hyperparameter at a time (activation function, initialisers, loss function, optimiser, learning rate, epochs, batch size, layer structure, or neuron count), retrains each neighbour ten times, and keeps neighbours whose mean accuracy is equal to or better than the parent's accuracy. Models that match or beat the known human fix are added as alternative ground truths. The paper also uses recall, precision, and $F_3$ (weighting recall three times more than precision) to score tools against the original and the extended ground-truth sets.

What would settle it

Re-run the same comparison but give every fault the same number of alternative fixes, for example ten randomly sampled neutral neighbours, and require each claimed match to reproduce a performance gain on a held-out test set; if the best tool's recall no longer approaches 0.61, the reported improvement is an artefact of the variable and large alternative sets.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims to show that existing DL fault localisation tools are more effective than a single-ground-truth evaluation suggests, and that the conventional evaluation is therefore misleading. Using a breadth-first search over hyperparameter mutations to build a neutrality network, the authors find between 1 and 291 alternative fixes per fault, with an average of 69 for artificial faults and 28 for real faults. Recalculating performance against these alternative fixes raises DeepFD's average recall from 0.31 to 0.61 and precision from 0.19 to 0.41, with Neuralint also improving substantially, while DeepDiagnosis and Umlaut improve much less. The paper concludes that any empirical evaluation of DL fault localisation should include some form of neutrality analysis, while acknowledging that even with it the recall values remain far from perfect.

Load-bearing premise

The large recall increase rests on counting a tool as successful when its reported fault matches at least one of the automatically generated alternative fixes, where 'equivalent' is defined by mean accuracy from ten retrainings.

Editorial extensions

If this is right

  • Judging DL fault localisation tools against a single human fix underestimates them and can mark correct alternative suggestions as wrong.
  • Future DL fault-localisation evaluations should report results both on the original ground truth and on an extended ground truth obtained by neutrality analysis.
  • DeepFD is the most effective tool in this comparison, but also the slowest because it retrains the model 20 times; Neuralint is the efficient static alternative.
  • Even with alternative fixes counted, the best average recall is 0.61, so DL fault localisation remains an open problem.
  • Tool rankings are largely stable across the two evaluation modes, with DeepFD significantly ahead of DeepDiagnosis and Umlaut.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the number of alternative fixes varies from 1 to 291 across faults, part of the recall gain may simply reflect the size of the alternative set; a fairer comparison would fix the number of alternatives per fault or weight them.
  • The neutrality-analysis methodology could transfer to other evaluation problems in deep learning, such as repair, hyperparameter optimisation, or test adequacy, wherever 'correct' has multiple equivalent outcomes.
  • A testable prediction follows: if tools are given credit for matching any one of many mutants, tool ordering may change on larger benchmarks; a stricter metric that requires a tool to propose the specific repaired configuration would likely keep recall lower.
  • One could directly validate the alternative fixes by checking on held-out data whether each matched 'equivalent' patch actually restores performance, rather than relying on mean training accuracy from the search.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. The paper presents an empirical evaluation of four state-of-the-art fault localisation tools for deep learning programs (DeepFD, DeepDiagnosis, Neuralint, and Umlaut) on a benchmark of 22 faulty models, 13 with artificially injected faults and 9 real faults curated from bug-reporting platforms. The evaluation uses precision, recall, and F3, first against a single ground truth and then against a set of alternative ground truths produced by a BFS-based neutrality analysis. The main finding is that under a single ground truth all tools achieve low effectiveness (maximum average recall of 0.31), but when alternative ground truths are considered, recall increases substantially, with DeepFD achieving the highest average recall of 0.61. The paper also investigates the stability, efficiency, and actionability of the tools' outputs.

Significance. This is the first third-party systematic comparison of these four DL fault localisation tools, and the authors contribute a curated benchmark and an augmented dataset with extended ground truths. The idea of considering multiple equivalent repairs in FL evaluation is novel and important for the community. The paper ships a replication package, which is a strength. However, the central effectiveness claim depends on a methodological choice—taking the maximum recall over up to 291 alternative ground truths—that is not adequately justified and can inflate scores regardless of tool quality. Without a random-output baseline or a well-defined aggregation rule, the quantitative ranking of tools is not trustworthy as it stands. The qualitative analyses of stability, efficiency, and output actionability are useful and less affected by this issue.

major comments (6)
  1. [§5.2, Tables 10–11] The post-neutrality effectiveness scores are computed by taking, for each fault, the maximum RC across all alternative ground truths (the number of alternatives ranges from 1 to 291 per fault, as shown in Table 9), and then averaging these maxima. This is a max-over-k protocol: the score is an upper bound that increases with the number of alternatives and with the number of fault types in a tool's output. A tool that outputs several common fault types will match at least one of the many search-generated alternatives purely by chance. To support the headline claim that recall grows from 0.31 to 0.61, the paper must report per-fault alternative counts, include a random-output control (or a control that outputs a fixed set of common fault types), and report an expected recall (e.g., the average over all alternative GTs) rather than the maximum. Without such a baseline, the reported improvement cannot be attributed to the tools' localisation skill.
  2. [§3, Algorithm 1 and footnote 1] The neutrality criterion is defined as 'equivalent or higher fitness' based on the mean accuracy of ten trained instances, with no statistical test or tolerance threshold. As a result, many nodes in the neutrality graph may have accuracy that is not meaningfully different from the buggy model, and they may not represent genuine repairs of the root-cause fault. The paper should validate that the alternative GTs indeed repair the fault as intended, e.g., by requiring a statistically significant improvement over the buggy model (as is done for the 'blue' nodes) or by reporting a sensitivity analysis with stricter neutrality thresholds. Currently, the alternative GTs used in Table 10 may include incidental hyperparameter changes that are unrelated to the actual fault, making the FL evaluation overly permissive.
  3. [§5.2, Table 10] The paper states that it reports 'the maximum RC observed across all ground truth variants and the average PR and F3 calculated on these GTs,' but it is not specified which GT is used to compute PR and F3 when multiple alternatives yield the same maximum recall. If PR and F3 are computed for the GT that maximises RC, then precision is also optimistically biased, and the values such as M1-DFD RC=0.67, PR=0.5, F3=0.65 are not directly comparable to the pre-neutrality metrics. The definition of the aggregated PR and F3 must be made explicit, and the same rule must be applied consistently to all tools, preferably by fixing an aggregation rule (e.g., average over all GTs) rather than a per-fault maximum.
  4. [§5.1, Tables 4–7] The preprocessing step that removes 'softmax' suggestions from DeepDiagnosis, Umlaut, and Neuralint, and overfitting warnings from Umlaut, is a subjective filter that can change the ranking. For example, in Table 7, Umlaut's output for M2 includes 'ACH(7), HLR', and the ACH match is retained; but for many other issues the softmax suggestion is the only one, and removing it makes the output empty. The paper should justify the filtering with a reproducible criterion and show how sensitive the results are to this decision. Currently, the filtering is applied post hoc and may favour tools that happen to avoid such messages.
  5. [§5.2, statistical tests] The Wilcoxon signed-rank tests comparing DeepFD with DeepDiagnosis and with Umlaut are reported with p-values (0.002 and 0.023, respectively), but the paper does not specify what data were used (per-issue RC? F3? before or after neutrality?) or how many issues were included. Moreover, the difference between DeepFD and Neuralint is reported as not statistically significant (p=0.066), which weakens the abstract's claim that 'DeepFD is the most effective tool'. The authors should clarify the statistical procedure and moderate the ranking claim accordingly.
  6. [§4.2 and §5.1] The real-fault section of the benchmark is derived from the evaluation dataset of DeepFD, and DeepFD's detectable fault types were chosen based on that dataset's fault distribution. This is acknowledged in Section 5.1, but the consequence is that DeepFD has an inherent advantage on the real-fault subset, which is where its improvement over other tools is largest (e.g., RF Avg. RC after neutrality: DFD 0.50 vs. NL 0.22). To support the general claim about tool effectiveness, the authors should either re-analyse the two benchmark halves separately as the primary result, or obtain an independent set of real faults. At minimum, the headline average recall of 0.61 should be accompanied by a clear caveat about this overlap.
minor comments (6)
  1. [Abstract] There are grammatical issues in the abstract, e.g., 'increases also the need' and 'such figures increase when considering alternative, equivalent patches'. These should be corrected.
  2. [§4.3] The definition of the post-neutrality aggregation (maximum RC, and average PR/F3 'calculated on these GTs') should be introduced in the evaluation metrics section, not only in Section 5.2, to avoid ambiguity.
  3. [Table 9] The 'Improvement' column shows values such as 0.000 for M1–M3, which seems inconsistent with the text saying improvements are marginal; consider adding more digits or explaining the rounding.
  4. [Table 10] The greyed-out issues (R1, R3, D2, D9) are not visually distinct in the rendered table; consider a footnote or other notation to indicate where alternative GTs were not found.
  5. [§6.1] The construct-validity section should explicitly discuss the threats posed by the max-over-alternatives aggregation and the manual filtering of tool outputs, as these are the most significant measurement choices.
  6. [§5.5] The actionability analysis is qualitative; adding a small table summarising how often each tool provides layer-level information and actionable fix suggestions would make it easier to compare tools.

Circularity Check

2 steps flagged · score 6.0 of 10

Headline recall gain is entailed by the max-over-alternative-GTs scoring definition; DFD's real-fault advantage uses its own evaluation benchmark as ground truth.

  1. self definitional [Section 5.2 (RQ1 after neutrality analysis), Table 10 and Table 11; Abstract]
    "Table 10 shows results for each tool and issue that can be observed when using all the alternative ground truths, in addition to the original one. ... In this table we report the maximum RC observed across all ground truth variants and the average PR and F3 calculated on these GTs."

    After neutrality analysis, per-issue Recall is defined as the maximum RC over the original GT plus all BFS-generated alternative GTs, a set that by construction contains the original GT. Hence the 'after' RC is always at least the 'before' RC, and the paper's reported improvement (average RC 0.31 to 0.61) is an arithmetic consequence of taking a maximum over an enlarged set, not evidence that the tools localise alternative fixes better. As the number of alternatives ranges from 1 to 291 (Table 9), the headline mostly tracks the size of the alternative set; without a random-output baseline or a control for k, the 0.61 number does not demonstrate localisation skill.

  2. fitted input called prediction [Section 5.1, RQ1 before neutrality analysis; Table 8; Abstract]
    "This can be influenced by the fact that the 'Real Fault' part of the benchmark comes from the evaluation benchmark of DEEP FD, as this was the only available source of truly reproducible real faults. The selection of fault types that DEEP FD is trained to detect was indeed influenced by the distribution of faults in the evaluation benchmark, as described in the corresponding article [3]."

    DEEP FD's detectable fault types (HLR, ACH, LCH, HNE, OCH) were chosen from the fault distribution of the DeepFD evaluation benchmark, and the real-fault half of the present benchmark is exactly that same DeepFD dataset, later curated in the present authors' prior work [16]. The paper then reports that 'DEEP FD, on average, exhibits the best performance and significantly outperforms other tools on real faults' and uses this to support the Abstract claim that DEEP FD is 'the most effective tool'. This is a fitted input called a prediction: the tool's output vocabulary is fitted to the evaluation benchmark's fault distribution, and the benchmark is then used to confirm the tool.

full rationale

The artificial-fault experiments are independent and the per-tool output tables are honest data. However, the two central claims are partially circular. First, the headline improvement in recall after neutrality analysis is entailed by the protocol: the 'after' recall for each issue is the maximum RC over a set of ground truths that includes the original, so an increase is guaranteed by the definition of maximum; the magnitude (0.31 to 0.61) is inflated by alternative sets ranging from 1 to 291 per fault, and no random baseline or per-issue k control is provided. Second, DEEP FD's real-fault advantage is evaluated on real faults taken from DEEP FD's own evaluation benchmark, whose fault-type distribution determined the fault types DEEP FD was trained to detect; the paper discloses this but still lets the real-fault section drive the 'DEEP FD is the most effective tool' conclusion. Both issues affect the abstract, RQ1, and conclusion, so partial circularity (score 6) is appropriate. If the authors reported per-issue k, averaged RC over a fixed random selection of alternatives, and excluded or re-weighted the real faults that overlap DEEP FD's training distribution, the core comparison would be substantially more self-contained.

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

This is an empirical evaluation rather than a derivation, so the ledger records the experimental choices and borrowed artifacts on which the conclusions rest. The neutrality search parameters (topk, time budget, number of trainings) directly control how many alternative ground truths are found and therefore how high post-neutrality recall can go. The benchmark ground truths and tool implementations are taken from prior work without re-verification. The most consequential assumption is methodological: that matching any one of many alternative fixes is a fair measure of a tool's effectiveness. The paper introduces no physical or conceptual entities requiring independent evidence.

free parameters (7)
  • topk (neutrality search branching factor) = 5
    Controls how many neutral neighbours are enqueued each BFS step; chosen by hand in Section 4.3 and directly affects which alternative ground truths are found.
  • Stopping condition SC (time budget) = 48 hours
    Stops the neutrality search in Section 4.3; a longer budget would likely find more alternative ground truths and change the reported recall improvements.
  • Number of trainings for fitness estimation = 10
    Each model in neutrality analysis is trained 10 times and the mean accuracy is used as fitness (Algorithm 1, Section 3). This choice defines the neutrality relation and the set of alternative GTs.
  • Number of retrainings for DeepFD features and stability runs = 20
    DeepFD features are collected from 20 retrainings and other tools are run 20 times (Section 4.3). These counts affect the reported stability and runtime figures.
  • F-beta parameter for F3 = 3
    Recall is weighted three times more than precision in Section 4.3; this choice affects all effectiveness rankings.
  • Significance thresholds for neutrality graph coloring = p < 0.05, Cohen's d > 0.5
    Used in Section 3 to mark alternative GTs as statistically better; affects the counts of statistically significant alternatives, though inclusion as an alternative GT uses mean accuracy only.
  • Mutation operator values for neutrality analysis = learning rate x10 or /10; epochs x2 or /2; batch sizes 16,32,64,128,256,512
    Adopted from Kim et al. [16] in Table 2; defines the neighbourhood searched for alternative ground truths.
assumptions (6)
  • domain assumption Benchmark ground truths from DeepCrime and Kim et al. are correct
    The study's central measurements rely on the ground-truth labels in Table 3 being the true faults; the authors adopt these from prior work without independent verification.
  • domain assumption Mean accuracy of 10 trainings is a sufficient fitness estimate for neutrality
    Algorithm 1 and Section 3 use the mean of ten accuracy values to decide neutrality; this ignores variance beyond the mean except in node coloring.
  • ad hoc to paper Maximum recall across alternative ground truths is a valid effectiveness measure
    Section 5.2 reports maximum recall over all found ground truths; this is optimistic because a tool needs to match only one of many mutants to count as correct, and the number of alternatives varies by fault from 1 to 291.
  • ad hoc to paper Filtering tool outputs (softmax suggestions, overfitting warnings) is valid
    Section 5.1 excludes misleading suggestions before scoring; this post-hoc editing of tool output affects precision and recall and is not a standardised protocol.
  • domain assumption Public implementations of the four tools faithfully represent the published techniques
    The evaluation uses replication packages [17-20]; any deviation between those implementations and the published algorithms affects the conclusions.
  • domain assumption F3 with beta = 3 is the right metric for fault localisation effectiveness
    Section 4.3 justifies giving recall more weight, but the choice changes the relative ranking of tools.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An Empirical Study of Fault Localisation Techniques for Deep Learning." pith.science (2026). https://pith.science/paper/XQK5HOTE

@misc{pith2026241211304,
  author       = {Pith},
  title        = {Pith review of: An Empirical Study of Fault Localisation Techniques for Deep Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XQK5HOTE}},
  note         = {Machine review of arXiv:2412.11304}
}
read the original abstract

With the increased popularity of Deep Neural Networks (DNNs), increases also the need for tools to assist developers in the DNN implementation, testing and debugging process. Several approaches have been proposed that automatically analyse and localise potential faults in DNNs under test. In this work, we evaluate and compare existing state-of-the-art fault localisation techniques, which operate based on both dynamic and static analysis of the DNN. The evaluation is performed on a benchmark consisting of both real faults obtained from bug reporting platforms and faulty models produced by a mutation tool. Our findings indicate that the usage of a single, specific ground truth (e.g., the human defined one) for the evaluation of DNN fault localisation tools results in pretty low performance (maximum average recall of 0.31 and precision of 0.23). However, such figures increase when considering alternative, equivalent patches that exist for a given faulty DNN. Results indicate that \dfd is the most effective tool, achieving an average recall of 0.61 and precision of 0.41 on our benchmark.

Figures

Figures reproduced from arXiv: 2412.11304 by the authors.

Figure 1
Figure 1. An example neutrality network of D4 model by one hyperparameter. The details of mutation operators adopted from Kim et al. [16] are shown in [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Average execution time and average performance ( [PITH_FULL_IMAGE:figures/full_fig_p018_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Fault Localisation and Repair for DL Systems: An Empirical Study with LLMs

    cs.SE 2025-06 conditional novelty 6.0 of 10

    LLMs, especially GPT-4, outperform existing fault localisation and repair tools for deep learning models in accuracy, speed, and stability.

Reference graph

Works this paper leans on

31 extracted references · 21 canonical work pages · cited by 1 Pith paper

  1. [1]

    Deeplocalize: Fault localization for deep neural networks,

    M. Wardat, W. Le, and H. Rajan, “Deeplocalize: Fault localization for deep neural networks,” in2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). Los Alamitos, CA, USA: IEEE Computer Society, may 2021, pp. 251–262. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/ICSE43902.2021.00034

    arXiv 2021

  2. [2]

    DeepDiagnosis: automatically diagnosing faults and recommending actionable fixes in deep learning programs,

    M. Wardat, B. D. Cruz, W. Le, and H. Rajan, “DeepDiagnosis: automatically diagnosing faults and recommending actionable fixes in deep learning programs,” in Proceedings of the 44th International Conference on Software Engineering, 2022, pp. 561–572

    work page 2022

  3. [3]

    Deepfd: Automated fault diagnosis and localization for deep learning programs,

    J. Cao, M. Li, X. Chen, M. Wen, Y . Tian, B. Wu, and S.-C. Cheung, “Deepfd: Automated fault diagnosis and localization for deep learning programs,” in Proceedings of the 44th International Conference on Software Engineering, ser. ICSE ’22. New York, NY , USA: Association for Computing Machinery, 2022, p. 573–585. [Online]. Available: https://doi.org/10.11...

    arXiv 2022

  4. [4]

    Automatic fault detection for deep learning programs using graph transformations,

    A. Nikanjam, H. B. Braiek, M. M. Morovati, and F. Khomh, “Automatic fault detection for deep learning programs using graph transformations,” ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 31, no. 1, pp. 1–27, 2021

    2021

  5. [5]

    Umlaut: Debugging deep learning programs using program structure and model behavior,

    E. Schoop, F. Huang, and B. Hartmann, “Umlaut: Debugging deep learning programs using program structure and model behavior,” in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 2021, pp. 1–16

    work page 2021

  6. [6]

    Autotrainer: An automatic dnn training problem detection and repair system,

    X. Zhang, J. Zhai, S. Ma, and C. Shen, “Autotrainer: An automatic dnn training problem detection and repair system,” in 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), 2021, pp. 359–371

    work page 2021

  7. [7]

    Detect, fix, and verify tensorflow api misuses,

    W. Baker, M. O’Connor, S. R. Shahamiri, and V . Terragni, “Detect, fix, and verify tensorflow api misuses,” in International Conference on Software Analysis, Evolution and Reengineering, 2022, pp. 1–5

    work page 2022

  8. [8]

    “Keras,” Available at https://keras.io

    work page

Show all 31 references
  1. [9]

    An empirical study on tensorflow program bugs,

    Y . Zhang, Y . Chen, S.-C. Cheung, Y . Xiong, and L. Zhang, “An empirical study on tensorflow program bugs,” in Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, ser. ISSTA 2018. New York, NY , USA: ACM, 2018, pp. 129–140. [Online]. ...

    2018

  2. [10]

    An introduction to kernel and nearest-neighbor nonparametric regression,

    N. S. Altman, “An introduction to kernel and nearest-neighbor nonparametric regression,” The American Statistician, vol. 46, no. 3, pp. 175–185, 1992. [Online]. Available: http://www.jstor.org/stable/2685209

    1992

  3. [11]

    Breiman, Classification and regression trees

    L. Breiman, Classification and regression trees. Routledge, 2017

    2017

  4. [12]

    Random decision forests,

    T. K. Ho, “Random decision forests,” inProceedings of 3rd international conference on document analysis and recognition, vol. 1. IEEE, 1995, pp. 278–282

    1995

  5. [13]

    Deepcrime: Mutation testing of deep learning systems based on real faults,

    N. Humbatova, G. Jahangirova, and P. Tonella, “Deepcrime: Mutation testing of deep learning systems based on real faults,” in Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, ser. ISSTA 2021. New York, NY , USA: Association for Comp...

    2021

  6. [14]

    Taxonomy of real faults in deep learning systems,

    N. Humbatova, G. Jahangirova, G. Bavota, V . Riccio, A. Stocco, and P. Tonella, “Taxonomy of real faults in deep learning systems,” in Proceedings of the 41st International Conference on Software Engineering, ICSE, 2020

    2020

  7. [15]

    Neutrality and epistasis in program space,

    J. Renzullo, W. Weimer, M. Moses, and S. Forrest, “Neutrality and epistasis in program space,” inProceedings of the 4th International Workshop on Genetic Improvement Workshop, ser. GI ’18. New York, NY , USA: Association for Computing Machinery, 2018, p. 1–8. [Online]. Availab...

    2018

  8. [16]

    Repairing dnn architecture: Are we there yet?

    J. Kim, N. Humbatova, G. Jahangirova, P. Tonella, and S. Yoo, “Repairing dnn architecture: Are we there yet?” in 2023 IEEE Conference on Software Testing, Verification and Validation (ICST), 2023

    2023

  9. [17]

    Replication package of deepfd,

    J. Cao, M. Li, X. Chen, M. Wen, Y . Tian, B. Wu, and S.-C. Cheung, “Replication package of deepfd,” https: //github.com/ArabelaTso/DeepFD, 2021

    2021

  10. [18]

    Replication package of umlaut,

    E. Schoop, F. Huang, and B. Hartmann, “Replication package of umlaut,” Available at https://github.com/ BerkeleyHCI/umlaut, 2021

    2021

  11. [19]

    Replication package of Neuralint,

    A. Nikanjam, H. B. Braiek, M. M. Morovati, and F. Khomh, “Replication package of Neuralint,” Available at https://github.com/neuralint/neuralint

  12. [20]

    Replication package of DeepDiagnosis,

    M. Wardat, B. D. Cruz, W. Le, and H. Rajan, “Replication package of DeepDiagnosis,” Available at https: //github.com/deepdiagnosis/icse2022, 2021

    2021

  13. [21]

    An empirical evaluation of mutation operators for deep learning systems,

    G. Jahangirova and P. Tonella, “An empirical evaluation of mutation operators for deep learning systems,” inIEEE International Conference on Software Testing, Verification and Validation, ser. ICST’20. IEEE, 2020, p. 12 pages. [Online]. Available: https://doi.org/10.1109/ICST4...

    2020

  14. [22]

    Empirical Comparison of Fault Localisation Techniques for DNNs (replication package),

    “Empirical Comparison of Fault Localisation Techniques for DNNs (replication package),” Available at https: //zenodo.org/doi/10.5281/zenodo.10387015, 2023

    2023 doi

  15. [23]

    Hebo: Pushing the limits of sample-efficient hyper-parameter optimisation,

    A. I. Cowen-Rivers, W. Lyu, R. Tutunov, Z. Wang, A. Grosnit, R. R. Griffiths, A. M. Maraval, H. Jianye, J. Wang, J. Peters et al., “Hebo: Pushing the limits of sample-efficient hyper-parameter optimisation,” Journal of Artificial Intelligence Research, vol. 74, pp. 1269–1349, 2022

    2022

  16. [24]

    Bohb: Robust and efficient hyperparameter optimization at scale,

    S. Falkner, A. Klein, and F. Hutter, “Bohb: Robust and efficient hyperparameter optimization at scale,” in International Conference on Machine Learning. PMLR, 2018, pp. 1437–1446

    2018

  17. [25]

    Arachne: Search based repair of deep neural networks,

    J. Sohn, S. Kang, and S. Yoo, “Arachne: Search based repair of deep neural networks,”ACM Trans. Softw. Eng. Methodol., sep 2022. [Online]. Available: https://doi.org/10.1145/3563210

    2022 doi

  18. [26]

    Causality-based neural network repair,

    B. Sun, J. Sun, L. H. Pham, and J. Shi, “Causality-based neural network repair,” in Proceedings of the 44th International Conference on Software Engineering, 2022, pp. 338–349

    2022

  19. [27]

    Genmunn: A mutation-based approach to repair deep neural network models,

    H. Wu, Z. Li, Z. Cui, and J. Liu, “Genmunn: A mutation-based approach to repair deep neural network models,” International Journal of Modeling, Simulation, and Scientific Computing, p. 2341008, 2022

    2022

  20. [28]

    Repairing misclassifications in neural networks using limited data,

    P. Henriksen, F. Leofante, and A. Lomuscio, “Repairing misclassifications in neural networks using limited data,” in Proceedings of the 37th ACM/SIGAPP Symposium on Applied Computing, 2022, pp. 1031–1038

    2022

  21. [29]

    Provable repair of deep neural networks,

    M. Sotoudeh and A. V . Thakur, “Provable repair of deep neural networks,” in Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation, 2021, pp. 588–603

    2021

  22. [30]

    Nn repair: Constraint-based repair of neural network classifiers,

    M. Usman, D. Gopinath, Y . Sun, Y . Noller, and C. S. P˘as˘areanu, “Nn repair: Constraint-based repair of neural network classifiers,” in International Conference on Computer Aided Verification. Springer, 2021, pp. 3–25

    2021

  23. [31]

    Apricot: A weight-adaptation approach to fixing deep learning models,

    H. Zhang and W. Chan, “Apricot: A weight-adaptation approach to fixing deep learning models,” in2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2019, pp. 376–387. 20

    2019

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.