Pith. sign in

REVIEW 4 major objections 4 minor 62 references

Mock Deep Testing: Toward Separate Development of Data and Models for Deep Learning

T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that the data-preparation and model-design stages of a deep-learning program can be tested independently using automatically generated mock objects, catching bugs before expensive integrated training begins.

desk verdict The framework is genuinely new, but the headline counts mostly reflect static assertions rather than mock behavior; it still deserves refereeing with a required re-analysis. read the letter →

arxiv 2502.07712 v1 pith:QMKP7HCZ submitted 2025-02-11 cs.SE

Ruchira Manke , Mohammad Wardat , Foutse Khomh , Hridesh Rajan This is my paper · ORCID
classification cs.SE
keywords deeplearningtestingmockobjectsunitdatapreparationmodeldesignKerasfaultlocalization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the two tightly coupled stages of a deep-learning program — data preparation and model design — can be split into independent modules and unit-tested separately with mock objects, the same stand-in technique traditional unit testing uses for unavailable dependencies. It presents a framework, built for the Keras library, that automatically generates a simple three-layer mock model to test prepared data, and synthetic scaled mock data to test a designed model. The claim is that these lightweight stand-ins reproduce the abnormal-training symptoms of the real data and model, so developers can locate the error-inducing stage before any integrated training. On 50 real buggy programs the mocks identified 10 data-preparation issues and 53 model-design issues, and 36 developers using the framework resolved 25 and 38 of those issues respectively. If correct, this moves deep-learning debugging from post-integration training-time monitoring to cheap, early unit testing of each stage.

What carries the argument

The machinery is an interface-based decoupling of the deep-learning pipeline into two modules. The data-preparation module exposes an interface with the number of features, data type, and task; the model-design module exposes an interface with architecture and task. From these interfaces the framework automatically creates mocks: a mock model for data testing follows a decision table that sets hidden-layer width, output activation, loss, and metrics for the task, and a mock data generator creates scaled synthetic samples whose size was tuned by sensitivity analysis. Unit tests then combine structural property assertions with short training-behavior checks — is the mock model learning, is loss oscillating — run on the mock rather than on the full pipeline.

What would settle it

Take a corpus of buggy deep-learning programs whose faulty stage is known by inspection; run the data-stage test with the generated mock model and the model-stage test with generated mock data, and compare each reported symptom with the symptom of the integrated original program. If many known data bugs make the mock model train cleanly, or many known model bugs fail to trigger a mock-data symptom, the consistency assumption is refuted. A narrower check: in a program whose only defect is an overly high learning rate, the data-preparation unit test should pass on clean data; a NaN reported by the mock model there would be a false positive.

Watch

Extended reading notes

Core claim

The central claim is that a simple generated mock model — three layers, one hidden layer as wide as the feature count, with task-appropriate output activation and loss — behaves consistently enough with the original model to expose data defects, and that synthetic scaled mock data free of missing values and outliers behaves consistently enough with the original data to expose model defects. That consistency is the load-bearing observation: when the integrated program shows NaN loss, oscillating loss, or failure to learn, the same symptom appears in the isolated stage, letting the framework's assertions name the stage that caused it and suggest an actionable fix. The paper further claims that stage-isolated testing catches issues that integrated training-time diagnosis misses or resolves only after multiple retraining iterations, particularly in programs with bugs in both stages.

Load-bearing premise

The approach hinges on the assumption that a simple three-layer mock model and clean synthetic mock data reproduce the same training symptoms as the real model and real data, so that a symptom seen in isolation truly points to the stage that caused it.

Editorial extensions

If this is right

  • Developers can verify data quality with a three-layer mock model before committing to a full architecture, so missing values, bad scaling, and failure to learn are caught in the data module rather than during training.
  • Developers can verify a model's structure and hyperparameters on synthetic mock data before the real dataset is ready, catching shape mismatches, wrong activations, and inappropriate loss functions early.
  • Programs with bugs in both stages no longer require fixing data issues before model issues become visible, because each stage is tested separately; the paper reports this is where integrated training-time diagnosis struggles.
  • The 15 supported issue types are encoded as test assertions, and users can refine or add assertions, which the authors identify as the path for covering edge cases the current catalog misses.
  • Mock-based unit testing is lightweight compared with retraining the full model after each fix, saving time and computational resources across debugging iterations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test is to measure, fault by fault, whether the mock's training symptom actually matches the original model's symptom; the four participant-dismissed false alarms mark the boundary where this consistency can break.
  • The interface-and-mock recipe should transfer beyond the two supported architectures and structured data, but the harder cases are sequence and image models, where synthetic mock data must preserve structure the current generator does not model; one testable extension is mock data that keeps sequence length and vocabulary.
  • Because the framework encodes only 15 issue types, bugs outside that list — such as output-label scaling tied to a dataset's label range, which the paper reports missing — stay invisible unless users extend the assertions, so the approach's ceiling is set by the completeness of its assertion catalog.
  • Stage-isolated tests are cheap enough to run on every code change, which suggests continuous-integration testing for deep-learning pipelines as a practice the paper does not explicitly propose.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces "mock deep testing," a methodology for unit testing deep learning (DL) applications by decoupling data preparation from model design through interfaces and automatically generated mocks (mock data and mock models). It presents KUnit, a Keras-based framework implementing 15 assertion types for detecting data- and model-related bugs. The evaluation includes an empirical study over 50 DL programs from Stack Overflow and GitHub, reporting 10 data-preparation issues and 53 model-design issues detected by KUnit, and a user study with 36 participants who resolved 25 data-preparation and 38 model-design issues. The authors also compare KUnit with DeepDiagnosis and report survey results on perceived usefulness.

Significance. If the central claim holds, this paper makes a useful contribution by adapting the well-established mock-object testing pattern to DL pipelines, potentially enabling earlier and more localized bug detection. The paper offers a concrete design paradigm, an open-source framework (KUnit), and a documented set of assertion categories grounded in published fault taxonomies (Islam et al., Humbatova et al.). The empirical and user studies are substantial in size, and the replication package is a strength. However, the central claim that mocks—specifically behavior-on-mock checks—are responsible for the reported bug detections is not yet convincingly supported by the presented data, as detailed in the major comments. The paper also does not include statistical testing or an independent validation of the key consistency assumption, which limits the strength of the conclusions.

major comments (4)
  1. [§IV-B1, Table IV] The reported counts of 10 data-preparation and 53 model-design issues are aggregated across two qualitatively different kinds of checks: static/property-based assertions (e.g., missing scaling, wrong output activation, wrong loss function) and behavior-on-mock checks (e.g., oscillating loss, slow convergence, model not learning). Table IV shows that, for the model-design stage, only 3 of the 53 detected issues fall into the behavior-based categories (Oscillating Loss/Slow Convergence), while 50 fall into categories that are primarily checked by inspecting the model structure or API usage. For the data-preparation stage, all 10 detected issues are statically checkable categories; no behavior-based 'Model not Learning' detection is reported in Table IV. Thus, the paper's claim that 'mocks effectively identified' these issues conflates mock-based verification with interface/property-based checks that do not require mock objects. Please disaggregate the counts by whether detection requires the mock, and, if the mock is not required, state so explicitly. The central claim about mock-based independent testing would then rest on far fewer instances than the headline numbers suggest.
  2. [§III-B, §II] The paper's motivating premise is that a simple three-layer mock model and synthetic, scaled, outlier-free mock data behave consistently with the original model and data for the purpose of exposing bugs. This consistency assumption is load-bearing but is not empirically validated. For example, the paper does not report whether bugs detected on mocks are confirmed to be present in the original integrated program, nor whether the absence of mock-detected issues corresponds to the absence of real issues (false-alarm rate). The user study itself reports 4 dismissed false alarms (Table VI and §IV-B1), which is direct evidence that the consistency assumption can fail. I request a dedicated experiment that, for a set of programs with known injected/fixed bugs, compares issues detected by KUnit on mocks with issues detected by a full training run on original data, reporting precision and recall per category. Without this, the contribution's central mechanism remains unsubstantiated.
  3. [§IV-A3, §IV-B1, §V] The evaluation pipeline involves manual inspection of each of the 50 programs to split them into data-preparation and model-design stages and to count the number of issues. The paper notes this manual step as a threat to validity but does not report any inter-rater reliability or a second independent annotation. Since the reported counts (e.g., 74 total issues in Table IV) are the numerator for the effectiveness claim, the subjectivity of this annotation is directly relevant to the result. Please provide the annotation protocol, a second annotator, and a measure of agreement (e.g., Cohen's kappa), or otherwise justify that the splitting is unambiguous. Additionally, no statistical tests are reported for the comparisons (e.g., KUnit vs. DeepDiagnosis debugging times); given the small sample sizes, reporting effect sizes and confidence intervals would strengthen the empirical claims.
  4. [§IV-A1, §IV-B1] The assertion conditions are described as derived from published taxonomies and external studies, which is good. However, some conditions include thresholds that appear to be tuned on the same evaluation data: the mock-data sample sizes (10× features for regression, 100 per class for classification) and the loss-monitoring window of every 5 epochs. The paper mentions a sensitivity analysis but does not report its details or justify that the chosen thresholds are not overfit to the 50-program benchmark. I recommend reporting the sensitivity analysis results and, if possible, validating the thresholds on held-out programs or a synthetic bug-injection suite.
minor comments (4)
  1. [Throughout] There are several typos and formatting issues, e.g., 'DeepDignosis' in §IV-B2, 'DeepLocalize' in the related-work section where 'DeepLocalize' is the intended name, and the References section has inconsistently formatted author names (e.g., ref [17] 'Goodfellow, Y. Bengio' versus 'I. Goodfellow, Y. Bengio'). Please proofread.
  2. [Fig. 2 and Table IV] The mapping between the issue labels in Fig. 2 (A–H and 1–7) and the categories in Table IV is not always obvious. For example, Table IV lists 'Learning Rate out of Common Range' and 'Oscillating Loss/Slow Convergence' separately, while Fig. 2 lists 'F. Slow Convergence' and 'H. Oscillating Loss' as separate categories. Please align the labels so that the reader can trace which issues are behavior-based versus property-based.
  3. [§IV-A5] The paper states that participants were monitored via Zoom and could access the internet. It would be helpful to state explicitly how long each task session lasted and whether the debugging-time analysis includes the mock-generation overhead, as the comparison in §IV-B2 reports average times of 12 and 15 minutes without clarifying what is included.
  4. [§III-B1] The description of the mock model generation says it is a three-layer network, but Decision Table I lists conditions for 'hidden layer neurons' and 'output layer neurons' without specifying the number of hidden layers. For clarity, please state whether the mock model always has exactly one hidden layer and how the hidden-layer count relates to the original model's depth.

Circularity Check

0 steps flagged · score 0.0 of 10

No definitional circularity; KUnit's mock-based testing is an empirical contribution, though the reported detection counts partly over-attribute static interface assertions to mock behavior.

full rationale

KUnit's derivation chain is not circular. The mock model and mock data are generated from interface definitions (Decision Table I, Figs. 3-4), while the assertions are grounded in published bug taxonomies (Islam et al. [6], Humbatova et al. [7]) and Keras documentation, and the evaluation uses 50 external Stack Overflow/GitHub programs plus a user study. The self-citations (e.g., Islam et al. [6] and DeepDiagnosis [21], which share authors with this paper) serve as taxonomy sources and a comparison baseline, but the central claim that mocks enable independent unit testing does not reduce to those citations; the framework is implemented and evaluated on programs the authors say they had not seen. A validity caveat, not a circularity, is that Table IV shows most of the 53 model-design detections (e.g., 27 of 53 for Wrong Output Layer Activation) come from static property assertions over interface metadata rather than from mock-data behavior; only Oscillating Loss/Slow Convergence (3 detections) and possibly Learning Rate out of Common Range (3) depend on mock behavior, and the 10 data-preparation detections are property checks such as Missing Scaling/Normalization rather than mock-model behavior checks. This undercuts the attribution wording 'mocks effectively identified 53 issues,' but it is an over-claim about which component does the detection, not a case where the prediction is equivalent to the input by construction. The paper explicitly distinguishes structure assertions A-E from mock-behavior checks F-H in Section III-B, so no equation, fitted parameter, or uniqueness theorem is reused as its own conclusion.

Assumptions & free parameters 3 free parameters · 3 assumptions · 2 invented entities

The central claim depends on the assumption that mocks preserve behavior, and on several tuned parameters (sample sizes, monitoring frequency, model complexity). The mock data and mock model are invented abstractions, but they are software constructs rather than physical entities, and their utility is supported by the internal evaluation.

free parameters (3)
  • Mock data sample size = 10 times number of features for regression; 100 samples per class for classification
    Chosen based on sensitivity analysis varying dataset size by +/-5%, 10%, 20%, not derived from a theoretical principle. This directly affects which model issues can be detected.
  • Loss monitoring frequency = every 5 epochs
    Used to detect oscillating loss or slow convergence. The interval is a tunable threshold that affects false positive and false negative rates.
  • Hidden layer neurons in mock model = equal to number of features
    Part of the mock model generation rules in Decision Table I, based on a simplicity heuristic. This choice influences whether data issues like mislabeled data or outliers are exposed.
assumptions (3)
  • domain assumption The behavior of original data on a mock model and original model on mock data is consistent enough to expose real bugs.
    This is stated in Section II (Motivation) and underlies the entire approach. If mocks do not preserve failure modes, the unit tests will miss genuine issues.
  • domain assumption Synthetic mock data generated by scikit-learn make_regression and make_classification, after scaling, adequately represents real data for model design testing.
    Invoked in Section III-B2. The mock data is normally distributed, balanced, and free of missing values and outliers, which may not capture data-specific bugs.
  • domain assumption A three-layer mock model is sufficient to detect data quality issues, following the 'start simple' principle.
    Cited from ML literature in Section III-B1. If a simple network fails to learn, the data is deemed problematic; this may not hold for all data distributions.
invented entities (2)
  • Mock Data
    purpose: Synthetic data used to test the model design stage in isolation, generated via scikit-learn functions and scaled.
    The paper provides empirical evidence of its usefulness in the evaluation, but this is internal to the study and not a falsifiable prediction outside the paper.
  • Mock Model
    purpose: A simplified three-layer neural network used to test the data preparation stage in isolation, generated from interface information.
    The mock model is a software construct with behavior validated only within the paper's experiments; it is not a standalone prediction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mock Deep Testing: Toward Separate Development of Data and Models for Deep Learning." pith.science (2026). https://pith.science/paper/QMKP7HCZ

@misc{pith2026250207712,
  author       = {Pith},
  title        = {Pith review of: Mock Deep Testing: Toward Separate Development of Data and Models for Deep Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QMKP7HCZ}},
  note         = {Machine review of arXiv:2502.07712}
}
read the original abstract

While deep learning (DL) has permeated, and become an integral component of many critical software systems, today software engineering research hasn't explored how to separately test data and models that are integral for DL approaches to work effectively. The main challenge in independently testing these components arises from the tight dependency between data and models. This research explores this gap, introducing our methodology of mock deep testing for unit testing of DL applications. To enable unit testing, we introduce a design paradigm that decomposes the workflow into distinct, manageable components, minimizes sequential dependencies, and modularizes key stages of the DL. For unit testing these components, we propose modeling their dependencies using mocks. This modular approach facilitates independent development and testing of the components, ensuring comprehensive quality assurance throughout the development process. We have developed KUnit, a framework for enabling mock deep testing for the Keras library. We empirically evaluated KUnit to determine the effectiveness of mocks. Our assessment of 50 DL programs obtained from Stack Overflow and GitHub shows that mocks effectively identified 10 issues in the data preparation stage and 53 issues in the model design stage. We also conducted a user study with 36 participants using KUnit to perceive the effectiveness of our approach. Participants using KUnit successfully resolved 25 issues in the data preparation stage and 38 issues in the model design stage. Our findings highlight that mock objects provide a lightweight emulation of the dependencies for unit testing, facilitating early bug detection. Lastly, to evaluate the usability of KUnit, we conducted a post-study survey. The results reveal that KUnit is helpful to DL application developers, enabling them to independently test each component effectively in different stages.

Figures

Figures reproduced from arXiv: 2502.07712 by the authors.

Figure 1
Figure 1. A buggy DL program and mocks in action. paradigm that considers each stage of the DL program, i.e., data preparation and model design, as separate modules. The unique challenge in independently testing these com￾ponents arises from the tight dependency between data and models. To handle the inherent dependencies among these modules, we propose defining clear interfaces to decouple them. These interfaces specify key … view at source ↗
Figure 2
Figure 2. Workflow of KUnit. II. MOTIVATION The current practice in DL application development involves sequentiality, where the data is prepared first followed by model design and training. The designed model is tested for crash and silent bugs using the data by monitoring and identifying abnormal behavior during training [19], [20], [21], [22], [23], [24], [25]. Bugs can originate from any stage of the DL pipeline, such as … view at source ↗
Figure 3
Figure 3. Interface definition and class description. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Mock object creation for different stages. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Survey results with participants ratings. [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 59 canonical work pages

  1. [1]

    Deep Learning Based Chatbot Models

    C. Richard, “Deep learning based chatbot models,” in arXiv preprint arXiv:1908.08835, 2019

    work page Pith review arXiv 1908

  2. [2]

    A vision and speech enabled, customizable, virtual assistant for smart environments,

    I. Giancarlo, L. L. Bello, A. Nucita, and G. M. Grasso, “A vision and speech enabled, customizable, virtual assistant for smart environments,” in In 2018 11th International Conference on Human System Interaction (HSI), 2018, pp. 50–56

    work page 2018

  3. [3]

    Deep learning detecting fraud in credit card transactions,

    R. Abhimanyu, J. Sun, R. Mahoney, L. Alonzi, S. Adams, and P. Beling, “Deep learning detecting fraud in credit card transactions,” in In 2018 systems and information engineering design symposium , 2018, pp. 129– 134

    work page 2018

  4. [4]

    An empirical study of common challenges in developing deep learning applications,

    T. Zhang, C. Gao, L. Ma, M. Lyu, and M. Kim, “An empirical study of common challenges in developing deep learning applications,” in 2019 IEEE 30th International Symposium on Software Reliability Engineering (ISSRE), 2019, pp. 104–115

    work page 2019

  5. [5]

    Software engineering practice in the development of deep learning applications,

    X. Zhang, Y . Yang, Y . Feng, and Z. Chen, “Software engineering practice in the development of deep learning applications,” in ICSE’20: The 42nd International Conference on Software Engineering , 2020

    work page 2020

  6. [6]

    A comprehensive study on deep learning bug characteristics,

    M. J. Islam, G. Nguyen, R. Pan, and H. Rajan, “A comprehensive study on deep learning bug characteristics,” in ESEC/FSE’19: The ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) , ser. ESEC/FSE 2019, August 2019

    work page 2019

  7. [7]

    Taxonomy of real faults in deep learning systems,

    N. Humbatova, G. Jahangirova, G. Bavota, V . Riccio, A. Stocco, and P. Tonella, “Taxonomy of real faults in deep learning systems,” in ICSE’20: The ACM/IEEE 42nd International Conference on Software Engineering, 2020, pp. 1110–1121

    work page 2020

  8. [8]

    An empirical study on tensorflow program bugs,

    Y . Zhang, C. Yifan, C. Shing-Chi, X. Yingfei, and Z. Lu, “An empirical study on tensorflow program bugs,” in27th ACM SIGSOFT International Symposium on Software Testing and Analysis , 2018, pp. 129–140

    work page 2018

Show all 62 references
  1. [9]

    Understanding performance problems in deep learning systems,

    J. Cao, B. Chen, C. Sun, L. Hu, S. Wu, and X. Peng, “Understanding performance problems in deep learning systems,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2022, pp. 357– 369

    2022

  2. [10]

    Machine learning application development: practitioners’ insights,

    M. S. Rahman, F. Khomh, A. Hamidi, J. Cheng, G. Antoniol, and H. Washizaki, “Machine learning application development: practitioners’ insights,” Software Quality Journal, vol. 31, no. 4, pp. 1065–1119, 2023

    2023

  3. [11]

    On decomposing a deep neural network into modules,

    R. Pan and H. Rajan, “On decomposing a deep neural network into modules,” in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2020, pp. 889–900

    2020

  4. [12]

    Decomposing convolutional neural networks into reusable and replaceable modules,

    ——, “Decomposing convolutional neural networks into reusable and replaceable modules,” in Proceedings of the 44th International Confer- ence on Software Engineering , 2022, pp. 524–535

    2022

  5. [13]

    The art and practice of data science pipelines: A comprehensive study of data science pipelines in theory, in-the-small, and in-the-large,

    S. Biswas, M. Wardat, and H. Rajan, “The art and practice of data science pipelines: A comprehensive study of data science pipelines in theory, in-the-small, and in-the-large,” in ICSE’22: The 44th Interna- tional Conference on Software Engineering , May 21-May 29 2022

    2022

  6. [14]

    Software engineering for machine learning: A case study,

    S. Amershi, A. Begel, C. Bird, R. DeLine, H. Gall, E. Kamar, N. Na- gappan, B. Nushi, and T. Zimmermann, “Software engineering for machine learning: A case study,” in 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SE...

    2019

  7. [15]

    Binder, Testing object-oriented systems: models, patterns, and tools

    R. Binder, Testing object-oriented systems: models, patterns, and tools . Addison-Wesley Professional, 2000

    2000

  8. [16]

    Endo-testing: unit testing with mock objects,

    T. Mackinnon, S. Freeman, and P. Craig, “Endo-testing: unit testing with mock objects,” in Extreme programming examined, 2000, pp. 287–301

    2000

  9. [17]

    Goodfellow, Y

    I. Goodfellow, Y . Bengio, and A. Courville, Deep learning. MIT press, 2016

    2016

  10. [18]

    https://github.com/Ruchira-1/KUnit, 2024

    2024

  11. [19]

    Umlaut: Debugging deep learning programs using program structure and model behavior,

    E. Schoop, F. Huang, and B. Hartmann, “Umlaut: Debugging deep learning programs using program structure and model behavior,” in Proceedings of the 2021 CHI Conference Extended Abstracts on Human Factors in Computing Systems , 2021

    2021

  12. [20]

    Deeplocalize: fault localization for deep neural networks,

    M. Wardat, W. Le, and H. Rajan, “Deeplocalize: fault localization for deep neural networks,” in ICSE’21: The 43nd International Conference on Software Engineering , 2021

    2021

  13. [21]

    Deepdiagnosis: Auto- matically diagnosing faults and recommending actionable fixes in deep learning programs,

    M. Wardat, B. D. Cruz, W. Le, and H. Rajan, “Deepdiagnosis: Auto- matically diagnosing faults and recommending actionable fixes in deep learning programs,” in ICSE’22: The 44th International Conference on Software Engineering, 2022

    2022

  14. [22]

    Testing feedforward neural networks training programs,

    H. Ben Braiek and F. Khomh, “Testing feedforward neural networks training programs,” ACM Trans. Softw. Eng. Methodol. , vol. 32, no. 4, may 2023. [Online]. Available: https://doi.org/10.1145/3529318

    2023 doi

  15. [23]

    Deepfd: Automated fault diagnosis and localization for deep learning programs,

    J. Cao, M. Li, X. Chen, M. Wen, Y . Tian, B. Wu, and S.-C. Cheung, “Deepfd: Automated fault diagnosis and localization for deep learning programs,” in Proceedings of the 44th International Conference on Software Engineering, 2022, pp. 573–585

    2022

  16. [24]

    Autotrainer: An automatic dnn training problem detection and repair system,

    X. Zhang, J. Zhai, S. Ma, and C. Shen, “Autotrainer: An automatic dnn training problem detection and repair system,” in ICSE’21: The 43rd International Conference on Software Engineering , 2021, pp. 359–371

    2021

  17. [25]

    Mutation- based fault localization of deep neural networks,

    A. Ghanbari, D.-G. Thomas, M. A. Arshad, and H. Rajan, “Mutation- based fault localization of deep neural networks,” in ASE’2023: 38th IEEE/ACM International Conference on Automated Software Engineer- ing, September 11–15 2023

    2023

  18. [26]

    Repairing deep neural networks: Fix patterns and challenges,

    M. J. Islam, R. Pan, G. Nguyen, and H. Rajan, “Repairing deep neural networks: Fix patterns and challenges,” in ICSE’20: The 42nd International Conference on Software Engineering , May 23-May 29, 2020 2020

    2020

  19. [27]

    A survey of unit testing practices,

    P. Runeson, “A survey of unit testing practices,” IEEE software, vol. 23, no. 4, pp. 22–29, 2006

    2006

  20. [28]

    On the criteria to be used in decomposing systems into modules,

    D. L. Parnas, “On the criteria to be used in decomposing systems into modules,” Communications of the ACM, vol. 15, no. 12, pp. 1053–1058, 1972

    1972

  21. [29]

    Deep neural networks and tabular data: A survey,

    V . Borisov, T. Leemann, K. Seßler, J. Haug, M. Pawelczyk, and G. Kasneci, “Deep neural networks and tabular data: A survey,” IEEE Transactions on Neural Networks and Learning Systems , 2022

    2022

  22. [30]

    Deep learning,

    Y . LeCun, Y . Bengio, and G. Hinton, “Deep learning,” nature, vol. 521, no. 7553, pp. 436–444, 2015

    2015

  23. [31]

    Hyper-parameter optimization: A review of algo- rithms and applications,

    T. Yu and H. Zhu, “Hyper-parameter optimization: A review of algo- rithms and applications,” arXiv preprint arXiv:2003.05689 , 2020

    2003 arXiv

  24. [32]

    Gradient-based learning applied to document recognition,

    Y . LeCun, L. Bottou, Y . Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” in Proceedings of IEEE, vol. 11, 1998, pp. 2278–2324

    1998

  25. [33]

    Imagenet classification with deep convolutional networks,

    A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional networks,” in NIPS’12: The 25th International Conference on Neural Information Processing Systems , vol. 1, 2012, p. 1097–1105

    2012

  26. [34]

    Efficient backprop,

    Y . LeCun, L. Bottou, G. B. Orr, and K.-R. M ¨uller, “Efficient backprop,” in Neural networks: Tricks of the trade . Springer, 2002, pp. 9–50

    2002

  27. [35]

    Practical recommendations for gradient-based training of deep architectures,

    Y . Bengio, “Practical recommendations for gradient-based training of deep architectures,” in Neural Networks: Tricks of the Trade: Second Edition. Springer, 2012, pp. 437–478

    2012

  28. [36]

    Keras: the Python deep learning library,

    Francois Chollet, “Keras: the Python deep learning library,” 2015, https: //keras.io/api/losses/

    2015

  29. [37]

    Machine learning course,

    A. Ng, “Machine learning course,” https://www.coursera.org/learn/ neural-networks-deep-learning

  30. [38]

    Occam’s razor,

    “Occam’s razor,” https://en.wikipedia.org/wiki/Occam%27s razor

  31. [39]

    sklearn.datasets: Samples generator,

    scikit-learn, “sklearn.datasets: Samples generator,” 2007, https:// scikit-learn.org/stable/modules/classes.html#module-sklearn.datasets

    2007

  32. [40]

    Efficient backprop

    Y . A. L. L´eon, B. B.Orr, and K.-R. M¨uller, “Efficient backprop.” Berlin, Heidelberg: Springer, 2012

    2012

  33. [41]

    Rule of 10,

    “Rule of 10,” https://machinelearningmastery.com/ much-training-data-required-machine-learning/, 2023

    2023

  34. [42]

    Lakshmanan, S

    V . Lakshmanan, S. Robinson, and M. Munn, Machine learning design patterns. O’Reilly Media, 2020

    2020

  35. [43]

    Automatic fault detection for deep learning programs using graph transformations,

    A. Nikanjam, B. B. Houssem, M. M. Mohammad, and K. Foutse, “Automatic fault detection for deep learning programs using graph transformations,” in ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 31, no. 1, 2021, pp. 1–27

    2021

  36. [44]

    Leveraging data characteristics for bug localization in deep learning programs,

    R. Manke, M. Wardat, F. Khomh, and H. Rajan, “Leveraging data characteristics for bug localization in deep learning programs,” ACM Transactions on Software Engineering and Methodology , 2024

    2024

  37. [45]

    Localizing and repairing faults in deep learning programs,

    M. A. S. Wardat, “Localizing and repairing faults in deep learning programs,” Ph.D. dissertation, Iowa State University, 2023

    2023

  38. [46]

    Design by contract for deep learning apis,

    S. Ahmed, S. M. Imtiaz, S. S. Khairunnesa, B. D. Cruz, and H. Ra- jan, “Design by contract for deep learning apis,” in ESEC/FSE’2023: The 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering , December 03- December 09 2023

    2023

  39. [47]

    What kinds of contracts do ml apis need?

    S. S. Khairunnesa, S. Ahmed, S. M. Imtiaz, H. Rajan, and G. T. Leavens, “What kinds of contracts do ml apis need?” Empirical Software Engineering, vol. 1, no. 1, March 2023

    2023

  40. [48]

    Keras: the Python deep learning library,

    Francois Chollet, “Keras: the Python deep learning library,” 2015, https: //keras.io/

    2015

  41. [49]

    Keras: the Python deep learning library,

    ——, “Keras: the Python deep learning library,” 2015, https://keras.io/ examples/. 12

    2015

  42. [50]

    Kaggle ,

    “Kaggle ,” https://www.kaggle.com/competitions, 2024

    2024

  43. [51]

    Codespaces, “Github,” https://github.com/features/codespaces

    G. Codespaces, “Github,” https://github.com/features/codespaces

  44. [52]

    Nanofuzz: A usable tool for automatic test generation,

    M. C. Davis, S. Choi, S. Estep, B. A. Myers, and J. Sunshine, “Nanofuzz: A usable tool for automatic test generation,” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2023, pp. 1114–1126

    2023

  45. [53]

    https://github.com/anon3173/KUnit/blob/main/RQ2 Results.xlsx, 2024

    2024

  46. [54]

    https://github.com/anon3173/KUnit/blob/main/RQ2 UserStudy Results.xlsx, 2024

    2024

  47. [55]

    https://github.com/anon3173/KUnit/blob/main/Analysis of Debugging Time.pdf, 2024

    2024

  48. [56]

    R. S. Weiss, Learning from strangers: The art and method of qualitative interview studies. Simon and Schuster, 1995

    1995

  49. [57]

    pdf, 2024

    https://github.com/anon3173/KUnit/blob/main/Participants Response. pdf, 2024

    2024

  50. [58]

    Personal Communication with Authors regarding DeepDiag- nosis: Automatically Diagnosing Faults and Recommending Actionable Fixes in Deep Learning Programs ,

    Authors, “Personal Communication with Authors regarding DeepDiag- nosis: Automatically Diagnosing Faults and Recommending Actionable Fixes in Deep Learning Programs ,” Email, March, 2024

    2024

  51. [59]

    Mock objects,

    D. Thomas and A. Hunt, “Mock objects,” IEEE Software, vol. 19, no. 3, pp. 22–24, 2002

    2002

  52. [60]

    Unit testing in multi-agent systems using mock agents and aspects,

    R. Coelho, U. Kulesza, A. von Staa, and C. Lucena, “Unit testing in multi-agent systems using mock agents and aspects,” in Proceedings of the 2006 international workshop on Software engineering for large-scale multi-agent systems, 2006, pp. 83–90

    2006

  53. [61]

    A framework for automated test mocking of mobile apps,

    M. Fazzini, A. Gorla, and A. Orso, “A framework for automated test mocking of mobile apps,” in Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, 2020, pp. 1204–1208

    2020

  54. [62]

    Moda: Automated test generation for database applications via mock objects,

    K. Taneja, Y . Zhang, and T. Xie, “Moda: Automated test generation for database applications via mock objects,” in Proceedings of the 25th IEEE/ACM International Conference on Automated Software Engineer- ing, 2010, pp. 289–292. 13

    2010

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.