Pith. sign in

REVIEW 5 major objections 6 minor 66 references

Deep Learning Framework Testing via Model Mutation: How Far Are We?

T0 review · 5 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Mutation-based DL framework testing is misaligned with developer priorities; it mostly reports duplicate crash/NaN defects and misses performance and resource-scheduling bugs.

desk verdict A genuinely useful developer-priority taxonomy and a plausible high-level story, but the empirical headline counts need an audit trail before I'd trust the numbers. read the letter →

arxiv 2506.17638 v2 pith:FY4RTQIH submitted 2025-06-21 cs.SE

Yanzhou Mu , Rong Wang , Juan Zhai , Chunrong Fang , Xiang Chen , Zhiyuan Peng , Peiran Yang , Ruixiang Qian
show 2 more authors
Shaoyu Yang Zhenyu Chen
This is my paper · ORCID
classification cs.SE
keywords deeplearningframeworktestingmodelmutationdifferentialdefecttaxonomydeveloperpriorityoperatorsempiricalstudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the current mutation-based methods for testing deep learning (DL) frameworks, which mutate models and then look for cross-framework inconsistencies, crashes, and NaN outputs, are largely ineffective at finding the defects that developers actually consider important. To show this, the authors built a developer-priority taxonomy from thousands of issue reports labeled by developers themselves and used it as an evaluation yardstick. They found that the existing methods rarely report inconsistency defects, flood with duplicate crash and NaN reports, and completely miss performance and resource-scheduling defects that developers rate as high priority. They then analyzed how mutation type, mutation order, and mutation position affect the usefulness of mutants, used those insights to re-target the latest coverage-guided method, and detected seven new defects, four confirmed as high-priority and three fixed. If the paper is right, the value of a mutation-based testing method should be judged by how well its mutants expose the execution and resource behaviors developers care about, not by how many crash or NaN signals it produces.

What carries the argument

The load-bearing machinery is the developer-priority defect taxonomy: 3,000 issue reports tagged as high priority by developers, independently labeled by six volunteers with an inter-rater agreement of 0.804 into seven major and 21 sub-types, which serves as the evaluation yardstick for judging whether a testing method finds defects developers would fix. The second mechanism is the mutation-factor analysis built on the layer-wise inconsistency change rate $R^{M,N}_{f_{L_i}}(x)$, the relative jump in mean absolute output difference between consecutive layers across two frameworks; experiments varying mutation type, mutation order, and mutation position show which mutant-generating choices expose stronger inconsistencies and which mostly produce illegal models. Together these mechanisms convert the question 'can this method find any defect?' into 'can this method find the defects developers would actually fix?'

What would settle it

Independently rank a random sample of the labeled high-priority reports by interviewing framework developers who did not see the original tags; if their priority rankings match the taxonomy no better than chance, the premise that tags reflect developer priorities fails. A cheaper check is to re-label the 1,500 TensorFlow reports the paper manually reviewed with a fresh set of developers and compare their severity assessments with the tag-based selection.

Watch

Extended reading notes

Core claim

The paper establishes that existing mutation-based DL framework testing methods do not align with what framework developers consider important defects. By classifying 3,000 high-priority defect reports from three popular frameworks into seven major types and 21 sub-types, the authors show that the most valued defect categories are resource scheduling (GPU and memory), performance (loss and evaluation metrics), and crashes caused by complex interface invocations. Running four existing mutation-based methods on a 23-model benchmark spanning eight industrial tasks, the paper finds that these methods rarely report inconsistency defects, produce many duplicate crash and NaN defects, and miss performance and resource-scheduling defects entirely. The paper then shows that mutation type, mutation order, and mutation position control whether mutants are useful, and uses that analysis to modify the latest coverage-guided mutation method; the modified method detects seven new defects, four confirmed as high-priority and three fixed, and the study as a whole reports 39 unique defects across 23 models, 31 confirmed and eight fixed.

Load-bearing premise

The whole analysis rests on the assumption that issue-tracker tags and volunteer labels truly capture which defects developers consider important; if those labels are biased, the conclusion that existing methods miss important defects is unsupported.

Editorial extensions

If this is right

  • A mutation-based testing method's yield should be measured in confirmed developer-priority defects, not raw counts of crashes, NaN values, or inconsistencies, because raw report counts inflate apparent value.
  • Prioritizing structure mutations, low mutation order, and backbone-layer mutation positions should let future methods expose cross-framework inconsistencies with fewer false positives.
  • Input-mutation operators, which enlarge tensors and slow execution, can be redirected to detect efficiency and resource-allocation defects such as slow operators and memory allocation failures.
  • Performance defects arising during model training, such as abnormal loss and metric deterioration, are the weakest area for existing methods, so oracles that monitor training-time behavior would fill a genuine gap.
  • Finding 39 unique defects (31 confirmed, eight fixed) using only 23 models indicates that a benchmark built from industrial tasks can be more productive than broad random model generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not say this, but if the developer-priority taxonomy is representative, other testing paradigms that generate random or template-based models likely share the same blind spot, because they also concentrate on static model execution rather than resource scheduling and training-time performance.
  • The backbone-versus-task-head result suggests a testable extension: mutation position could be guided by a learned model-structure prior, concentrating perturbations on deep feature-extraction layers to raise defect yield per mutant.
  • The high-order-mutation false-positive result implies a cost trade-off the paper does not compute directly: under a fixed time budget, many low-order mutants may outperform fewer high-order mutants, since high-order mutations mostly generate illegal models.
  • If developers continue to confirm resource and efficiency defects, differential testing of DL frameworks could add execution-time oracles, such as GPU memory, latency, and loss curves, alongside output-correctness oracles; the paper identifies this gap but does not build such oracles itself.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. This paper revisits mutation-based testing methods for deep learning frameworks. It constructs a developer-priority defect taxonomy from issue reports of PyTorch, TensorFlow, and MindSpore (seven major types, 21 sub-types), evaluates CRADLE, LEMON, AUDEE, and COMET against this taxonomy using 23 models, and studies how mutation type, mutation order, and mutation position affect mutant output inconsistency. Based on the findings, the authors propose optimization strategies and report that these strategies helped detect seven new defects, four confirmed as high-priority and three fixed, while the paper overall reports 39 unique defects, 31 confirmed and eight fixed. The central claim is that existing mutation-based methods detect few developer-important defects, that they rarely report inconsistency defects while producing many duplicate crash and NAN defects, and that they miss the performance and resource-scheduling defects developers rate as important.

Significance. If the claims hold, this is a valuable empirical contribution: it provides a novel developer-priority defect taxonomy, a controlled replication of four mutation-based methods, and concrete optimization directions. Strengths include the breadth of issue-report collection, a Fleiss kappa of 0.804 for volunteer labeling, a benchmark of 23 models covering several industrial task types, and a public replication website. The seven post-optimization defects, with developer confirmation and three fixes, are concrete falsifiable artifacts. However, the headline defect counts and the central effectiveness comparison currently rest on provenance and parameter-selection choices that are not transparent enough for the conclusions to be accepted as stated.

major comments (5)
  1. [Section 5.2, Abstract, Contribution bullets] The count of 39 'new framework defects' is not supported by the paper's own methodology. Section 5.2 states that the collected defects consist of '(1) defects detected in previous studies and (2) defects detected during their execution of our new benchmark,' and Fig. 6 explicitly counts both kinds, yet the Abstract and Introduction say 'we identified 39 unique defects' and the contribution list says '39 new framework defects' and '32 defects were detected with our newly constructed benchmark.' These are different populations. Please provide a per-defect audit table linking each of the 39 defects to its source (prior paper vs. new run), the method and model that found it, its HP sub-type, and its confirmed/fixed status, and revise the summary claims to distinguish newly detected defects from collected and re-checked defects. Without this provenance, the headline number and the HP-detection distribution in Fig. 7 cannot be verified.
  2. [Section 4.3] Test Oracle 3 sets the inconsistency threshold t to 1e3 'based on our experimental results.' This is a free parameter fitted on the same data that is later used to compute the RQ2 inconsistency-defect counts, making the finding that existing methods 'rarely detect inconsistency defects' sensitive to an ad hoc choice. Please state exactly how t was selected, which experimental results were used, and whether any held-out data informed the choice. A sensitivity analysis over reasonable threshold values (e.g., 1e2 and 1e4) showing how the RQ2 counts and Fig. 7 change would make the conclusion robust.
  3. [Section 5.3] The mutation-round settings for RQ3 are chosen 'based on the large-scale experiment results' to reduce invalid mutants, but the selection protocol is not reported. Since Findings 7-10 compare output inconsistency and illegal-mutant rates across mutation types, orders, and positions, the conclusions may be artifacts of the chosen rounds for each operator and model. Please report the per-operator and per-model rules used to select rounds (or provide the underlying data on the website) and include a sensitivity check for at least one representative model to show that the rankings among operators and positions are stable.
  4. [Section 3.2] The entire high-priority taxonomy rests on issue tags plus volunteer labels. The paper validates TensorFlow's 'type bug' tag by manually checking 1,500 reports and finding over 90% considered high priority, but it does not report equivalent validation for PyTorch's 'high priority'/'bug' tags or MindSpore's 'Main'/'Serious'/'kind/bug' tags, nor how many of the selected tags were explicitly confirmed by developers. Because Findings 5 and 6—and the paper's central claim that existing methods miss important defects—depend on this taxonomy, please add per-tag validation results and a fuller account of developer confirmation, or soften the 'High Priority' wording accordingly.
  5. [Section 5.2, Fig. 6] The arithmetic of the reported defect counts is not internally consistent. Section 5.2 reports that COMET produced 320 crash defects reduced to six unique after manual inspection, and that COMET's total includes 7 NAN defects and 28 previous defects; the Abstract claims 39 unique defects overall; and Section 5.2 later refers to 'the 34 detected defects.' Please reconcile these numbers, define the deduplication rule across methods and across previous/new detections, and state the unit of counting (unique defect IDs, reports, or confirmed defects) for each figure and table.
minor comments (6)
  1. [Section 2.2, References] The statement that mutation testing was 'initially proposed by Lipton et al. [3]' cites reference [3], which is a 1971 fault-diagnosis review by Bennetts and Lewin; the attribution and the reference list need to be reconciled.
  2. [Table 3, Section 5.3] The activation-layer operator is written as 'ARFm/ARFp' in Table 3 but as 'AFRm' and 'AFRp' in the text and in Fig. 9; please unify the notation.
  3. [Fig. 8 caption] The caption refers to the 'RA mutation operator,' but the operator discussed in the text is AFRp; please align the caption with the text.
  4. [Section 5.2] The sentence 'Among the 34 detected defects' appears after totals of 39 in the Abstract and 38 in Fig. 6; please use one consistent universe of defects throughout the paper.
  5. [Fig. 5] The y-axis of Fig. 5 is truncated and the bar labels show counts up to 7,740 while the axis does not; please use a labeled axis or a log scale so the distribution is legible.
  6. [Acknowledgments] The heading 'ACKNOWLEGEMENT' should be spelled 'ACKNOWLEDGMENTS.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the developer-priority taxonomy, mutation-method evaluation, and developer-confirmed defects are independent of the paper's conclusions; threshold calibration is a limitation, not a circular step.

full rationale

The paper's derivation chain is empirical rather than definitional. RQ1 constructs a developer-priority taxonomy from 3,000 tagged issue reports; RQ2 runs CRADLE, LEMON, COMET (and collects AUDEE's prior defects) and classifies the detected defects against that taxonomy; RQ3 analyzes mutation factors and uses the findings to propose optimizations; the optimizations are then instantiated, and the resulting seven defects were confirmed by framework developers. Each step uses an input that is external to the claimed conclusion: the taxonomy comes from issue reports, not from the mutation methods, so using it as an evaluation rubric does not force the observed distribution of detected defect types. The factor analysis and optimization suggestions are derived from the same benchmark, which limits generalizability, but the 'seven new defects' claim is supported by independent developer confirmation, so it is not a renamed prediction. The Test Oracle 3 threshold t=1e3 is calibrated on experimental results, but final defect counts are based on manual false-positive removal and developer confirmation; the threshold only generates candidates and does not by itself determine the headline counts. The mixing of previously reported defects with newly detected ones in the 39-defect total and the lack of per-defect provenance is a reporting and validity concern, not circularity. Self-citations in the related work are not load-bearing for any central claim. Therefore, no circular step is established.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claims rest on developer-tag filtering, volunteer labeling, a threshold calibrated on experimental results, and mutation rounds also selected from the same experiments. No new physical or computational entities are introduced.

free parameters (2)
  • Test Oracle 3 threshold t = 1000
    In Section 4.3, t is set to 1e3 'based on our experimental results' to achieve low false positives; the oracle is calibrated on the evaluation data.
  • Mutation rounds per operator/model = 5-100 depending on operator and model
    In Section 5.3, mutation rounds are chosen from large-scale experiment results to avoid empty or invalid models; these settings affect the reported inconsistency and false-positive rates.
assumptions (4)
  • domain assumption Output inconsistency, crash, and NAN across frameworks indicate framework defects.
    Used throughout; the differential testing oracle in Section 4.3 assumes these symptoms are caused by framework defects rather than by mutation or floating-point noise.
  • domain assumption GitHub/Gitee tags such as 'type bug', 'high priority', and 'Serious' reliably identify high-priority defects.
    Section 3.2 filters HP defect reports using these tags and manually checks 1,500 TensorFlow reports, finding over 90% are high priority; this underpins RQ1.
  • domain assumption Volunteer labels with Fleiss kappa 0.804 are an accurate proxy for developer priorities.
    Section 3.2 uses six volunteers' labels to build the taxonomy; the paper consults developers only on typical cases, not all labels.
  • domain assumption Default parameter settings of existing methods fairly represent their capability.
    Sections 4.2 and 5.2 run CRADLE, LEMON, and COMET under default or recommended settings; if the defaults are poorly tuned, the methods' effectiveness would be underestimated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Deep Learning Framework Testing via Model Mutation: How Far Are We?." pith.science (2026). https://pith.science/paper/FY4RTQIH

@misc{pith2026250617638,
  author       = {Pith},
  title        = {Pith review of: Deep Learning Framework Testing via Model Mutation: How Far Are We?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FY4RTQIH}},
  note         = {Machine review of arXiv:2506.17638}
}
read the original abstract

Deep Learning (DL) frameworks are a fundamental component of DL development. Therefore, the detection of DL framework defects is important and challenging. As one of the most widely adopted DL testing techniques, model mutation has recently gained significant attention. In this study, we revisit the defect detection ability of existing mutation-based testing methods and investigate the factors that influence their effectiveness. To begin with, we reviewed existing methods and observed that many of them mutate DL models (e.g., changing their parameters) without any customization, ignoring the unique challenges in framework testing. Another issue with these methods is their limited effectiveness, characterized by a high rate of false positives caused by illegal mutations arising from the use of generic, non-customized mutation operators. Moreover, we tracked the defects identified by these methods and discovered that most of them were ignored by developers. Motivated by these observations, we investigate the effectiveness of existing mutation-based testing methods in detecting important defects that have been authenticated by framework developers. We begin by collecting defect reports from three popular frameworks and classifying them based on framework developers' ratings to build a comprehensive dataset. We then perform an in-depth analysis to uncover valuable insights. Based on our findings, we propose optimization strategies to address the shortcomings of existing approaches. Following these optimizations, we identified seven new defects, four of which were confirmed by developers as high-priority issues, with three resolved. In summary, we identified 39 unique defects across just 23 models, of which 31 were confirmed by developers, and eight have been fixed.

Figures

Figures reproduced from arXiv: 2506.17638 by the authors.

Figure 1
Figure 1. An example of developer response on defect report [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Workflow of mutation-based DL framework testing methods [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Overview of our methodology collected defect reports. The total counts of HP defects, all the defects, and all the issue reports are shown in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: The statistics of collected issue reports [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: The distribution of labelled HP defect type [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: The number of defects detected by existing methods [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: The number of HP defects detected by existing methods [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Output inconsistency of ResNet50-1 mutants generated by different types of mutation operators [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Comparison results about the output inconsistency of SSIM-AE mutants generated in different [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

66 extracted references · 31 canonical work pages

  1. [1]

    Christophe Andrieu, Nando De Freitas, Arnaud Doucet, and Michael I Jordan. 2003. An introduction to MCMC for machine learning. Machine learning 50 (2003), 5–43

    2003

  2. [2]

    Vinay Arora, Rajesh Kumar Bhatia, and Maninder Pal Singh. 2016. A systematic review of approaches for testing concurrent programs. Concurrency and Computation: Practice and Experience 28 (2016), 1572 – 1611. https://api. semanticscholar.org/CorpusID:45879522

    work page 2016

  3. [3]

    RG Bennetts and DW Lewin. 1971. Fault diagnosis of digital systems-a review. Computer 4, 4 (1971), 12–20

    work page 1971

  4. [4]

    Chenyi Chen, Ari Seff, Alain Kornhauser, and Jianxiong Xiao. 2015. Deepdriving: Learning affordance for direct perception in autonomous driving. In Proceedings of the IEEE international conference on computer vision . 2722–2730

    2015

  5. [5]

    Junjie Chen, Xiaoting He, Qingwei Lin, Yong Xu, Hongyu Zhang, Dan Hao, Feng Gao, Zhangwei Xu, Yingnong Dang, and Dongmei Zhang. 2019. An empirical investigation of incident triage for online service systems. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) . IEEE, 111–120

    work page 2019

  6. [6]

    Junjie Chen, Xiaoting He, Qingwei Lin, Hongyu Zhang, Dan Hao, Feng Gao, Zhangwei Xu, Yingnong Dang, and Dongmei Zhang. 2019. Continuous incident triage for large-scale online service systems. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 364–375

    2019

  7. [7]

    Jinyin Chen, Chengyu Jia, Yunjie Yan, Jie Ge, Haibin Zheng, and Yao Cheng. 2024. A Miss Is as Good as A Mile: Metamorphic Testing for Deep Learning Operators. Proceedings of the ACM on Software Engineering 1, FSE (2024), 2005–2027

    2024

  8. [8]

    Junjie Chen, Yihua Liang, Qingchao Shen, and Jiajun Jiang. 2022. Toward Understanding Deep Learning Framework Bugs. ACM Transactions on Software Engineering and Methodology (2022). https://api.semanticscholar.org/CorpusID: 247315490

    work page 2022

Show all 66 references
  1. [9]

    Yuting Chen, Ting Su, Chengnian Sun, Zhendong Su, and Jianjun Zhao. 2016. Coverage-directed differential testing of JVM implementations. Proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and Implementation (2016). https://api.semanticscholar.org/Cor...

    2016

  2. [10]

    Rémi Coulom. 2006. Efficient Selectivity and Backup Operators in Monte-Carlo Tree Search. In Computers and Games. https://api.semanticscholar.org/CorpusID:16724115

    2006

  3. [11]

    DeMillo, Richard J

    Richard A. DeMillo, Richard J. Lipton, and Frederick G. Sayward. 1978. Hints on Test Data Selection: Help for the Practicing Programmer. Computer 11 (1978), 34–41. https://api.semanticscholar.org/CorpusID:342522

    1978

  4. [12]

    Yinlin Deng, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang. 2023. Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models. In Proceedings of the 32nd ACM SIGSOFT international symposium on software testing and...

    2023

  5. [13]

    Yinlin Deng, Chunqiu Steven Xia, Chenyuan Yang, Shizhuo Dylan Zhang, Shujing Yang, and Lingming Zhang. 2023. Large language models are edge-case fuzzers: Testing deep learning libraries via fuzzgpt.arXiv preprint arXiv:2304.02014 (2023)

    2023 arXiv

  6. [14]

    Yinlin Deng, Chenyuan Yang, Anjiang Wei, and Lingming Zhang. 2022. Fuzzing deep-learning libraries via automated relational API inference. Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (2022)...

    2022

  7. [15]

    Junhua Ding, Xiaojun Kang, and Xin-Hua Hu. 2017. Validating a deep learning framework by metamorphic testing. In 2017 IEEE/ACM 2nd International Workshop on Metamorphic Testing (MET) . IEEE, 28–34

    2017

  8. [16]

    Xiaotao Feng, Ruoxi Sun, Xiaogang Zhu, Minghui Xue, Sheng Wen, Dongxi Liu, Surya Nepal, and Yang Xiang. 2021. Snipuzz: Black-box Fuzzing of IoT Firmware via Message Snippet Inference. Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security (2021)....

    2021

  9. [17]

    Andrea Fioraldi, Dominik Christian Maier, Heiko Eißfeldt, and Marc Heuse. 2020. AFL++ : Combining Incremental Steps of Fuzzing Research. In WOOT @ USENIX Security Symposium . https://api.semanticscholar.org/CorpusID:221178641 , Vol. , No. , Article . Publication date: . 26 MU et al

    2020

  10. [18]

    Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin 76, 5 (1971), 378

    1971

  11. [19]

    Jiazhen Gu, Xuchuan Luo, Yangfan Zhou, and Xin Wang. 2022. Muffin: Testing deep learning libraries via neural architecture fuzzing. In Proceedings of the 44th International Conference on Software Engineering . 1418–1430

    2022

  12. [20]

    Qianyu Guo, Xiaofei Xie, Yi Li, Xiaoyu Zhang, Yang Liu, Xiaohong Li, and Chao Shen. 2020. Audee: Automated testing for deep learning frameworks. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering. 486–498

    2020

  13. [21]

    Elahe Habibi and Seyed-Hassan Mirian-Hosseinabadi. 2015. Event-driven web application testing based on model-based mutation testing. Inf. Softw. Technol. 67 (2015), 159–179. https://api.semanticscholar.org/CorpusID:39301616

    2015

  14. [22]

    Junxiao Han, Emad Shihab, Zhiyuan Wan, Shuiguang Deng, and Xin Xia. 2020. What do Programmers Discuss about Deep Learning Frameworks. Empirical Software Engineering 25 (2020), 2694 – 2747. https://api.semanticscholar.org/ CorpusID:208522886

    2020

  15. [23]

    Qiang Hu, Lei Ma, Xiaofei Xie, Bing Yu, Yang Liu, and Jianjun Zhao. 2019. Deepmutation++: A mutation testing frame- work for deep learning systems. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 1158–1161

    2019

  16. [24]

    Nargiz Humbatova, Gunel Jahangirova, and Paolo Tonella. 2021. Deepcrime: mutation testing of deep learning systems based on real faults. In Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis . 67–78

    2021

  17. [25]

    Goodfellow Ian, Bengio Yoshua, and Courville Aaron. 2017. Deep learning: Adaptive computation and machine learning

    2017

  18. [26]

    Li Jia, Hao Zhong, Xiaoyin Wang, Linpeng Huang, and Xuansheng Lu. 2021. The symptoms, causes, and repairs of bugs inside a deep learning library. Journal of Systems and Software 177 (2021), 110935

    2021

  19. [27]

    Jittor Accessed: 2022. Jittor. https://cg.cs.tsinghua.edu.cn/jittor/

    2022

  20. [28]

    Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. nature 521, 7553 (2015), 436–444

    2015

  21. [29]

    Junqiang Li, Senyi Li, Jiawei Wu, Long Luo, Yang Bai, and Hongfang Yu. 2022. MMOS: Multi-Staged Mutation Operator Scheduling for Deep Learning Library Testing. InGLOBECOM 2022-2022 IEEE Global Communications Conference . IEEE, 6103–6108

    2022

  22. [30]

    Meiziniu Li, Jialun Cao, Yongqiang Tian, Tsz On Li, Ming Wen*, and Shing-Chi Cheung*. 2023. COMET: Coverage-guided Model Generation For Deep Learning Library Testing. ACM Transactions on Software Engineering and Methodology (2023)

    2023

  23. [31]

    Xia Li, Wei Li, Yuqun Zhang, and Lingming Zhang. 2019. Deepfl: Integrating multiple fault diagnosis dimensions for deep fault localization. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. 169–180

    2019

  24. [32]

    Jiawei Liu, Yuheng Huang, Zhijie Wang, Lei Ma, Chunrong Fang, Mingzheng Gu, Xufan Zhang, and Zhenyu Chen

  25. [33]

    Jiawei Liu, Jinjun Peng, Yuyao Wang, and Lingming Zhang. 2023. Neuri: Diversifying dnn generation via inductive rule inference. arXiv preprint arXiv:2302.02261 (2023)

    2023 arXiv

  26. [34]

    Weisi Luo, Dong Chai, Xiaoyue Ruan, Jiang Wang, Chunrong Fang, and Zhenyu Chen. 2021. Graph-based fuzz testing for deep learning inference engines. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) . IEEE, 288–299

    2021

  27. [35]

    Lei Ma, Fuyuan Zhang, Jiyuan Sun, Minhui Xue, Bo Li, Felix Juefei-Xu, Chao Xie, Li Li, Yang Liu, Jianjun Zhao, et al

  28. [36]

    Tarek Makkouk, Dong Jae Kim, and Tse-Hsun Peter Chen. 2022. An Empirical Study on Performance Bugs in Deep Learning Frameworks. 2022 IEEE International Conference on Software Maintenance and Evolution (ICSME) (2022), 35–46. https://api.semanticscholar.org/CorpusID:254903881

    2022

  29. [37]

    Tarek Makkouk, Dong Jae Kim, and Tse-Hsun Peter Chen. 2022. An empirical study on performance bugs in deep learning frameworks. In 2022 ieee international conference on software maintenance and evolution (icsme) . IEEE, 35–46

    2022

  30. [38]

    MindSpore

    MindSpore Accessed: 2020. MindSpore. https://www.mindspore.cn/

    2020

  31. [39]

    MindSpore issue report

    MindSporeDfect Accessed: 2020. MindSpore issue report. https://gitee.com/mindspore/mindspore/issues

    2020

  32. [40]

    Ziad Obermeyer and Ezekiel J Emanuel. 2016. Predicting the future—big data, machine learning, and clinical medicine. The New England journal of medicine 375, 13 (2016), 1216

    2016

  33. [41]

    ONNX Accessed: 2019. ONNX. https://onnx.ai/

    2019

  34. [42]

    Rohan Padhye, Caroline Lemieux, Koushik Sen, Mike Papadakis, and Yves Le Traon. 2018. Semantic fuzzing with zest. Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (2018). https: //api.semanticscholar.org/CorpusID:143423391 , Vol. , N...

    2018

  35. [43]

    Leo Hyun Park, Soochang Chung, Jaeuk Kim, and Taekyoung Kwon. 2023. GradFuzz: Fuzzing deep neural networks with gradient vector coverage for adversarial examples. Neurocomputing 522 (2023), 165–180

    2023

  36. [44]

    Hung Viet Pham, Thibaud Lutellier, Weizhen Qi, and Lin Tan. 2019. CRADLE: cross-backend validation to detect and localize bugs in deep learning libraries. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) . IEEE, 1027–1038

    2019

  37. [45]

    PyTorch Accessed: 2019. Pytorch. https://pytorch.org/

    2019

  38. [46]

    PyTorch issue report

    PytorchDefect Accessed: 2018. PyTorch issue report. https://github.com/pytorch/pytorch/issues

    2018

  39. [47]

    Data Avaiable

    sharelink Accessed: 2024. Data Avaiable. https://github.com/icse2024DLFrameworkempirical/ icse2024DLFrameworkempirical

    2024

  40. [48]

    Florian Tambon, Amin Nikanjam, Le An, Foutse Khomh, and Giuliano Antoniol. 2021. Silent Bugs in Deep Learning Frameworks: An Empirical Study of Keras and TensorFlow. ArXiv abs/2112.13314 (2021). https://api.semanticscholar. org/CorpusID:245502137

    2021 arXiv

  41. [49]

    Florian Tambon, Amin Nikanjam, Le An, Foutse Khomh, and Giuliano Antoniol. 2024. Silent bugs in deep learning frameworks: an empirical study of keras and tensorflow. Empirical Software Engineering 29, 1 (2024), 10

    2024

  42. [50]

    Tensorflow

    Tensorflow Accessed: 2019. Tensorflow. https://www.tensorflow.org/

    2019

  43. [51]

    TensorFlow issue report

    TensorFlowDefect Accessed: 2016. TensorFlow issue report. https://github.com/tensorflow/tensorflow/issues

    2016

  44. [52]

    teslanews

    teslanews Accessed: 2023. teslanews. https://www.tesladeaths.com/

    2023

  45. [53]

    Jiannan Wang, Thibaud Lutellier, Shangshu Qian, Hung Viet Pham, and Lin Tan. 2022. EAGLE: creating equivalent graphs to test deep learning libraries. In Proceedings of the 44th International Conference on Software Engineering . 798–810

    2022

  46. [54]

    Zan Wang, Ming Yan, Junjie Chen, Shuang Liu, and Dongdi Zhang. 2020. Deep learning library testing via effective model generation. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering ...

    2020

  47. [55]

    Deng, Chenyuan Yang, and Lingming Zhang

    Anjiang Wei, Y. Deng, Chenyuan Yang, and Lingming Zhang. 2022. Free Lunch for Testing: Fuzzing Deep-Learning Libraries from Open Source. 2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE) (2022), 995–1007. https://api.semanticscholar.org/CorpusID:246016475

    2022

  48. [56]

    Zhang, and Michael W

    Danning Xie, Yitong Li, Mijung Kim, Hung Viet Pham, Lin Tan, X. Zhang, and Michael W. Godfrey. 2021. Leveraging Documentation to Test Deep Learning Library Functions. ArXiv abs/2109.01002 (2021). https://api.semanticscholar. org/CorpusID:237386423

    2021 arXiv

  49. [57]

    Zhang, Lin Tan, Gang Wang, and Zan Wang

    Ming Yan, Junjie Chen, X. Zhang, Lin Tan, Gang Wang, and Zan Wang. 2021. Exposing numerical bugs in deep learning via gradient back-propagation. Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software E...

    2021

  50. [58]

    Chenyuan Yang, Yinlin Deng, Jiayi Yao, Yuxing Tu, Hanchi Li, and Lingming Zhang. 2023. Fuzzing Automatic Differentiation in Deep-Learning Libraries. 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) (2023), 1174–1186. https://api.semanticscholar.org/Co...

    2023

  51. [59]

    Ru Zhang, Wencong Xiao, Hongyu Zhang, Yu Liu, Haoxiang Lin, and Mao Yang. 2020. An Empirical Study on Program Failures of Deep Learning Jobs. 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE) (2020), 1159–1170. https://api.semanticscholar.org/CorpusID:...

    2020

  52. [60]

    Ru Zhang, Wencong Xiao, Hongyu Zhang, Yu Liu, Haoxiang Lin, and Mao Yang. 2020. An empirical study on program failures of deep learning jobs. In Proceedings of the ACM/IEEE 42nd international conference on software engineering . 1159–1170

    2020

  53. [61]

    Xufan Zhang, Jiawei Liu, Ning Sun, Chunrong Fang, Jia Liu, Jiang Wang, Dong Chai, and Zhenyu Chen. 2021. Duo: Differential fuzzing for deep learning operators. IEEE Transactions on Reliability 70, 4 (2021), 1671–1685

    2021

  54. [62]

    Xufan Zhang, Ning Sun, Chunrong Fang, Jiawei Liu, Jia Liu, Dong Chai, Jiang Wang, and Zhenyu Chen. 2021. Predoo: precision testing of deep learning operators. In Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis. 400–412

    2021

  55. [63]

    Xu Zhang, Yong Xu, Qingwei Lin, Bo Qiao, Hongyu Zhang, Yingnong Dang, Chunyu Xie, Xinsheng Yang, Qian Cheng, Ze Li, et al. 2019. Robust log-based anomaly detection on unstable log data. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conferen...

    2019

  56. [64]

    Yinglong Zou, Haofeng Sun, Chunrong Fang, Jiawei Liu, and Zhenping Zhang. 2023. Deep learning framework testing via hierarchical and heuristic model generation. Journal of Systems and Software 201 (2023), 111681. , Vol. , No. , Article . Publication date:

    2023

  57. [2018]

    In 2018 IEEE 29th international symposium on software reliability engineering (ISSRE)

    Deepmutation: Mutation testing of deep learning systems. In 2018 IEEE 29th international symposium on software reliability engineering (ISSRE). IEEE, 100–111

    2018

  58. [2023]

    ACM Transactions on Software Engineering and Methodology 33, 2 (2023), 1–28

    Generation-based Differential Fuzzing for Deep Learning Libraries. ACM Transactions on Software Engineering and Methodology 33, 2 (2023), 1–28

    2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.