Pith. sign in

REVIEW 3 major objections 6 minor 41 references

Subgraph-Oriented Testing for Deep Learning Libraries

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that testing frequent subgraphs of real model computation graphs with realistic inputs detects thousands of precision bugs in PyTorch that individual API testing misses.

desk verdict Nice new testing granularity—frequent subgraphs of computation graphs—but the paper's headline result rests on an unvalidated CPU-GPU threshold oracle, so the 3,376 'precision bugs' are not yet credible. read the letter →

arxiv 2412.06430 v2 pith:2IGPSJRG submitted 2024-12-09 cs.SE

Xiaoyuan Xie , Yan Song , Songqiang Chen , Jinfu Chen This is my paper · ORCID
classification cs.SE
keywords deeplearninglibrarytestingfrequentsubgraphminingprecisionbugsdifferentialinputvalidityAPIinteractionPyTorchcomputationgraphs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SORT proposes that deep learning library testing should shift from individual APIs to frequent subgraphs of model computation graphs, which capture API interactions that actually occur in real models. The method mines 728 such subgraphs from 49 popular PyTorch models, collects runtime input features (type, shape, value range) by instrumenting API calls, and generates test inputs that mimic real data. Differential testing on CPU and GPU then reports 3,376 precision-bug cases, of which 18 were identified as precision bugs in PyTorch and reported to developers, while all baseline methods miss the precision bugs. A sympathetic reader would take away that realistic API interactions plus realistic inputs expose small numerical differences that single-API testing cannot see.

What carries the argument

The central object is the frequent subgraph of a model computation graph: a connected set of two to seven deep learning library API calls whose data-flow edges appear repeatedly across a collection of real models. SORT mines these with the gSpan algorithm, instruments each API to record the type, shape, and value range of real runtime inputs, and generates entry-API inputs from those records while feeding preceding APIs' outputs into the non-entry APIs' dependent parameters. Differential testing then compares CPU and GPU outputs element-wise against a 1e-3 tolerance, with the subgraph's intermediate outputs recorded so that a violated threshold implicates the failing API and its predecessors.

What would settle it

Run SORT's 728 subgraphs with the same generated inputs but compare each output against a float64 CPU reference instead of the float32 GPU result; if most differences above 1e-3 persist when the GPU result is also compared to that reference, the CPU-vs-GPU oracle is mostly reporting hardware rounding rather than library bugs.

Watch

Extended reading notes

Core claim

SORT's central claim is that testing frequent subgraphs of computation graphs, sequences of two to seven APIs that recur across popular models, with inputs generated from observed runtime data, detects more precision bugs than testing individual APIs or arbitrarily combined APIs. In the paper's evaluation on 728 subgraphs from 49 PyTorch models, SORT achieves a 100% valid input generation rate, reports 3,376 precision-bug cases across 15 frequently used APIs, and identifies 18 precision bugs in PyTorch that were reported to developers and tagged as numerical-stability issues. The method also demonstrates that API interactions magnify small CPU/GPU output differences: several APIs never exceed the 1e-3 threshold in single-API testing yet do so when preceded by other APIs in the subgraph.

Load-bearing premise

The load-bearing premise is that any element-wise output difference above 1e-3 between CPU and GPU marks a precision bug, with no confirmation that such differences are not simply legitimate floating-point non-associativity.

Editorial extensions

If this is right

  • SORT's 100% valid input rate suggests that runtime-derived features capture the constraints that static documentation-based and code-based fuzzers miss.
  • Precision bugs in heavily used APIs such as conv2d, batch_norm, linear, and __add__ that are invisible in single-API testing become detectable when interactions amplify small differences.
  • Testing the top-20% most frequent subgraphs prioritises API interactions most likely to affect real model deployments.
  • The reported 18 PyTorch precision bugs, tagged numerical-stability by developers, indicate that some CPU/GPU discrepancies at or above the threshold are acknowledged as impactful.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the 1e-3 CPU/GPU oracle is kept without a reference-computation check, the 3,376 figure likely counts benign floating-point non-associativity together with genuine defects; a float64 reference or confirmed-issue validation would be needed to separate the two.
  • The approach transfers to other libraries and frameworks whenever computation graphs can be built, so the same subgraph-mining plus runtime-feature pipeline could be pointed at other deep learning frameworks.
  • A natural extension is severity ranking: if small precision differences are weighted by how many downstream model deployments use the triggering subgraph, the reported cases could be prioritised by real-world impact.
  • One could test whether the magnification effect is necessary by comparing SORT against a version that adds random intermediate perturbations; if random perturbations reproduce the same precision cases, the specific subgraph choice matters less than the perturbation itself.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes SORT, a differential testing approach for deep learning libraries that uses frequent subgraphs of model computation graphs as test subjects and generates inputs from runtime input features (type, shape, value range) collected by instrumenting models executed on real datasets. The evaluation on 728 subgraphs from 49 PyTorch models reports a 100% valid input generation rate, 3,376 precision bugs detected versus near-zero for five baseline methods, and 18 precision bugs reported to PyTorch developers. An ablation (Single-API) and a developer survey are used to support the claim that API interactions magnify small numerical differences into detectable errors. The central methodological idea—testing realistic API interaction patterns at subgraph granularity—is plausible and novel, but the quantitative evaluation has load-bearing issues: the 100% validity rate is circular as reported, the precision-bug oracle is unvalidated and conflates threshold violations with confirmed defects, and the relationship between the 3,376 candidate bugs and the 18 reported bugs is never reconciled.

Significance. If the claims were supported, SORT would make a useful contribution to DL library testing by providing a middle ground between model-level testing (realistic interactions, poor localization) and API-level testing (good localization, unrealistic inputs). The use of frequent computation subgraphs as test subjects is a reasonable and transferable idea, and the instrumentation-based input generation is practical. The paper also releases an artifact, compares against several established baselines, and includes an ablation for API interaction. However, the strongest quantitative claims—100% valid inputs and detecting more precision bugs than baselines—are not established by the reported experiments because the metrics are either constructed to be 100% or rest on an unvalidated oracle. The significance of the paper therefore depends on a revision that provides confirmed bug counts, honest generation denominators, and a clear separation between candidate numerical differences and verified defects.

major comments (3)
  1. [Section V-A, Table I] The RQ1 comparison is circular as reported. Section V-A states that the protocol 'run[s] each method to generate a total of 15,000 valid test inputs' and, for SORT, 'ensure[s] a total of 15,000 valid test inputs for APIs in all subgraphs.' Given the metric R defined in Section IV-D as valid inputs divided by all generated inputs, fixing the numerator at 15,000 without reporting SORT's total generation attempts makes the 100% rate a consequence of the protocol rather than a measured property of the generator. Please report the denominator (total attempts) for SORT and for every baseline, and specify the stopping rule applied when the 1,000-valid-input target per API is not reached within 10,000 attempts.
  2. [Section IV-E, Section V-B, Abstract] The precision-bug oracle is not validated, so the headline RQ2 comparison is unsupported. Section IV-E defines a precision bug solely as an element-wise CPU-GPU output difference exceeding 1e-3, but such differences can legitimately arise from floating-point non-associativity, different reduction orders, TF32 mode, cuDNN heuristics, and hardware-specific math implementations. Section V-C then explicitly states 'we cannot confirm the buggy APIs,' while Table II reports 3,376 precision bugs. The abstract's '18 precision bugs' is never reconciled with that count. Please (a) distinguish confirmed or developer-triaged bugs from raw threshold violations, (b) describe the confirmation process that led to the 18 issues, and (c) either use confirmed bugs as the primary RQ2 metric or reword the claims to say 'threshold violations.' Without this, the claim that SORT detects more precision bugs than existing methods is not established.
  3. [Section V-C, Fig. 14] The RQ3 evidence conflates amplification of legitimate numerical differences with bug detection. The example in Fig. 14 shows layer_norm's sub-threshold difference being magnified by linear, but because no API is confirmed buggy, an above-threshold final difference only demonstrates that floating-point error can accumulate through an interaction; it does not demonstrate that an API interaction reveals a defect. The frequency metric in Fig. 13b is computed from threshold-violating executions, so calling these 'erroneous behaviors' is misleading unless those executions are independently confirmed. Please add manual triage or controlled reproductions that separate genuine bugs from expected numerical variation.
minor comments (6)
  1. [Fig. 13] The two subfigures contain garbled/unrendered label text (e.g., the '/uni00000044/...' sequences); the labels need to be regenerated so each API name is readable.
  2. [Section III-A] The description of computation graph construction is brief ('iterate through the model code, identify each API function call, and map it to a graph node'). Please clarify how API calls are resolved, how aliases and control flow are handled, and whether the construction was validated on the 49 models.
  3. [Section IV-E] The hardware configuration is underspecified: the TF32 setting, GPU model and driver, cuDNN version, and PyTorch version (2.3.0 is mentioned later) should be stated, since they directly affect the magnitude of CPU-GPU differences used by the oracle.
  4. [Section V-A] There is a typo in the legend text: 'ACETets-G' should be 'ACETest-G.'
  5. [Section V-A] The phrase 'we can collect the top 20% frequently appearing subgraphs' in Section IV-E is not explained; reporting the distribution of subgraph frequencies and a sensitivity analysis for the frequency threshold would make the choice of 728 subgraphs more transparent.
  6. [Section VII] The threats-to-validity section addresses dataset completeness and generalizability but does not discuss the validity of the precision-bug oracle or the circularity of the valid-input metric; both should be acknowledged as threats.

Circularity Check

2 steps flagged · score 6.0 of 10

Two SORT metrics are partly constructed: the 100% valid-input rate is ensured before measurement, and API 'involvement' is defined to include all predecessors in a subgraph; the core differential-testing comparison remains external.

  1. fitted input called prediction [Section V-A (RQ1), Table I]
    "For a fair comparison, we run each method to generate a total of 15,000 valid test inputs across 15 target APIs. For baseline methods, we tried to generate 1,000 valid test inputs for each API by following their setup and we recorded the total number of test inputs generated. For SORT, we ensure a total of 15,000 valid test inputs for APIs in all subgraphs."

    The validity rate is defined as R = number of API inputs that passed validity check / number of all API inputs. For baselines, the paper records total generation attempts and even halts after 10,000 attempts per 1,000 valid inputs, but for SORT it reports no total number of generation attempts; it simply 'ensures' 15,000 valid inputs. The 100% figure in Table I is therefore a selection precondition rather than a measured generation rate. Announcing the number of valid inputs before measuring the rate makes RQ1's headline validity result equivalent to the experimental setup instead of an empirical finding.

  2. self definitional [Section V-C (RQ3), Fig. 13b]
    "Given that we cannot confirm the buggy APIs, we do not study the explicit relation between API interactions and actual bug detection ability in this work. ... when SORT reveals a bug, both the API whose outputs violate the difference threshold and its previous APIs in the frequent subgraph relate to the observed abnormal behavior; meanwhile, single-API testing only associates the API under testing with the observed abnormal behavior."

    The metric 'API involved in erroneous behavior' is defined so that every predecessor of an over-threshold API counts as involved when SORT runs a subgraph, while single-API testing credits only the one API under test. The later finding that SORT involves twelve APIs and single-API testing involves only three therefore follows in part from the definition of 'involved,' not from independent evidence that those predecessor APIs are buggy; the paper itself concedes it cannot confirm buggy APIs. The involvement-frequency comparison is thus partly definitional, although the average output-difference comparison in Fig. 13a is empirical.

full rationale

SORT is an empirical differential-testing study rather than a mathematical derivation, and most of its comparison is against external baselines (FreeFuzz, DeepREL, DocTer, ACETest, TitanFuzz) on the same PyTorch APIs. The 18 reported/triaged issues and the externally sourced 1e-3 threshold give the core precision-bug claim independent content; the paper's self-citations ([12], [41]) are background and not load-bearing. However, two headline metrics are partly constructed. First, the 100% valid-input rate in RQ1 is reported after the paper 'ensures' 15,000 valid SORT inputs, with no count of SORT generation attempts, so the validity ratio is not independently evaluable from the reported data. Second, RQ3's 'involvement' metric defines all predecessors of an over-threshold API as involved for SORT while single-API testing credits only the tested API, making part of the 12-vs-3 involvement result true by definition; the paper explicitly disclaims confirmed buggy APIs. The unvalidated CPU-GPU oracle is a threat to validity and correctness, but that is not circularity because the paper does not claim the threshold is derived from the bug counts. Overall, the central differential-testing comparison remains externally grounded, so the circularity is partial rather than total.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper's central claims depend on two hand-chosen thresholds and the representativeness of 49 OpenMMLab models. No invented entities are introduced.

free parameters (2)
  • tolerance threshold = 1e-3
    Element-wise absolute difference between CPU and GPU outputs used to define a precision bug; adopted from prior work, not validated for this dataset.
  • frequency threshold for gSpan = 5 (top 20%)
    Minimum support for frequent subgraph extraction, chosen from preliminary experiments to include top 20% of subgraphs; determines which APIs are tested.
assumptions (3)
  • domain assumption CPU-GPU output differences above 1e-3 indicate a bug in the DL library.
    Central oracle; not validated against confirmed defects.
  • domain assumption Static analysis of model source code produces computation graphs that faithfully represent runtime API interactions.
    Section III-A constructs graphs by iterating through code and signature analysis.
  • domain assumption The 49 OpenMMLab models and their recommended datasets are representative of real-world DL usage.
    Section IV-E; generalizability of the evaluation depends on this.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Subgraph-Oriented Testing for Deep Learning Libraries." pith.science (2026). https://pith.science/paper/2IGPSJRG

@misc{pith2026241206430,
  author       = {Pith},
  title        = {Pith review of: Subgraph-Oriented Testing for Deep Learning Libraries},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2IGPSJRG}},
  note         = {Machine review of arXiv:2412.06430}
}
read the original abstract

Deep Learning (DL) libraries, such as PyTorch, are widely used for building and deploying DL models on various hardware platforms. Meanwhile, they are found to contain bugs that lead to incorrect calculation results and cause issues like non-convergence training and inaccurate prediction of DL models. Thus, many efforts have been made to test DL libraries and reveal bugs. However, existing DL library testing methods manifest limitations: model-level testing methods cause complexity in fault localization. Meanwhile, API-level testing methods often generate invalid inputs or primarily focus on extreme inputs that lead to crash failures; they also ignore testing realistic API interactions. These limitations may lead to missing detection of bugs, even in the frequently used APIs. To address these limitations, we propose SORT (Subgraph-Oriented Realistic Testing) to differential test DL libraries on different hardware platforms. SORT takes popular API interaction patterns, represented as frequent subgraphs of model computation graphs, as test subjects. In this way, it introduces realistic API interaction sequences while maintaining efficiency in locating faulty APIs for observed errors. Besides, SORT prepares test inputs by referring to extensive features of runtime inputs for each API in executing real-life benchmark data. The generated inputs are expected to better simulate such valid real inputs and reveal bugs more likely to happen in real-life usage. Evaluation on 728 frequent subgraphs of 49 popular PyTorch models demonstrates that SORT achieves a 100% valid input generation rate, detects more precision bugs than existing methods, and reveals interaction-related bugs missed by single-API testing. 18 precision bugs in PyTorch are identified.

Figures

Figures reproduced from arXiv: 2412.06430 by the authors.

Figure 3
Figure 3. Example Precision Differences in Testing with API [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Part of the Code Defining the ResNet Model [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figure 5
Figure 5. Part of the Code Defining the DenseNet Model [PITH_FULL_IMAGE:figures/full_fig_p003_5.png] view at source ↗
Figures from the paper (9 more)
Figure 2
Figure 2. Figure 2: Example Precision Differences in Testing Single API [PITH_FULL_IMAGE:figures/full_fig_p003_2.png]
Figure 6
Figure 6. Figure 6: Computation Graphs of ResNet and DenseNet Models [PITH_FULL_IMAGE:figures/full_fig_p004_6.png]
Figure 7
Figure 7. Figure 7: Overview of SORT to record input feature information during each execution. For the tensor parameters, we record feature information about the type, shape, and value range. For the non-tensor parameters, the values are recorded. When an API is invoked once in a model w…
Figure 8
Figure 8. Figure 8: Recorded Input Feature Information of conv2d C. Test Input Generation With the collected feature information, we generate test inputs for each frequent subgraph. To systematically generate the input values for parameters, we categorize parameters into three types: para…
Figure 9
Figure 9. Figure 9: Recorded Input Feature Information of batch_norm D. Test Case Execution and Result Verification The last step of SORT is to perform differential testing over each frequent graph using the generated test inputs. We follow existing works [20], [14], [18] to run the test …
Figure 10
Figure 10. Figure 10: Invalid Test Case Generated by FreeFuzz and ACETest-G7 show ratios of valid inputs of 6.80% and 6.72%, respectively. They generate invalid inputs due to their inability to accurately solve constraints in input validation loops, C++ pointers, and tensor constraints [19…
Figure 12
Figure 12. Figure 12: Invalid Test Case Generated by TitanFuzz [PITH_FULL_IMAGE:figures/full_fig_p008_12.png]
Figure 14
Figure 14. Figure 14: An Example of Subgraph Magnifying Difference [PITH_FULL_IMAGE:figures/full_fig_p010_14.png]
Figure 13
Figure 13. Figure 13: Testing Results of SORT and Single-API Testing [PITH_FULL_IMAGE:figures/full_fig_p010_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 22 canonical work pages

  1. [1]

    Computational ap- proaches for circrna-disease association prediction: a review,

    M. Niu, Y . Chen, C. Wang, Q. Zou, and L. Xu, “Computational ap- proaches for circrna-disease association prediction: a review,” Frontiers of Computer Science , vol. 19, no. 4, p. 194904, 2025

    work page 2025

  2. [2]

    Deep learning for recommender systems: A netflix case study,

    H. Steck, L. Baltrunas, E. Elahi, D. Liang, Y . Raimond, and J. Basilico, “Deep learning for recommender systems: A netflix case study,” AI Magazine, vol. 42, no. 3, pp. 7–18, 2021

    work page 2021

  3. [3]

    Deep learning–based text classification: a comprehensive review,

    S. Minaee, N. Kalchbrenner, E. Cambria, N. Nikzad, M. Chenaghlu, and J. Gao, “Deep learning–based text classification: a comprehensive review,”ACM computing surveys (CSUR), vol. 54, no. 3, pp. 1–40, 2021

    2021

  4. [4]

    Image segmentation using deep learning: A survey,

    S. Minaee, Y . Boykov, F. Porikli, A. Plaza, N. Kehtarnavaz, and D. Terzopoulos, “Image segmentation using deep learning: A survey,” IEEE transactions on pattern analysis and machine intelligence, vol. 44, no. 7, pp. 3523–3542, 2021

    2021

  5. [5]

    Prediction of heart disease using a combination of machine learning and deep learning,

    R. Bharti, A. Khamparia, M. Shabaz, G. Dhiman, S. Pande, and P. Singh, “Prediction of heart disease using a combination of machine learning and deep learning,” Computational intelligence and neuroscience , vol. 2021, 2021

    work page 2021

  6. [6]

    Large sequence models for sequential decision-making: a survey,

    M. Wen, R. Lin, H. Wang, Y . Yang, Y . Wen, L. Mai, J. Wang, H. Zhang, and W. Zhang, “Large sequence models for sequential decision-making: a survey,” Frontiers of Computer Science , vol. 17, no. 6, p. 176349, 2023

    work page 2023

  7. [7]

    Pytorch: An imperative style, high-performance deep learning library,

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al. , “Pytorch: An imperative style, high-performance deep learning library,” Advances in neural information processing systems , vol. 32, 2019

    2019

  8. [8]

    A survey of deep learning library testing methods,

    X. Zhang, W. Jiang, C. Shen, Q. Li, Q. Wang, C. Lin, and X. Guan, “A survey of deep learning library testing methods,” arXiv preprint arXiv:2404.17871, 2024

    arXiv 2024

Show all 41 references
  1. [9]

    A comprehensive study of real-world numerical bug characteristics,

    A. Di Franco, H. Guo, and C. Rubio-Gonz ´alez, “A comprehensive study of real-world numerical bug characteristics,” in 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 2017, pp. 509–519

    2017

  2. [10]

    CRADLE: Cross-backend validation to detect and localize bugs in deep learning libraries,

    H. V . Pham, T. Lutellier, W. Qi, and L. Tan, “CRADLE: Cross-backend validation to detect and localize bugs in deep learning libraries,” in 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, may 2019

    2019

  3. [11]

    Generation-based differential fuzzing for deep learning libraries,

    J. Liu, Y . Huang, Z. Wang, L. Ma, C. Fang, M. Gu, X. Zhang, and Z. Chen, “Generation-based differential fuzzing for deep learning libraries,” ACM Trans. Softw. Eng. Methodol. , vol. 33, no. 2, dec 2023. [Online]. Available: https://doi.org/10.1145/3628159

    2023 doi

  4. [12]

    Audee: Au- tomated testing for deep learning frameworks,

    Q. Guo, X. Xie, Y . Li, X. Zhang, Y . Liu, X. Li, and C. Shen, “Audee: Au- tomated testing for deep learning frameworks,” in 2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE) . Melbourne, VIC, Australia: IEEE, 2020, pp. 486–498

    2020

  5. [13]

    Deep learning library testing via effective model generation,

    Z. Wang, M. Yan, J. Chen, S. Liu, and D. Zhang, “Deep learning library testing via effective model generation,” in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering . ACM, nov 2020

    2020

  6. [14]

    Free lunch for testing,

    A. Wei, Y . Deng, C. Yang, and L. Zhang, “Free lunch for testing,” in Proceedings of the 44th International Conference on Software Engineer- ing. ACM, may 2022

    2022

  7. [15]

    Improving fault localization with pre-training,

    Z. Zhang, Y . Li, J. Xue, and X. Mao, “Improving fault localization with pre-training,” Frontiers of Computer Science , vol. 18, no. 1, p. 181205, 2024

    2024

  8. [16]

    Docter: documentation-guided fuzzing for testing deep learning api functions,

    D. Xie, Y . Li, M. Kim, H. V . Pham, L. Tan, X. Zhang, and M. W. Godfrey, “Docter: documentation-guided fuzzing for testing deep learning api functions,” in Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, ser. ISSTA 2022. New York,...

    2022

  9. [17]

    Fuzzing deep-learning libraries via automated relational API inference,

    Y . Deng, C. Yang, A. Wei, and L. Zhang, “Fuzzing deep-learning libraries via automated relational API inference,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering . ACM, nov 2022

    2022

  10. [18]

    Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models,

    Y . Deng, C. S. Xia, H. Peng, C. Yang, and L. Zhang, “Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models,” in Proceedings of the 32nd ACM SIGSOFT interna- tional symposium on software testing and analysis , 2023, pp. 423–435

    2023

  11. [19]

    Acetest: Automated constraint extraction for testing deep learning operators,

    J. Shi, Y . Xiao, Y . Li, Y . Li, D. Yu, C. Yu, H. Su, Y . Chen, and W. Huo, “Acetest: Automated constraint extraction for testing deep learning operators,” arXiv preprint arXiv:2305.17914 , 2023

    2023 arXiv

  12. [20]

    Predoo: precision testing of deep learning operators,

    X. Zhang, N. Sun, C. Fang, J. Liu, J. Liu, D. Chai, J. Wang, and Z. Chen, “Predoo: precision testing of deep learning operators,” in Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, ser. ISSTA 2021. New York, NY , USA: Association f...

    2021

  13. [21]

    Fuzzing automatic differentiation in deep-learning libraries,

    C. Yang, Y . Deng, J. Yao, Y . Tu, H. Li, and L. Zhang, “Fuzzing automatic differentiation in deep-learning libraries,” in Proceedings of the 45th International Conference on Software Engineering , ser. ICSE ’23. IEEE Press, 2023, p. 1174–1186. [Online]. Available: https://doi...

    2023

  14. [22]

    Efficient generation of error-inducing floating-point inputs via symbolic execution,

    H. Guo and C. Rubio-Gonz ´alez, “Efficient generation of error-inducing floating-point inputs via symbolic execution,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering , 2020, pp. 1261–1272

    2020

  15. [23]

    Graph-based fuzz testing for deep learning inference engines,

    W. Luo, D. Chai, X. Run, J. Wang, C. Fang, and Z. Chen, “Graph-based fuzz testing for deep learning inference engines,” in Proceedings of the 43rd International Conference on Software Engineering , ser. ICSE ’21. Madrid, Spain: IEEE Press, 11 2021, p. 288–299. [Online]. Availa...

    2021

  16. [24]

    Muffin: Testing deep learning libraries via neural architecture fuzzing,

    J. Gu, X. Luo, Y . Zhou, and X. Wang, “Muffin: Testing deep learning libraries via neural architecture fuzzing,” in Proceedings of the 44th International Conference on Software Engineering . Pittsburgh, PA, USA: IEEE, 2022, pp. 1418–1430

    2022

  17. [25]

    Comet: Coverage-guided model generation for deep learning library testing,

    M. Li, J. Cao, Y . Tian, T. O. Li, M. Wen*, and S.-C. Cheung*, “Comet: Coverage-guided model generation for deep learning library testing,” ACM Transactions on Software Engineering and Methodology , 2 2023. [Online]. Available: https://doi.org/10.1145/3583566

    2023 doi

  18. [26]

    Eagle: creating equivalent graphs to test deep learning libraries,

    J. Wang, T. Lutellier, S. Qian, H. V . Pham, and L. Tan, “Eagle: creating equivalent graphs to test deep learning libraries,” in Proceedings of the 44th International Conference on Software Engineering , ser. ICSE ’22. New York, NY , USA: Association for Computing Machinery, 7...

    2022

  19. [27]

    Replication package and details of the survey for sort,

    “Replication package and details of the survey for sort,” https://github .com/SORT-1/SORT, 2024

    2024

  20. [28]

    Effi- cient search for inputs causing high floating-point errors,

    W.-F. Chiang, G. Gopalakrishnan, Z. Rakamaric, and A. Solovyev, “Effi- cient search for inputs causing high floating-point errors,” inProceedings of the 19th ACM SIGPLAN symposium on Principles and practice of parallel programming, 2014, pp. 43–52

    2014

  21. [29]

    An empirical study on numerical bugs in deep learning programs,

    G. Wang, Z. Wang, J. Chen, X. Chen, and M. Yan, “An empirical study on numerical bugs in deep learning programs,” in Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, 2022, pp. 1–5

    2022

  22. [30]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770–778

    2016

  23. [31]

    Densely connected convolutional networks,

    G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in Proceedings of the IEEE confer- ence on computer vision and pattern recognition , 2017, pp. 4700–4708

    2017

  24. [32]

    gspan: graph-based substructure pattern mining,

    X. Yan and J. Han, “gspan: graph-based substructure pattern mining,” in 2002 IEEE International Conference on Data Mining, 2002. Proceed- ings., 2002, pp. 721–724

    2002

  25. [33]

    A survey of frequent subgraph mining algorithms,

    C. Jiang, F. Coenen, and M. Zito, “A survey of frequent subgraph mining algorithms,” The Knowledge Engineering Review, vol. 28, no. 1, pp. 75– 105, 2013

    2013

  26. [34]

    Toward understanding deep learning framework bugs,

    J. Chen, Y . Liang, Q. Shen, J. Jiang, and S. Li, “Toward understanding deep learning framework bugs,” ACM Transactions on Software Engineering and Methodology , 3 2023. [Online]. Available: https://doi.org/10.1145/3587155

    2023 doi

  27. [35]

    An empirical study on bugs inside pytorch: A replication study,

    S. C. Y . Ho, V . Majdinasab, M. Islam, D. E. Costa, E. Shihab, F. Khomh, S. Nadi, and M. Raza, “An empirical study on bugs inside pytorch: A replication study,” in 2023 IEEE International Conference on Software Maintenance and Evolution (ICSME) . IEEE, 2023, pp. 220–231. 13

    2023

  28. [36]

    The symptoms, causes, and repairs of bugs inside a deep learning library,

    L. Jia, H. Zhong, X. Wang, L. Huang, and X. Lu, “The symptoms, causes, and repairs of bugs inside a deep learning library,” Journal of Systems and Software , vol. 177, p. 110935, 2021

    2021

  29. [37]

    Conformer: Local features coupling global representations for visual recognition,

    Z. Peng, W. Huang, S. Gu, L. Xie, Y . Wang, J. Jiao, and Q. Ye, “Conformer: Local features coupling global representations for visual recognition,” in Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 367–376

    2021

  30. [38]

    Imagenet large scale visual recognition challenge,

    O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al., “Imagenet large scale visual recognition challenge,” International journal of computer vision, vol. 115, pp. 211–252, 2015

    2015

  31. [39]

    Learning multiple layers of features from tiny images,

    A. Krizhevsky, G. Hinton et al. , “Learning multiple layers of features from tiny images,” 2009

    2009

  32. [40]

    A study of oracle approximations in testing deep learning libraries,

    M. Nejadgholi and J. Yang, “A study of oracle approximations in testing deep learning libraries,” in 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 2019, pp. 785–796

    2019

  33. [41]

    A theoretical analysis of the risk evaluation formulas for spectrum-based fault localization,

    X. Xie, T. Y . Chen, F.-C. Kuo, and B. Xu, “A theoretical analysis of the risk evaluation formulas for spectrum-based fault localization,” ACM Transactions on software engineering and methodology (TOSEM), vol. 22, no. 4, pp. 1–40, 2013

    2013

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.