REVIEW 3 major objections 4 minor 106 references
Towards Understanding the Impact of Data Bugs on Deep Learning Models in Software Engineering
T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Training-time gradients, weights, and biases can fingerprint which data bug is poisoning a deep learning model.
desk verdict A large-scale, useful map of training-time symptoms for dirty data in DL4SE, but the specific bug-to-symptom attributions are confounded because the 'buggy' datasets are pre-existing dirty corpora with multiple intertwined quality issues, not controlled injections. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is the layer-wise monitoring of gradients, weights, and biases during training, aggregated into statistics such as mean, median, variance, skewness, kurtosis, and sparsity, with Z-scores, interquartile range outliers, and distribution-shape thresholds used to flag deviations. These aggregated statistics are compared across clean, buggy, and missing-preprocessing variants to define recognizable 'symptoms'; post-hoc analyses with attention weights, t-SNE embeddings, and Grad-CAM maps are used to confirm that the affected models lose focus on meaningful input features. The comparison of parameter statistics across variants is what turns training dynamics into diagnostic fingerprints.
What would settle it
Audit the actual label error rates, class imbalance ratios, and distribution shifts in the clean and buggy variants of Devign, BigVul, Eclipse, Hadoop, OpenStack, and QT; if a buggy variant differs from its clean pair in multiple quality dimensions at once, the claimed fingerprint for the named issue would not be supported.
Extended reading notes
Core claim
The paper claims that each data type has a distinct symptom signature. Quality issues in code data produce near-zero biases, smaller weights, and gradient instability; quality issues in text data produce abnormal weight distributions, gradient skewness, and overfitting to noisy patterns; quality issues in metric data produce sparse parameter updates, vanishing gradients, and higher training loss. Missing preprocessing produces its own set of signatures: slow convergence, extreme bias values, and skewed parameter distributions for code; extreme weights and skewed bias distributions for text; and exploding gradients with high weight variance for metrics. The paper further claims that these signatures generalize across six additional datasets and that cleaning data and restoring preprocessing removes nearly all symptoms, with occurrence rates dropping from 70-85% of runs to below 12%.
Load-bearing premise
The study assumes that each buggy dataset contains only the named quality problem, and that its paired clean version isolates that problem; if a buggy dataset mixes several quality issues, the symptom-to-bug attributions do not hold.
Editorial extensions
If this is right
- Monitoring systems can flag near-zero biases, smaller weights, or gradient instability during training as early warnings of label noise or concept drift in code-based datasets.
- Text-based models that show abnormal weight distributions, gradient skewness, or unusually high bias values are candidates for a data-quality audit targeting concept drift or class imbalance.
- Metric-based models exhibiting vanishing gradients, sparse parameter updates, or stagnant loss can be traced to class imbalance, while exploding gradients and high weight variance point to missing feature normalization.
- Omitting preprocessing is itself a detectable bug: slow convergence, extreme or skewed parameter distributions, and exploding gradients appear within a few epochs and disappear after restoring preprocessing.
- Cleaning data and restoring preprocessing reduces the occurrence of these symptoms from the 70-85% range to below 12%, suggesting that monitoring can double as a validation step before deployment.
Reading between the lines
- If training-time parameter statistics really fingerprint data bugs, the same approach could be applied beyond software engineering to any domain with paired clean and buggy datasets, and the paper's own code-summarization prototype suggests this extension is plausible.
- A stronger test of the claimed signatures would train on datasets with a single artificially injected bug at controlled severity levels, producing dose-response curves for each symptom rather than relying on pre-existing buggy variants.
- The symptom-to-bug attributions assume each buggy variant differs from its clean pair only in the named issue; measuring the actual label error rates, class ratios, and distribution shifts in the variants would decide whether the signatures are specific or conflated.
- Because the prototype monitor detects the correct root cause in only 10 to 12 of 15 runs, the remaining runs hint at a false-negative tail, and a practical system would likely need to combine training-dynamics thresholds with direct data-side checks.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports an empirical study of how data-quality and preprocessing bugs affect the training behaviour of deep learning models for software engineering tasks. Using three data types (code, text, metric), two datasets per type, and state-of-the-art baselines (LineVul, CodeBERT, DCCNN, BERT-MLP, DeepJIT, CodeBERTJIT), the authors train clean and buggy variants, log gradients/weights/biases with Weights & Biases, identify statistical symptoms, and triangulate with attention, t-SNE, and GradCAM analyses. They then validate their findings on six additional datasets and present a prototype monitoring tool, DataGuardian. The headline claim is that specific data bugs (label noise, class imbalance, concept drift) and missing preprocessing produce distinct, generalizable training-time symptoms that can serve as fingerprints for detection.
Significance. If the central claim holds, this is a useful contribution: training-time gradient, weight, and bias statistics would provide a practical, low-cost signal for monitoring data quality in SE deep-learning pipelines, complementing existing performance-based evaluations. The study has clear strengths: it covers multiple data types and tasks, runs each configuration fifteen times to address stochasticity, triangulates quantitative metrics with three XAI techniques, validates on six held-out datasets, and provides a replication package and a working prototype. These design choices make the empirical skeleton credible and the results potentially actionable for MLOps-oriented work.
major comments (3)
- [§3.3.2, §3.2.4, Tables 4/8/12] The central claim that specific data bugs produce identifiable fingerprints requires that each buggy variant differ from its clean counterpart only in the named bug. Section 3.3.2 states only that buggy datasets were 'obtained from the same existing studies [10, 104, 102]' and never describes how label noise, class imbalance, or concept drift were injected into Devign, BigVul, Eclipse, Hadoop, OpenStack, or QT. Section 3.2.4, however, reports that each of these datasets contains multiple simultaneous quality issues (e.g., Devign and BigVul have label noise and temporal relevance issues; OpenStack and QT have labelling errors, class imbalance, and concept drift). If the same pre-existing dirty dataset is labelled as more than one bug type, or if a 'label-noise' variant also contains drift and imbalance, the symptom-to-bug attributions in Tables 4, 8, and 12 are confounded. Section 3.5's reference to 'controlled experiments' and to introducing 'a bug like class imbalance' does not resolve this because no injection protocol appears anywhere in the manuscript. The authors must specify the construction of each clean/buggy pair and, ideally, verify that the induced perturbation is the only difference.
- [§3.5] The symptom-determination procedure is partly circular. Section 3.5 derives the statistical thresholds (Z-score > 3, IQR fences, skewness > 1, |kurtosis| > 3) by comparing the buggy models against the bug-free baseline, and then uses those same thresholds to declare which symptoms are present in the buggy models and to attribute them to data-quality issues. Because the thresholds are fit to the very comparisons they are used to explain, the prevalence percentages in Tables 4-15 and the odds ratios in Section 4.4.2 are not independent evidence. The authors should either fix the thresholds a priori (e.g., from literature or from a separate calibration run) or clearly present the validation-dataset results as the only non-circular check, with thresholds locked before that validation.
- [§4.4 and §5.4] The generalizability evaluation is reported at too coarse a granularity to support the claim. Section 4.4 gives aggregate percentages across the six held-out datasets but no per-dataset tables, per-symptom counts, or dataset-level statistical results, so the assertion that findings align 'within a 1-5% margin' cannot be checked. Likewise, Section 5.4 reports DataGuardian's success rates (12/15, 11/15, etc.) on Juliet, Mozilla, and JDT without defining what counts as a correct root-cause identification or providing a comparison against clean-data runs; since the thresholds that define the symptoms are the same thresholds DataGuardian uses to infer root causes, this evaluation does not independently validate the bug-type attribution.
minor comments (4)
- [§4.3.1(a)] The text says 'As shown in Table 3, DeepJIT witnessed a 4.45-9.24% performance drop', but the metric-based performance results appear in Table 13, not Table 3; please correct the cross-reference.
- [§4.1.3] In the buggy-data paragraph, 'the models trained on LineVul (with label noise)' appears to refer to a dataset (Devign) rather than the model; please clarify the naming.
- [Table 1 and §3.3.1] There are small formatting slips, including 'T ext-Based' in Table 1 and 'F rameworks' in Section 3.3.1; please proofread the manuscript.
- [§4.4.2] The heading 'Signifiance Tests' is misspelled, and the odds-ratio range reported (15.53 to 806.10) should be accompanied by confidence intervals, since McNemar's test alone does not quantify uncertainty in the odds ratio.
Circularity Check
Mild circularity: symptom labels are defined from the same clean-vs-buggy comparisons later used to confirm the bug-symptom association; external held-out validation keeps the central claim independent.
-
self definitional
[Section 3.5 (Determining the Symptoms); Section 4.4.2 (Statistical Significance Tests)]
"These observed statistical deviations are the "symptoms" of data quality issues. The correlation between a symptom and a specific data quality issue is established by observing consistent patterns in our controlled experiments. ... McNemar's test results consistently rejected the null hypothesis across all symptoms for data quality and data preprocessing issues (p <0.01)."
The symptom flags are constructed by comparing buggy runs against bug-free baselines from the same experiment set (Z-score > 3, IQR outliers, skewness > 1). Section 4.4.2 then uses McNemar's test to establish that data quality issues and these symptom flags are dependent, but on the same kind of runs that generated the symptom definitions; a buggy run was, by construction, the source of the deviations that defined the symptoms. This makes the internal significance test partially a restatement of the labeling rule rather than an independent confirmation.
full rationale
The paper's headline findings are empirical observations rather than derived predictions: the bug-to-symptom mappings in Tables 4-15 come from comparing buggy and clean model runs, the statistical criteria (Z-score, IQR, skewness) are standard outlier rules, and the paper validates on six held-out datasets and evaluates DataGuardian on Juliet, Mozilla, and JDT, which were not part of the threshold-related analysis. The only mild circularity is that the symptom definitions themselves are derived from the same clean-versus-buggy comparisons that Section 4.4.2's McNemar test then confirms, making that internal significance test partly tautological. This does not undermine the central generalizability claim because the held-out validation and DataGuardian experiments provide independent checks. The lack of documented per-bug injection in the pre-existing dirty datasets is a construct-validity and confounding concern, not a circularity, and therefore does not raise the circularity score further.
Assumptions & free parameters
free parameters (3)
- Near-zero bias threshold =
< 0.01
- Outlier detection thresholds =
Z > 3; IQR 1.5
- Distribution shape thresholds =
skewness > 1; |kurtosis| > 3
assumptions (4)
- domain assumption The cleaned versions of datasets obtained from prior studies [10, 104, 102] are genuinely free of the targeted data quality issues.
- domain assumption The original uncleaned versions of the primary datasets contain predominantly the specific bug type each RQ attributes to them (label noise for code, concept drift for text, class imbalance for metrics).
- domain assumption Gradient, weight, and bias statistics logged by Weights and Biases hooks faithfully represent the learning dynamics relevant to data bugs.
- standard math McNemar's test assumptions hold for the paired symptom presence/absence data.
Cite this review
Pith. "Pith review of Towards Understanding the Impact of Data Bugs on Deep Learning Models in Software Engineering." pith.science (2026). https://pith.science/paper/IKIL3VFS
@misc{pith2026241112137,
author = {Pith},
title = {Pith review of: Towards Understanding the Impact of Data Bugs on Deep Learning Models in Software Engineering},
year = {2026},
howpublished = {\url{https://pith.science/paper/IKIL3VFS}},
note = {Machine review of arXiv:2411.12137}
}
read the original abstract
Deep learning (DL) techniques have achieved significant success in various software engineering tasks (e.g., code completion by Copilot). However, DL systems are prone to bugs from many sources, including training data. Existing literature suggests that bugs in training data are highly prevalent, but little research has focused on understanding their impacts on the models used in software engineering tasks. In this paper, we address this research gap through a comprehensive empirical investigation focused on three types of data prevalent in software engineering tasks: code-based, text-based, and metric-based. Using state-of-the-art baselines, we compare the models trained on clean datasets with those trained on datasets with quality issues and without proper preprocessing. By analysing the gradients, weights, and biases from neural networks under training, we identify the symptoms of data quality and preprocessing issues. Our analysis reveals that quality issues in code data cause biased learning and gradient instability, whereas problems in text data lead to overfitting and poor generalisation of models. On the other hand, quality issues in metric data result in exploding gradients and model overfitting, and inadequate preprocessing exacerbates these effects across all three data types. Finally, we demonstrate the validity and generalizability of our findings using six new datasets. Our research provides a better understanding of the impact and symptoms of data bugs in software engineering datasets. Practitioners and researchers can leverage these findings to develop better monitoring systems and data-cleaning methods to help detect and resolve data bugs in deep learning systems.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
URL: https://www.theguardian.com/technology/2024/ apr/26/tesla-autopilot-fatal-crash
The Guardian , April 2024. URL: https://www.theguardian.com/technology/2024/ apr/26/tesla-autopilot-fatal-crash
2024
-
[2]
Data preprocessing impact on machine learning algorithm performance
Alberto Amato and Vincenzo Di Lecce. Data preprocessing impact on machine learning algorithm performance. Open Computer Science , 13(1):20220278, 2023
2023
-
[3]
Deep speech 2: End-to-end speech recognition in english and mandarin
Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. Deep speech 2: End-to-end speech recognition in english and mandarin. In International conference on machine learning , pages 173–182. PMLR, 2016
2016
-
[4]
URL: https://issues.apache.org/jira/projects/HADOOP/issues
Apache. URL: https://issues.apache.org/jira/projects/HADOOP/issues
-
[5]
A hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering
Andrea Arcuri and Lionel Briand. A hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering. Software Testing, Verification and Reliability, 24(3):219–250, 2014
2014
-
[6]
Software engi- neering challenges of deep learning
Anders Arpteg, Bj¨ orn Brinne, Luka Crnkovic-Friis, and Jan Bosch. Software engi- neering challenges of deep learning. In 2018 44th euromicro conference on software engineering and advanced applications (SEAA) , pages 50–59. IEEE, 2018
2018
-
[7]
Sicong Cao, Xiaobing Sun, Ratnadira Widyasari, David Lo, Xiaoxue Wu, Lili Bo, Jiale Zhang, Bin Li, Wei Liu, Di Wu, et al. A systematic literature review on explain- ability for machine/deep learning-based software engineering research. arXiv preprint arXiv:2401.14617, 2024
arXiv 2024
-
[8]
A deep learning model for estimating story points
Morakot Choetkiertikul, Hoa Khanh Dam, Truyen Tran, Trang Pham, Aditya Ghose, and Tim Menzies. A deep learning model for estimating story points. IEEE Transac- tions on Software Engineering , 45(7):637–656, 2018
2018
Show all 106 references
-
[9]
Data cleaning and machine learning: a systematic literature review
Pierre-Olivier Cˆ ot´ e, Amin Nikanjam, Nafisa Ahmed, Dmytro Humeniuk, and Foutse Khomh. Data cleaning and machine learning: a systematic literature review. Auto- mated Software Engineering, 31(2):54, 2024
2024
-
[10]
Data quality for software vul- nerability datasets
Roland Croft, M Ali Babar, and M Mehdi Kholoosi. Data quality for software vul- nerability datasets. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 121–133. IEEE, 2023. 46 Mehil B. Shah et al
2023
-
[11]
Data preparation for soft- ware vulnerability prediction: A systematic literature review
Roland Croft, Yongzheng Xie, and Muhammad Ali Babar. Data preparation for soft- ware vulnerability prediction: A systematic literature review. IEEE Transactions on Software Engineering, 49(3):1044–1063, 2022
2022
-
[12]
Compiler fuzzing through deep learning
Chris Cummins, Pavlos Petoumenos, Alastair Murray, and Hugh Leather. Compiler fuzzing through deep learning. In Proceedings of the 27th ACM SIGSOFT interna- tional symposium on software testing and analysis , pages 95–105, 2018
2018
-
[13]
Vulnerability de- tection with code language models: How far are we? arXiv preprint arXiv:2403.18624, 2024
Yangruibo Ding, Yanjun Fu, Omniyyah Ibrahim, Chawin Sitawarin, Xinyun Chen, Basel Alomair, David Wagner, Baishakhi Ray, and Yizheng Chen. Vulnerability de- tection with code language models: How far are we? arXiv preprint arXiv:2403.18624, 2024
2024 arXiv
-
[14]
Ac/c++ code vulnerability dataset with code changes and cve summaries
Jiahao Fan, Yi Li, Shaohua Wang, and Tien N Nguyen. Ac/c++ code vulnerability dataset with code changes and cve summaries. In Proceedings of the 17th International Conference on Mining Software Repositories , pages 508–512, 2020
2020
-
[15]
The impact of changes mislabeled by szz on just-in-time defect pre- diction.(2019)
Yuanrui F AN, Xin XIA, Daniel A COSTA, David LO, Ahmed E HASSAN, and Shanping LI. The impact of changes mislabeled by szz on just-in-time defect pre- diction.(2019). IEEE Transactions on Software Engineering , pages 1–26, 2019
2019
-
[16]
A comprehensive introduction to label noise
Beno ˆ ıt Fr´ enay, Ata Kab´ an, et al. A comprehensive introduction to label noise. In ESANN. Citeseer, 2014
2014
-
[17]
Classification in the presence of label noise: a survey
Beno ˆ ıt Fr´ enay and Michel Verleysen. Classification in the presence of label noise: a survey. IEEE transactions on neural networks and learning systems , 25(5):845–869, 2013
2013
-
[18]
Gpt2sp: A transformer-based agile story point estimation approach
Michael Fu and Chakkrit Tantithamthavorn. Gpt2sp: A transformer-based agile story point estimation approach. IEEE Transactions on software engineering , 49(2):611– 625, 2022
2022
-
[19]
Linevul: A transformer-based line-level vulnerability prediction
Michael Fu and Chakkrit Tantithamthavorn. Linevul: A transformer-based line-level vulnerability prediction. In Proceedings of the 19th International Conference on Min- ing Software Repositories, pages 608–620, 2022
2022
-
[20]
Shuzheng Gao, Xin-Cheng Wen, Cuiyun Gao, Wenxuan Wang, Hongyu Zhang, and Michael R Lyu. What makes good in-context demonstrations for code intelligence tasks with llms? In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) , pages 761–773. IEEE, 2023
2023
-
[21]
Accessed on June 29, 2024
GeeksforGeeks, Jan 2024. Accessed on June 29, 2024. URL: https://www. geeksforgeeks.org/best-ides-for-machine-learning/
2024
-
[22]
On the use of deep learning in software defect prediction
G¨ orkem Giray, Kwabena Ebo Bennin,¨Omer K¨ oksal,¨Onder Babur, and Bedir Tekiner- dogan. On the use of deep learning in software defect prediction. Journal of Systems and Software, 195:111537, 2023
2023
-
[23]
Understanding the difficulty of training deep feed- forward neural networks
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feed- forward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages 249–256. JMLR Workshop and Conference Proceedings, 2010
2010
-
[24]
Deep learning, volume 196
Ian Goodfellow. Deep learning, volume 196. MIT press, 2016
2016
-
[25]
A survey of deep learning techniques for autonomous driving
Sorin Grigorescu, Bogdan Trasnea, Tiberiu Cocias, and Gigel Macesanu. A survey of deep learning techniques for autonomous driving. Journal of Field Robotics, 37(3):362– 386, 2020
2020
-
[26]
Code to com- ment” translation” data, metrics, baselining & evaluation
David Gros, Hariharan Sezhiyan, Prem Devanbu, and Zhou Yu. Code to com- ment” translation” data, metrics, baselining & evaluation. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, pages 746–757, 2020
2020
-
[27]
Chan, and Bo Jiang
Yuxiang Guo, Xiaopeng Gao, Zhenyu Zhang, W.K. Chan, and Bo Jiang. A study on the impact of pre-trained model on just-in-time defect prediction. In 2023 IEEE 23rd International Conference on Software Quality, Reliability, and Security (QRS) , pages 105–116, 2023. doi:10.1109/QR...
2023
-
[28]
Duplicate bug report de- tection using dual-channel convolutional neural networks
Jianjun He, Ling Xu, Meng Yan, Xin Xia, and Yan Lei. Duplicate bug report de- tection using dual-channel convolutional neural networks. In Proceedings of the 28th International Conference on Program Comprehension , pages 117–127, 2020
2020
-
[29]
Deepjit: an end-to-end deep learning framework for just-in-time defect prediction
Thong Hoang, Hoa Khanh Dam, Yasutaka Kamei, David Lo, and Naoyasu Ubayashi. Deepjit: an end-to-end deep learning framework for just-in-time defect prediction. In 2019 IEEE/ACM 16th International Conference on Mining Software Repositories Title Suppressed Due to Excessive Lengt...
2019
-
[30]
A deep dive into large language models for auto- mated bug localization and repair
Soneya Binta Hossain, Nan Jiang, Qiang Zhou, Xiaopeng Li, Wen-Hao Chiang, Yingjun Lyu, Hoan Nguyen, and Omer Tripp. A deep dive into large language models for auto- mated bug localization and repair. Proceedings of the ACM on Software Engineering , 1(FSE):1471–1493, 2024
2024
-
[31]
Taxonomy of real faults in deep learning systems
Nargiz Humbatova, Gunel Jahangirova, Gabriele Bavota, Vincenzo Riccio, Andrea Stocco, and Paolo Tonella. Taxonomy of real faults in deep learning systems. In Pro- ceedings of the ACM/IEEE 42nd International Conference on Software Engineering , page 1110–1121, Seoul South Korea...
2020
-
[32]
Deepcrime: mutation test- ing of deep learning systems based on real faults
Nargiz Humbatova, Gunel Jahangirova, and Paolo Tonella. Deepcrime: mutation test- ing of deep learning systems based on real faults. In Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis , pages 67–78, 2021
2021
-
[33]
Multi- modal api recommendation
Ivana Clairine Irsan, Ting Zhang, Ferdian Thung, Kisub Kim, and David Lo. Multi- modal api recommendation. In 2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) , pages 272–283. IEEE, 2023
2023
-
[34]
A compre- hensive study on deep learning bug characteristics
Md Johirul Islam, Giang Nguyen, Rangeet Pan, and Hridesh Rajan. A compre- hensive study on deep learning bug characteristics. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Sym- posium on the Foundations of Software Engineerin...
2019
-
[35]
A large-scale empirical study of just-in-time quality assurance
Yasutaka Kamei, Emad Shihab, Bram Adams, Ahmed E Hassan, Audris Mockus, Anand Sinha, and Naoyasu Ubayashi. A large-scale empirical study of just-in-time quality assurance. IEEE Transactions on Software Engineering , 39(6):757–773, 2012
2012
-
[36]
Apachejit: a large dataset for just-in- time defect prediction
Hossein Keshavarz and Meiyappan Nagappan. Apachejit: a large dataset for just-in- time defect prediction. In Proceedings of the 19th international conference on mining software repositories, pages 191–195, 2022
2022
-
[37]
Dealing with noise in defect prediction
Sunghun Kim, Hongyu Zhang, Rongxin Wu, and Liang Gong. Dealing with noise in defect prediction. In Proceedings of the 33rd International Conference on Software Engineering, pages 481–490, 2011
2011
-
[38]
Understanding black-box predictions via influence functions
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning , pages 1885–1894. PMLR, 2017
2017
-
[39]
Towards the generation of machine learning defect reports
Tuan Dung Lai. Towards the generation of machine learning defect reports. In 2021 36th IEEE/ACM International Conference on Automated Software Engineer- ing (ASE) , pages 1038–1042. IEEE, 2021
2021
-
[40]
Generating duplicate bug datasets
Alina Lazar, Sarah Ritchey, and Bonita Sharif. Generating duplicate bug datasets. In Proceedings of the 11th working conference on mining software repositories , pages 392–395, 2014
2014
-
[41]
Efficient backprop
Yann LeCun, L´ eon Bottou, Genevieve B Orr, and Klaus-Robert M¨ uller. Efficient backprop. In Neural networks: Tricks of the trade , pages 9–50. Springer, 2002
2002
-
[42]
Detecting outliers: Do not use standard deviation around the mean, use absolute de- viation around the median
Christophe Leys, Christophe Ley, Olivier Klein, Philippe Bernard, and Laurent Licata. Detecting outliers: Do not use standard deviation around the mean, use absolute de- viation around the median. Journal of experimental social psychology , 49(4):764–766, 2013
2013
-
[43]
Neurrev: train better sparse neural network practically via neuron revitalization
Gen Li, Lu Yin, Jie Ji, Wei Niu, Minghai Qin, Bin Ren, Linke Guo, Shiwei Liu, and Xiaolong Ma. Neurrev: train better sparse neural network practically via neuron revitalization. In 12th International Conference on Learning Representations, ICLR 2024, 2024
2024
-
[44]
Cclearner: A deep learning-based clone detection approach
Liuqing Li, He Feng, Wenjie Zhuang, Na Meng, and Barbara Ryder. Cclearner: A deep learning-based clone detection approach. In 2017 IEEE international conference on software maintenance and evolution (ICSME) , pages 249–260. IEEE, 2017
2017
-
[45]
Ro- bust learning of deep predictive models from noisy and imbalanced software engineering datasets
Zhong Li, Minxue Pan, Yu Pei, Tian Zhang, Linzhang Wang, and Xuandong Li. Ro- bust learning of deep predictive models from noisy and imbalanced software engineering datasets. In Proceedings of the 37th IEEE/ACM International Conference on Auto- mated Software Engineering, page...
2022
-
[46]
Advances, challenges and opportunities in creating data for trustwor- thy ai
Weixin Liang, Girmaw Abebe Tadesse, Daniel Ho, Li Fei-Fei, Matei Zaharia, Ce Zhang, and James Zou. Advances, challenges and opportunities in creating data for trustwor- thy ai. Nature Machine Intelligence , 4(8):669–677, 2022
2022
-
[47]
Deep learning based code smell detection
Hui Liu, Jiahao Jin, Zhifeng Xu, Yanzhen Zou, Yifan Bu, and Lu Zhang. Deep learning based code smell detection. IEEE transactions on Software Engineering , 47(9):1811– 1837, 2019
2019
-
[48]
Understand- ing the difficulty of training transformers
Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. Understand- ing the difficulty of training transformers. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing ...
2020 doi
-
[49]
A comparative study on the effect of data imbalance on software defect prediction
Yanbin Liu, Wen Zhang, Guangjie Qin, and Jiangpeng Zhao. A comparative study on the effect of data imbalance on software defect prediction. Procedia Computer Science, 214:1603–1616, 2022
2022
-
[50]
On inter-dataset code duplication and data leakage in large language models
Jos´ e Antonio Hern´ andez L´ opez, Boqi Chen, Tushar Sharma, and D´ aniel Varr´ o. On inter-dataset code duplication and data leakage in large language models. arXiv preprint arXiv:2401.07930, 2024
2024 arXiv
-
[51]
Robust and rapid adaption for concept drift in software system anomaly detection
Minghua Ma, Shenglin Zhang, Dan Pei, Xin Huang, and Hongwei Dai. Robust and rapid adaption for concept drift in software system anomaly detection. In 2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE) , pages 13–24. IEEE, 2018
2018
-
[52]
Unveiling code pre-trained models: Investigating syntax and semantics capacities
Wei Ma, Shangqing Liu, Mengjie Zhao, Xiaofei Xie, Wenhang Wang, Qiang Hu, Jie Zhang, and Yang Liu. Unveiling code pre-trained models: Investigating syntax and semantics capacities. ACM Transactions on Software Engineering and Methodology , 33(7):1–29, 2024
2024
-
[53]
Defectors: A large, diverse python dataset for defect prediction
Parvez Mahbub, Ohiduzzaman Shuvo, and Mohammad Masudur Rahman. Defectors: A large, diverse python dataset for defect prediction. In 2023 IEEE/ACM 20th Inter- national Conference on Mining Software Repositories (MSR) , pages 393–397. IEEE, 2023
2023
-
[54]
Zaheed Mahmood, David Bowes, Peter CR Lane, and Tracy Hall. What is the impact of imbalance on software defect prediction performance? In Proceedings of the 11th international conference on predictive models and data analytics in software engineer- ing, pages 1–4, 2015
2015
-
[55]
An extensive study on cross- project predictive mutation testing
Dongyu Mao, Lingchao Chen, and Lingming Zhang. An extensive study on cross- project predictive mutation testing. In 2019 12th IEEE Conference on Software Test- ing, Validation and Verification (ICST) , pages 160–171. IEEE, 2019
2019
-
[56]
Are fix-inducing changes a moving target? a longitudinal case study of just-in-time defect prediction
Shane McIntosh and Yasutaka Kamei. Are fix-inducing changes a moving target? a longitudinal case study of just-in-time defect prediction. In Proceedings of the 40th international conference on software engineering , pages 560–560, 2018
2018
-
[57]
Duplicate bug report detection using an attention-based neural language model
Montassar Ben Messaoud, Asma Miladi, Ilyes Jenhani, Mohamed Wiem Mkaouer, and Lobna Ghadhab. Duplicate bug report detection using an attention-based neural language model. IEEE Transactions on Reliability , 2022
2022
-
[58]
Duplicate bug report detection using an attention-based neural language model
Montassar Ben Messaoud, Asma Miladi, Ilyes Jenhani, Mohamed Wiem Mkaouer, and Lobna Ghadhab. Duplicate bug report detection using an attention-based neural language model. IEEE Transactions on Reliability, 72(2):846–858, 2023. doi:10.1109/ TR.2022.3193645
2023
-
[59]
Mondal, M
S. Mondal, M. Rahman, and C. K. Roy. Can we identify stack overflow questions requir- ing code snippets? investigating the cause & effect of missing code snippets. In 2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), pages 764–775, L...
2024
-
[60]
Developer-intent driven code comment generation
Fangwen Mu, Xiao Chen, Lin Shi, Song Wang, and Qing Wang. Developer-intent driven code comment generation. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) , pages 768–780. IEEE, 2023
2023
-
[61]
The impact of nondeterminism on reproducibility in deep reinforcement learning
Prabhat Nagarajan, Garrett Warnell, and Peter Stone. The impact of nondeterminism on reproducibility in deep reinforcement learning. 2018. Title Suppressed Due to Excessive Length 49
2018
-
[62]
A modern take on the bias-variance tradeoff in neural networks
Brady Neal, Sarthak Mittal, Aristide Baratin, Vinayak Tantia, Matthew Scicluna, Simon Lacoste-Julien, and Ioannis Mitliagkas. A modern take on the bias-variance tradeoff in neural networks. arXiv preprint arXiv:1810.08591 , 2018
-
[63]
Understanding and tackling label errors in deep learning-based vulnerability detec- tion (experience paper)
Xu Nie, Ningke Li, Kailong Wang, Shangguang Wang, Xiapu Luo, and Haoyu Wang. Understanding and tackling label errors in deep learning-based vulnerability detec- tion (experience paper). In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Anal...
2023
-
[64]
Confident learning: Estimating un- certainty in dataset labels
Curtis Northcutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating un- certainty in dataset labels. Journal of Artificial Intelligence Research , 70:1373–1411, 2021
2021
-
[65]
The impact of concept drift and data leakage on log level prediction models
Youssef Esseddiq Ouatiti, Mohammed Sayagh, Noureddine Kerzazi, Bram Adams, and Ahmed E Hassan. The impact of concept drift and data leakage on log level prediction models. Empirical Software Engineering, 29(5):1–37, 2024
2024
-
[66]
On the difficulty of training recurrent neural networks
R Pascanu. On the difficulty of training recurrent neural networks. arXiv preprint arXiv:1211.5063, 2013
2013 arXiv
-
[67]
Effective use of the mcnemar test
Matilda QR Pembury Smith and Graeme D Ruxton. Effective use of the mcnemar test. Behavioral Ecology and Sociobiology, 74:1–9, 2020
2020
-
[68]
Deeplinedp: Towards a deep learning approach for line-level defect prediction
Chanathip Pornprasit and Chakkrit Kla Tantithamthavorn. Deeplinedp: Towards a deep learning approach for line-level defect prediction. IEEE Transactions on Software Engineering, 49(1):84–98, 2022
2022
-
[69]
Guidelines for conducting and reporting case study research in software engineering
Per Runeson and Martin H¨ ost. Guidelines for conducting and reporting case study research in software engineering. Empirical software engineering, 14:131–164, 2009
2009
-
[70]
everyone wants to do the model work, not the data work
Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Pari- tosh, and Lora M Aroyo. “everyone wants to do the model work, not the data work”: Data cascades in high-stakes ai. In proceedings of the 2021 CHI Conference on Human Factors in Computing Systems ,...
2021
-
[71]
On challenges in machine learning model management
Sebastian Schelter, Felix Biessmann, Tim Januschowski, David Salinas, Stephan Seufert, and Gyuri Szarvas. On challenges in machine learning model management. 2015
2015
-
[72]
Grad-cam: Visual explanations from deep networks via gradient-based localization
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision , pages 618–626, 2017
2017
-
[73]
Towards understanding the impact of data bugs on deep learning models in software engineering, 2025
Mehil Shah. Towards understanding the impact of data bugs on deep learning models in software engineering, 2025. URL: https://zenodo.org/records/15270651
2025
-
[74]
Towards enhancing the reproducibility of deep learning bugs: an empirical study
Mehil B Shah, Mohammad Masudur Rahman, and Foutse Khomh. Towards enhancing the reproducibility of deep learning bugs: an empirical study. Empirical Software Engineering, 30(1):23, 2025
2025
-
[75]
Are we building on the rock? on the importance of data preprocessing for code summarization
Lin Shi, Fangwen Mu, Xiao Chen, Song Wang, Junjie Wang, Ye Yang, Ge Li, Xin Xia, and Qing Wang. Are we building on the rock? on the importance of data preprocessing for code summarization. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Sympos...
2022
-
[76]
A comprehensive investigation of the role of imbalanced learning for software defect prediction
Qinbao Song, Yuchen Guo, and Martin Shepperd. A comprehensive investigation of the role of imbalanced learning for software defect prediction. IEEE Transactions on Software Engineering, 45(12):1253–1269, 2018
2018
-
[77]
Richard Stevens, Bill Fenner, and Andrew M
W. Richard Stevens, Bill Fenner, and Andrew M. Rudoff. UNIX network programming. 1: The sockets networking API / W. Richard Stevens; Bill Fenner; Andrew M. Rudoff . Addison-Wesley professional computing series. Addison-Wesley, Boston, 3rd ed edition, 2013
2013
-
[78]
Unstable gradients in deep neural nets
Ludvig Storm. Unstable gradients in deep neural nets. 2020
2020
-
[79]
Explaining odds ratios
Magdalena Szumilas. Explaining odds ratios. Journal of the Canadian academy of child and adolescent psychiatry , 19(3):227, 2010
2010
-
[80]
The impact of mislabelling on the performance and interpretation of defect prediction models
Chakkrit Tantithamthavorn, Shane McIntosh, Ahmed E Hassan, Akinori Ihara, and Kenichi Matsumoto. The impact of mislabelling on the performance and interpretation of defect prediction models. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering , volume 1...
2015
-
[81]
Evaluating representation learning of code changes for predicting patch correctness in program repair
Haoye Tian, Kui Liu, Abdoul Kader Kabor´ e, Anil Koyuncu, Li Li, Jacques Klein, and Tegawend´ e F Bissyand´ e. Evaluating representation learning of code changes for predicting patch correctness in program repair. In Proceedings of the 35th IEEE/ACM International Conference on...
2020
-
[82]
Using pre-trained models to boost code review automa- tion
Rosalia Tufano, Simone Masiero, Antonio Mastropaolo, Luca Pascarella, Denys Poshy- vanyk, and Gabriele Bavota. Using pre-trained models to boost code review automa- tion. In Proceedings of the 44th international conference on software engineering , pages 2291–2302, 2022
2022
-
[83]
Exploratory data analysis
John W Tukey. Exploratory data analysis. Reading/Addison-Wesley, 1977
1977
-
[84]
Visualizing data using t-sne
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008
2008
-
[85]
Attention is all you need
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017
2017
-
[86]
Optimizing datasets for code summarization: Is code-comment coher- ence enough?, 2025
Antonio Vitale, Antonio Mastropaolo, Rocco Oliveto, Massimiliano Di Penta, and Si- mone Scalabrino. Optimizing datasets for code summarization: Is code-comment coher- ence enough?, 2025. URL: https://arxiv.org/abs/2502.07611, arXiv:2502.07611
2025 arXiv
-
[87]
Self-driving uber car kills pedestrian in arizona, where robots roam, Mar 2018
Daisuke Wakabayashi. Self-driving uber car kills pedestrian in arizona, where robots roam, Mar 2018. URL: https://www.nytimes.com/2018/03/19/technology/ uber-driverless-fatality.html
2018
-
[88]
An empirical study on noisy label learning for program understanding
Wenhan Wang, Yanzhou Li, Anran Li, Jian Zhang, Wei Ma, and Yang Liu. An empirical study on noisy label learning for program understanding. In Proceed- ings of the IEEE/ACM 46th International Conference on Software Engineering , ICSE ’24, New York, NY, USA, 2024. Association fo...
2024
-
[89]
Detecting and explaining self-admitted technical debts with attention-based neural networks
Xin Wang, Jin Liu, Li Li, Xiao Chen, Xiao Liu, and Hao Wu. Detecting and explaining self-admitted technical debts with attention-based neural networks. In Proceedings of the 35th IEEE/ACM international conference on automated software engineering , pages 871–882, 2020
2020
-
[90]
Weights and biases: An ai developer platform
Weights and Biases. Weights and biases: An ai developer platform. https://wandb.ai/site, 2021. [Online; accessed 21-February-2024]
2021
-
[91]
Kurtosis as peakedness, 1905–2014
Peter H Westfall. Kurtosis as peakedness, 1905–2014. rip. The American Statistician , 68(3):191–195, 2014
1905
-
[92]
Data collection and quality challenges for deep learning
Steven Euijong Whang and Jae-Gil Lee. Data collection and quality challenges for deep learning. Proceedings of the VLDB Endowment , 13(12):3429–3432, 2020
2020
-
[93]
Expressivity of neural networks with random weights and learned biases
Ezekiel Williams, Avery Hee-Woon Ryoo, Thomas Jiralerspong, Alexandre Payeur, Matthew G Perich, Luca Mazzucato, and Guillaume Lajoie. Expressivity of neural networks with random weights and learned biases. ICLR 2025 , 2025
2025
-
[94]
Data quality matters: A case study on data label correctness for security bug report prediction
Xiaoxue Wu, Wei Zheng, Xin Xia, and David Lo. Data quality matters: A case study on data label correctness for security bug report prediction. IEEE Transactions on Software Engineering, 48(7):2541–2556, 2021
2021
-
[95]
Data quality matters: A case study of obsolete comment detection
Shengbin Xu, Yuan Yao, Feng Xu, Tianxiao Gu, Jingwei Xu, and Xiaoxing Ma. Data quality matters: A case study of obsolete comment detection. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 781–793. IEEE, 2023
2023
-
[96]
Code comment inconsistency detection based on confidence learning
Zhengkang Xu, Shikai Guo, Yumiao Wang, Rong Chen, Hui Li, Xiaochen Li, and He Jiang. Code comment inconsistency detection based on confidence learning. IEEE Transactions on Software Engineering , 2024
2024
-
[97]
A survey on deep learning for software engineering
Yanming Yang, Xin Xia, David Lo, and John Grundy. A survey on deep learning for software engineering. ACM Comput. Surv. , 54(10s), sep 2022. doi:10.1145/3505243
2022 doi
-
[98]
Mining software defects: Should we consider affected releases? In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) , pages 654–665, 2019
Suraj Yatish, Jirayus Jiarpakdee, Patanamon Thongtanunam, and Chakkrit Tan- tithamthavorn. Mining software defects: Should we consider affected releases? In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) , pages 654–665, 2019. doi:10.1109/ICSE.2019.00075
2019
-
[99]
On the importance of consistency in training deep neural networks
Chengxi Ye, Yezhou Yang, Cornelia Fermuller, and Yiannis Aloimonos. On the importance of consistency in training deep neural networks. arXiv preprint arXiv:1708.00631, 2017
2017 arXiv
-
[100]
Case study research: Design and methods , volume 5
Robert K Yin. Case study research: Design and methods , volume 5. sage, 2009. Title Suppressed Due to Excessive Length 51
2009
-
[101]
Dynamic data fault localization for deep neural networks
Yining Yin, Yang Feng, Shihao Weng, Zixi Liu, Yuan Yao, Yichi Zhang, Zhihong Zhao, and Zhenyu Chen. Dynamic data fault localization for deep neural networks. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Softw...
2023
-
[102]
Zhengran Zeng, Yuqun Zhang, Haotian Zhang, and Lingming Zhang. Deep just- in-time defect prediction: how far are we? In Proceedings of the 30th ACM SIG- SOFT International Symposium on Software Testing and Analysis , ISSTA 2021, page 427–438, New York, NY, USA, 2021. Associati...
2021
-
[103]
Learning to handle exceptions
Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, Yanjun Pu, and Xudong Liu. Learning to handle exceptions. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering , pages 29–41, 2020
2020
-
[104]
Duplicate bug report detection: How far are we? ACM Transactions on Software Engineering and Methodology, 32(4):1–32, 2023
Ting Zhang, DongGyun Han, Venkatesh Vinayakarao, Ivana Clairine Irsan, Bowen Xu, Ferdian Thung, David Lo, and Lingxiao Jiang. Duplicate bug report detection: How far are we? ACM Transactions on Software Engineering and Methodology, 32(4):1–32, 2023
2023
-
[105]
When are bias-free relu networks like linear networks? arXiv preprint arXiv:2406.12615 , 2024
Yedi Zhang, Andrew Saxe, and Peter E Latham. When are bias-free relu networks like linear networks? arXiv preprint arXiv:2406.12615 , 2024
2024 arXiv
-
[106]
Devign: Ef- fective vulnerability identification by learning comprehensive program semantics via graph neural networks
Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. Devign: Ef- fective vulnerability identification by learning comprehensive program semantics via graph neural networks. Advances in neural information processing systems , 32, 2019. 52 Mehil B. Shah et al. App...
2019
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.