Pith. sign in

REVIEW 3 major objections 6 minor 53 references

Today's Cat Is Tomorrow's Dog: Accounting for Time-Based Changes in the Labels of ML Vulnerability Detection Approaches

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Machine-learning vulnerability detectors evaluated on standard retrospective datasets overstate how they will perform in the field, because labels change as vulnerabilities are discovered over calendar time.

desk verdict The R-P protocol is a genuine and useful contribution, but the paper's central 'no learning' claim is undercut by a test set that excludes late-discovered vulnerabilities. read the letter →

arxiv 2506.11939 v1 pith:HIDMQF7B submitted 2025-06-13 cs.CR cs.AI

Ranindya Paramitha , Yuan Feng , Fabio Massacci This is my paper · ORCID
classification cs.CRcs.AI
keywords vulnerabilitydetectionmachinelearningevaluationretrospectiveinformationperspectivetestingtime-baseddatasetslabeldriftovertimeMann-Kendalltrendtestsoftwaresecurity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vulnerability datasets used to train and test machine-learning detectors carry retrospective information: they label code fragments with vulnerabilities that were discovered only later. The paper argues that this hindsight makes reported performance higher than what a deployed model would achieve, and proposes a methodology that rebuilds a dataset year by year so that both training and test labels reflect only what was known at the time. Validated on four CVE-based datasets and five models, the methodology shows that performance on next-year perspective tests often falls short of retrospective tests, and that precision and recall do not rise consistently as more historical data accumulates. The paper concludes that most detectors are not learning in a field-deployable sense and that evaluations should present time-annotated trends rather than single retrospective numbers.

What carries the argument

The mechanism is a relabeling algorithm that takes a retrospective dataset enriched with CVE publication dates and, for each year $t$, outputs one training set of all records known by $t$ and two test sets: R-R tests on data and labels available at $t$, while R-P keeps only records whose label dates fall in $(t, t+\delta]$ with $\delta=12$ months, simulating next-year field use. The Mann-Kendall nonparametric trend test applied to the resulting yearly precision and recall series is what turns the visual 'no trend' observation into a statistical claim.

What would settle it

Train each of the five models on the same four datasets while recording performance on a fixed validation set whose true labels do not change over calendar time. If validation performance improves over training years but R-P recall stays flat, the flat trend is caused by changing label distributions rather than absence of learning; if validation performance is also flat, the 'not learning' conclusion is supported.

Watch

Extended reading notes

Core claim

The paper's central claim is that a label in a vulnerability dataset is not a fixed property of the code: it flips when a CVE is published, so 'today's cat is tomorrow's dog.' Concretely, the paper establishes that (i) if evaluation ignores this, up to one-third of the vulnerabilities seen in a training period were misclassified at the time and up to 81.5% of the next-year test data was never seen, and (ii) when models are tested on the next year's data with next-year labels (the R-P scenario), none of the five models shows a significant increasing trend in precision, and only one model-dataset combination shows a significant increasing trend in recall, namely Vuldeepecker on its own NVD dataset. This is the evidence behind the conclusion that most detectors are not learning.

Load-bearing premise

The load-bearing premise is that a model that is genuinely learning should show steadily improving perspective-test performance as more historical data accumulates, so the absence of such a trend is read as evidence that the model is not learning.

Editorial extensions

If this is right

  • Reported retrospective performance is an upper bound: several models show statistically significant recall drops when tested on next-year perspective data, so published numbers likely exceed field performance.
  • Evaluation should present performance as a time-indexed series with a trend test, not as a single aggregate score.
  • Vulnerability datasets should carry discovery dates rather than only final labels, because complete-information labels cannot reproduce the conditions a deployed model actually faces.
  • Adding more historical training data does not by itself produce better next-year detection: the Mann-Kendall tests find no increasing precision trend for any of the five models.
  • The methodology extends to other time-stamped prediction tasks such as bug and defect detection and commit classification, as the paper argues.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the flat or fluctuating R-P trend could stem from label distribution shift rather than from models failing to learn; the paper does not measure label stability, so its 'not learning' reading is one plausible explanation, not a proven one.
  • The paper's exclusion of large language models points to a harder future test: an LLM cannot be untrained on the past, so a proper perspective evaluation would need a sequence of checkpoints pretrained at different dates, something current practice does not provide.
  • If the trend result holds, it reframes 'bigger dataset improves detector' findings in the literature: gains attributed to data volume may actually be gains in retrospective label information.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a methodology for restructuring a time-stamped vulnerability dataset into a series of ``perspective'' datasets in which training and test labels reflect only the information available at each calendar time point. The authors instantiate the methodology on four CVE-based datasets (three BigVul projects and the Vuldeepecker NVD dataset) and five ML models (Code2Vec, CodeBERT, LineVul, ReGVD, Vuldeepecker), comparing Retrospective Training--Retrospective Testing (R-R) with Retrospective Training--Perspective Testing (R-P). They report that R-P performance is often lower than R-R performance and that Mann-Kendall trend tests show no significant increasing trend in R-P performance as retrospective information accumulates, leading to the conclusion that ``most models are not learning.'' The paper also includes an additional experiment on ``seen but believed negatives'' and discusses threats to validity, including the use of labeling dates as a proxy for availability dates.

Significance. If the findings stand, the paper makes a useful methodological contribution: it gives the ML4VD community a concrete, reproducible procedure for avoiding retrospective label leakage in evaluation, and it proposes presenting model performance as a statistically tested temporal trend, which is an improvement over single-number evaluations. The replication packages on Zenodo are a strength, as is the use of external statistical benchmarks (Mann-Kendall, Wilcoxon, Vargha--Delaney effect sizes) that are not fitted to the target result. The observed gap between R-R and R-P performance is an important empirical caution. However, the paper's headline conclusion that most models are not learning is not supported by the evidence as presented, and the R-P test construction excludes the very late-labeling phenomenon that motivates the paper. These issues materially weaken the central RQ2 claim and require revision.

major comments (3)
  1. [§7.2, Table 8, and abstract] The inference from the absence of a significant Mann-Kendall trend to ``most models are not learning'' is not supported. The premise stated in the abstract (``If the model is actually learning, it should improve its performance over time as more data becomes available and data becomes more stable'') is an unvalidated assumption; the paper does not measure label stability, and the R-P test composition changes substantially from year to year (Table 6: never-seen negatives range from 52.8% to 95.8% and perspective-test positives from about 0.3% to 52.8% across projects). Under distribution shift, a model can learn while its field performance remains flat or fluctuates. The conclusion should be weakened to ``no consistent trend in R-P performance was observed'' or supported by a sensitivity analysis that controls for test-set difficulty and label stability.
  2. [Algorithm 1, lines 5-9; §6; §9.2] The R-P test set excludes records whose label_date exceeds timeline + delta, so the late-labeled fragments that motivate the paper (for example, the CVE-2021-4104 log4j scenario in §10) never appear in R-P testing. Because availability dates are proxied by labeling dates, a fragment committed in 2010 but assigned a CVE in 2021 is treated as nonexistent before 2021; the R-P test is thus confined to records newly labeled or newly appearing within the next delta window and does not measure the ``today's cat is tomorrow's dog'' label-change phenomenon. Section 8's ``seen but believed negatives'' experiment adds these cases only to the retrospective test set, not to R-P. The paper should either extend R-P to include late-labeled records (using availability dates for inclusion and label dates for labeling) or explicitly state that the R-P trend result does not address the motivating late-discovery scenario.
  3. [Table 8] The Bonferroni correction is applied only across the four datasets (p < 0.0125), but the paper tests 20 model-dataset combinations; with a correction for 20 tests (p < 0.0025), the single reported significant increasing recall trend (Vuldeepecker, p = 0.01) is no longer significant, which changes the ``only one significant increasing trend'' statement. Additionally, the Poppler rows report MK = 0 and p = 1.00 for every model, which is uninformative because the Poppler Perspective test sets are empty or tiny in most years (Table 5); these series should not be presented as evidence of no trend. The paper should report the number of non-missing observations per series and apply a consistent multiple-testing correction across all tests performed.
minor comments (6)
  1. [Table 7] The ``St.Dev.'' entry for OpenSSL Precision shows -25.4%, a negative standard deviation; this appears to be a typo for +25.4% or a formatting error. The caption ``difference in the precision and recall statistic'' should also be made plural and clearer.
  2. [Table 5] The row header ``Validating'' for the Poppler dataset should read ``Validation'' for consistency with the other datasets. Also, the Linux training rows begin in 2008 while other rows begin in 2009 or later; please verify the year alignment across rows.
  3. [§4, step (2)] The term ``testing delta'' is used before it is defined in the input description; define ``testing delta'' earlier in the methodology section, e.g., in Table 4 or in the step that introduces it.
  4. [Figure 5] Figure 5 is referenced in §7.2 but the figure itself is not visible in the provided text; if the final version will include it, please ensure it clearly distinguishes R-P and R-R series and include confidence intervals or per-point counts to help readers judge the stability of the plotted metrics.
  5. [§9.5] The statement ``We did not investigate further the reasons why there is no trend and why the results fluctuate over the years'' is an honest limitation, but it should be reflected in the abstract's wording rather than only in the threats section, since the abstract currently presents ``most models are not learning'' as a definite finding.
  6. [References] Reference [Johnb110 2022] points to a GitHub repository rather than a peer-reviewed source; consider citing the original Vuldeepecker implementation and noting the repository as a secondary resource. Also, the Zenodo references in §12 include placeholder text (``Zenodo link'') and should be completed with the direct DOI links.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: central findings are empirical trends on external datasets; self-citations are background, and flagged threats (label-date proxy, learning premise) are validity issues, not derivation-by-construction.

full rationale

This paper's claimed derivation chain is an empirical validation, not a formal derivation: Algorithm 1 defines R-R and R-P test sets by calendar dates and label dates; five published models are then trained and tested on external datasets (BigVul projects, NVD/Vuldeepecker), and the 'no consistent trend' finding is a Mann-Kendall statistic that could have come out either way. No parameter is fitted to the target result, no equation maps the construction to the finding, and the R-P construction (records with label_date in (t, t+delta]) does not by itself force a flat or fluctuating trend. Self-citations (Massacci et al. 2011; Nguyen et al. 2016; Marjanov et al. 2022; and the four Zenodo replication packages) are background motivation or artifacts, not load-bearing premises. The manuscript itself flags the two substantive caveats: Sec. 6/9.2 acknowledges that 'we use labeling dates as a proxy for availability dates' (so R-P test records are by construction future-dated under that proxy, which threatens the field-deployment interpretation of Finding #2), and Sec. 9.5 states 'We did not investigate further the reasons why there is no trend and why the results fluctuate over the years.' The abstract's premise that genuine learning should monotonically improve ('If the model is actually learning, it should improve its performance over time...') is asserted rather than established and makes the 'most models are not learning' conclusion logically stronger than the Mann-Kendall result supports. These are correctness/validity threats to the interpretation, not circularity: the reported trend is not equivalent, by construction or by fitted parameter, to its inputs. Hence score 1.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central empirical finding rests on two hand-set methodological parameters (testing delta, split ratio), a proxy for code availability (labeling date), and an interpretive assumption that learning should produce monotonic improvement. No new physical or conceptual entities are postulated.

free parameters (3)
  • testing_delta = 12 months
    Chosen by the authors to define the perspective testing window (train on year t, test on year t+1). The trend results depend on this choice; the paper notes shorter deltas would oscillate.
  • train_validation_test_split = 80/10/10
    Default split percentage used to build training, validation, and retrospective test sets at each time point. Chosen by hand, not tuned to the trend result.
  • timeline_dates = calendar years, dataset-dependent
    Observation points are chosen as calendar years, and the Mann-Kendall trend results are conditional on these windows. Different timelines could change the trend.
assumptions (3)
  • domain assumption Code availability date is assumed to be the same as the labeling (CVE publication) date because the original datasets do not record commit availability.
    Stated in Section 6 and Section 9.2; this is a proxy and is acknowledged as a threat to validity.
  • ad hoc to paper If a model is genuinely learning, its perspective-test performance should improve monotonically as retrospective data accumulates and labels stabilize.
    Abstract; this is the interpretive bridge from no Mann-Kendall trend to models are not learning, and it is not independently established.
  • domain assumption Calendar time, rather than release time, is the correct causal timeline for simulating field deployment.
    Footnotes 1 and 2 argue releases and vulnerability discovery are independent, so release-based splits can conflate causes. This motivates the methodology.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Today's Cat Is Tomorrow's Dog: Accounting for Time-Based Changes in the Labels of ML Vulnerability Detection Approaches." pith.science (2026). https://pith.science/paper/HIDMQF7B

@misc{pith2026250611939,
  author       = {Pith},
  title        = {Pith review of: Today's Cat Is Tomorrow's Dog: Accounting for Time-Based Changes in the Labels of ML Vulnerability Detection Approaches},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HIDMQF7B}},
  note         = {Machine review of arXiv:2506.11939}
}
read the original abstract

Vulnerability datasets used for ML testing implicitly contain retrospective information. When tested on the field, one can only use the labels available at the time of training and testing (e.g. seen and assumed negatives). As vulnerabilities are discovered across calendar time, labels change and past performance is not necessarily aligned with future performance. Past works only considered the slices of the whole history (e.g. DiverseVUl) or individual differences between releases (e.g. Jimenez et al. ESEC/FSE 2019). Such approaches are either too optimistic in training (e.g. the whole history) or too conservative (e.g. consecutive releases). We propose a method to restructure a dataset into a series of datasets in which both training and testing labels change to account for the knowledge available at the time. If the model is actually learning, it should improve its performance over time as more data becomes available and data becomes more stable, an effect that can be checked with the Mann-Kendall test. We validate our methodology for vulnerability detection with 4 time-based datasets (3 projects from BigVul dataset + Vuldeepecker's NVD) and 5 ML models (Code2Vec, CodeBERT, LineVul, ReGVD, and Vuldeepecker). In contrast to the intuitive expectation (more retrospective information, better performance), the trend results show that performance changes inconsistently across the years, showing that most models are not learning.

Figures

Figures reproduced from arXiv: 2506.11939 by the authors.

Figure 1
Figure 1. Motivating example on the benefit of perspective: when you see what you should not see. [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Our proposed methodology to eliminate retrospectives. [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Distribution of CVE count (#CVE) in BigVul Dataset [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Example instantiation of the proposed methodology. [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Evolution of Precision and Recall of 5 ML models with more and more retrospectives. [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 18 canonical work pages

  1. [4]

    InProceedings of the 17th International Conference on Predictive Models and Data Analytics in Software Engineering

    CVEfixes: automated collection of vulnerabilities and their fixes from open-source software. InProceedings of the 17th International Conference on Predictive Models and Data Analytics in Software Engineering. 30–39. doi:10.1145/3475960.3475985 Saikat Chakraborty, Rahul Krishna, Yangruibo Ding, and Baishakhi Ray

    arXiv

  2. [5]

    doi:10.1109/TSE.2021.3087402 Haipeng Chen, Rui Liu, Noseong Park, and VS Subrahmanian

    Deep learning based vulnerability detection: Are we there yet.IEEE Transactions on Software Engineering(2021). doi:10.1109/TSE.2021.3087402 Haipeng Chen, Rui Liu, Noseong Park, and VS Subrahmanian

    arXiv 2021

  3. [6]

    InProceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data Mining

    Using twitter to predict when vulnerabilities will be exploited. InProceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data Mining. 3143–3152. doi:10.1145/3292500.3330742 Yizheng Chen, Zhoujie Ding, Lamya Alowain, Xinyun Chen, and David Wagner

    arXiv

  4. [10]

    In2021 36th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW)

    JavaBERT: Training a Transformer-Based Model for the Java Programming Language. In2021 36th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW). 90–95. doi:10.1109/ASEW52652.2021.00028 Geanderson E dos Santos and Eduardo Figueiredo

    arXiv 2021

  5. [13]

    In Proceedings of the 19th International Conference on Mining Software Repositories

    Linevul: A transformer-based line-level vulnerability prediction. In Proceedings of the 19th International Conference on Mining Software Repositories. 608–620. doi:10.1145/3524842.3528452 Lobna Ghadhab, Ilyes Jenhani, Mohamed Wiem Mkaouer, and Montassar Ben Messaoud

    arXiv

  6. [14]

    doi:10.1016/j.infsof.2021.106566 Hazim Hanif and Sergio Maffeis

    Augmenting commit classification by using fine-grained source code changes and a pre-trained deep neural language model.Information and Software Technology135 (2021), 106566. doi:10.1016/j.infsof.2021.106566 Hazim Hanif and Sergio Maffeis

    arXiv 2021

  7. [15]

    In2022 International joint conference on neural networks (IJCNN)

    Vulberta: Simplified source code pre-training for vulnerability detection. In2022 International joint conference on neural networks (IJCNN). IEEE, 1–8. doi:10.1109/IJCNN55064.2022.9892280 Jinchang Hu, Lyuye Zhang, Chengwei Liu, Sen Yang, Song Huang, and Yang Liu

    arXiv 2022

  8. [19]

    InProceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

    The importance of accounting for real-world labelling when predicting software vulnerabilities. InProceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 695–705. doi:10.1145/3338906.3338941 HyunChul Joh and Yashwant K Malaiya

    arXiv 2019

Show all 53 references
  1. [22]

    In2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE)

    Deepcva: Automated commit-level vulnerability assessment with deep multi-task learning. In2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 717–729. doi:10.1109/ASE51524.2021.9678622 Stanislav Levin and Amiram Yehudai

    2021

  2. [24]

    InProceedings of the IEEE/ACM 46th International Conference on Software Engineering(Lisbon, Portugal)(ICSE ’24)

    On the Effectiveness of Function-Level Vulnerability Detectors for Inter-Procedural Vulnerabilities. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering(Lisbon, Portugal)(ICSE ’24). Association for Computing Machinery, New York, NY, USA, Article...

  3. [25]

    doi:10.1109/ACCESS.2019.2930578 Zhen Li, Deqing Zou, Shouhuai Xu, Hai Jin, Yawei Zhu, and Zhaoxuan Chen

    A comparative study of deep learning-based vulnerability detection system.IEEE Access7 (2019), 103184–103197. doi:10.1109/ACCESS.2019.2930578 Zhen Li, Deqing Zou, Shouhuai Xu, Hai Jin, Yawei Zhu, and Zhaoxuan Chen. 2022a. SySeVR: A Framework for Using Deep Learning to Detect S...

    2019

  4. [26]

    InProceedings 2018 Network and Distributed System Security Symposium

    VulDeePecker: A Deep Learning-Based System for Vulnerability Detection. InProceedings 2018 Network and Distributed System Security Symposium. doi:10.14722/ndss.2018.23158 arXiv:1801.01681 [cs] Zhen Li, Deqing Zou, Shouhuai Xu, Xinyu Ou, Hai Jin, Sujuan Wang, Zhijun Deng, and Y...

    2018

  5. [27]

    doi:10.48550/arXiv.1907.11692 arXiv:1907.11692 [cs.CL] Lucy Ellen Lwakatare, Aiswarya Raj, Ivica Crnkovic, Jan Bosch, and Helena Holmström Olsson

    RoBERTa: A Robustly Optimized BERT Pretraining Approach. doi:10.48550/arXiv.1907.11692 arXiv:1907.11692 [cs.CL] Lucy Ellen Lwakatare, Aiswarya Raj, Ivica Crnkovic, Jan Bosch, and Helena Holmström Olsson

    1907 doi arXiv

  6. [28]

    doi:10.1016/j.infsof.2020.106368 Proc

    Large-scale machine learning systems in real-world industrial settings: A review of challenges and solutions.Information and software technology127 (2020), 106368. doi:10.1016/j.infsof.2020.106368 Proc. ACM Softw. Eng., Vol. 2, No. FSE, Article FSE016. Publication date: July

    2020

  7. [29]

    In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering(Rochester, MI, USA)(ASE ’22)

    A transformer-based IDE plugin for vulnerability detection. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering(Rochester, MI, USA)(ASE ’22). Association for Computing Machinery, New York, NY, USA, Article 149, 4 pages. doi:10.1145/35...

  8. [30]

    doi:10.48550/arXiv.2406.09701 Tina Marjanov, Ivan Pashchenko, and Fabio Massacci

    Towards Effectively Detecting and Explaining Vulnerabilities Using Large Language Models.arXiv preprint arXiv:2406.09701(2024). doi:10.48550/arXiv.2406.09701 Tina Marjanov, Ivan Pashchenko, and Fabio Massacci

    2024 doi arXiv

  9. [31]

    doi:10.1109/MSEC.2022.3176058 Fabio Massacci, Stephan Neuhaus, and Viet Hung Nguyen

    Machine Learning for Source Code Vulnerability Detection: What Works and What Isn’t There Yet.IEEE Security & Privacy20, 5 (2022), 60–76. doi:10.1109/MSEC.2022.3176058 Fabio Massacci, Stephan Neuhaus, and Viet Hung Nguyen

    2022

  10. [34]

    https://dl.acm.org/doi/abs/10.5555/3620237.3620604 Van-Anh Nguyen, Dai Quoc Nguyen, Van Nguyen, Trung Le, Quan Hung Tran, and Dinh Phung

  11. [35]

    InProceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings

    ReGVD: Revisiting graph neural networks for vulnerability detection. InProceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings. 178–182. doi:10.1145/3510454.3516865 Viet Hung Nguyen, Stanislav Dashevskyi, and Fabio Massacci

  12. [36]

    doi:10.1007/s10664-015-9408-2 Chao Ni, Xin Yin, Kaiwen Yang, Dehai Zhao, Zhenchang Xing, and Xin Xia

    An automatic method for assessing the versions affected by a vulnerability.Empirical Software Engineering21 (2016), 2268–2297. doi:10.1007/s10664-015-9408-2 Chao Ni, Xin Yin, Kaiwen Yang, Dehai Zhao, Zhenchang Xing, and Xin Xia

    2016 doi

  13. [37]

    InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering(San Francisco, CA, USA)(ESEC/FSE 2023)

    Distinguishing Look-Alike Innocent and Vulnerable Code by Subtle Semantic Representation Learning and Explanation. InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering(San Francisco, CA, USA)(ESEC...

    2023

  14. [38]

    InProceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

    CrossVul: a cross-language vulnerability dataset with commit data. InProceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1565–1569. doi:10.1145/3468264.3473122 NIST. 2024a. National Vul...

  15. [41]

    doi:10.5555/1267336.1267343 Jalaj Pachouly, Swati Ahirrao, Ketan Kotecha, Ganeshsree Selvachandran, and Ajith Abraham

    10–5555. doi:10.5555/1267336.1267343 Jalaj Pachouly, Swati Ahirrao, Ketan Kotecha, Ganeshsree Selvachandran, and Ajith Abraham

  16. [42]

    doi:10.1016/j.engappai.2022.104773 Shengyi Pan, Lingfeng Bao, Xin Xia, David Lo, and Shanping Li

    A systematic literature review on software defect prediction using artificial intelligence: Datasets, Data Validation Methods, Approaches, and Tools.Engineering Applications of Artificial Intelligence111 (2022), 104773. doi:10.1016/j.engappai.2022.104773 Shengyi Pan, Lingfeng ...

    2022

  17. [45]

    InProceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security

    Vccfinder: Finding potential vulnerabilities in open-source projects to assist code audits. InProceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security. 426–437. doi:10.1145/2810103.2813604 Serena E. Ponta, Henrik Plate, Antonino Sabetta, Michele Be...

  18. [46]

    InProceedings of the 16th International Conference on Mining Software Repositories

    A Manually-Curated Dataset of Fixes to Vulnerabilities of Open-Source Software. InProceedings of the 16th International Conference on Mining Software Repositories. doi:10.1109/MSR.2019.00064 Proc. ACM Softw. Eng., Vol. 2, No. FSE, Article FSE016. Publication date: July

    2019

  19. [47]

    doi:10.5555/3698900.3699138 John Ruscio and Benjamin Lee Gera

  20. [48]

    doi:10.1080/00273171.2012.738184 Rebecca Russell, Louis Kim, Lei Hamilton, Tomo Lazovich, Jacob Harer, Onur Ozdemir, Paul Ellingwood, and Marc McConley

    Generalizations and extensions of the probability of superiority effect size estimator.Multivariate behavioral research48, 2 (2013), 208–219. doi:10.1080/00273171.2012.738184 Rebecca Russell, Louis Kim, Lei Hamilton, Tomo Lazovich, Jacob Harer, Onur Ozdemir, Paul Ellingwood, a...

    2013

  21. [49]

    In2018 17th IEEE international conference on machine learning and applications (ICMLA)

    Automated vulnerability detection in source code using deep representation learning. In2018 17th IEEE international conference on machine learning and applications (ICMLA). IEEE, 757–762. doi:10.1109/CCWC60891.2024.10427574 Nasir Safdari

    2024

  22. [50]

    doi:10.1109/TSE.2014.2340398 Martin Shepperd, Qinbao Song, Zhongbin Sun, and Carolyn Mair

    Predicting Vulnerable Software Compo- nents via Text Mining.IEEE Transactions on Software Engineering40, 10 (2014), 993–1006. doi:10.1109/TSE.2014.2340398 Martin Shepperd, Qinbao Song, Zhongbin Sun, and Carolyn Mair

    2014

  23. [51]

    doi:10.1109/TSE.2013.11 Yonghee Shin, Andrew Meneely, Laurie Williams, and Jason Osborne

    Data quality: Some comments on the nasa software defect datasets.IEEE Transactions on software engineering39, 9 (2013), 1208–1215. doi:10.1109/TSE.2013.11 Yonghee Shin, Andrew Meneely, Laurie Williams, and Jason Osborne

    2013 doi

  24. [52]

    Software Eng.37 (11 2011), 772–787

    Evaluating Complexity, Code Churn, and Developer Activity Metrics as Indicators of Software Vulnerabilities.IEEE Trans. Software Eng.37 (11 2011), 772–787. doi:10.1109/TSE.2010.81 Benjamin Steenhoek, Md Mahbubur Rahman, Richard Jiles, and Wei Le

    2011 doi

  25. [54]

    In2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)

    Mining software defects: Should we consider affected releases?. In2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 654–665. doi:10.1109/ICSE.2019.00075 Bin Yuan, Yifan Lu, Yilin Fang, Yueming Wu, Deqing Zou, Zhen Li, Zhi Li, and Hai Jin

    2019

  26. [55]

    In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)

    Enhancing Deep Learning-based Vulnerability Detection by Building Behavior Graph Model. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). 2262–2274. doi:10.1109/ICSE48619.2023.00190 Yunhui Zheng, Saurabh Pujar, Burn Lewis, Luca Buratti, Edward Epste...

    2023

  27. [56]

    In2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)

    D2a: A dataset built for ai-based vulnerability detection methods using differential analysis. In2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 111–120. doi:10.1109/ICSE-SEIP52600.2021.00020 Yaqin Zhou, ...

  28. [57]

    InAdvances in Neural Information Processing Systems

    Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks. InAdvances in Neural Information Processing Systems. 10197–10207. doi:10.5555/3454287.3455202 Received 2024-09-11; accepted 2025-01-14 Proc. ACM Softw. Eng., V...

  29. [117]

    Ranindya Paramitha, Yuan Feng, and Fabio Massacci. 2025d. Replication package on Zenodo Part 4 (POPPLER Dataset). Zenodo link. If the link does not work, copy and paste the following link https://doi.org/10.5281/zenodo.14713143. Henning Perl, Sergej Dechand, Matthew Smith, Dan...

    doi

  30. [297]

    doi:10.6028/NIST.SP.500-297 Andy Ozment and Stuart E Schechter

    doi

  31. [2007]

    InProceedings of the 22nd IEEE/ACM International Conference on Automated Software Engineering(Atlanta, Georgia, USA)(ASE ’07)

    Extraction of Bug Localization Benchmarks from History. InProceedings of the 22nd IEEE/ACM International Conference on Automated Software Engineering(Atlanta, Georgia, USA)(ASE ’07). Association for Computing Machinery, New York, NY, USA, 433–436. doi:10.1145/1321631.1321702 N...

  32. [2009]

    In2009 International Conference on Software Testing Verification and Validation

    Seasonal variation in the vulnerability discovery process. In2009 International Conference on Software Testing Verification and Validation. IEEE, 191–200. doi:10.1109/ICST.2009.9 Johnb110

    2009 doi

  33. [2011]

    InInternational Symposium on Engineering Secure Software and Systems

    After-life vulnerabilities: a study on firefox evolution, its vulnerabilities, and fixes. InInternational Symposium on Engineering Secure Software and Systems. Springer, 195–208. doi:10.1007/978-3-642-19125-1_15 Andreas Mauczka, Florian Brosch, Christian Schanes, and Thomas Grechenig

    doi

  34. [2013]

    Report on the static analysis tool exposition (sate) iv.NIST Special Publication500 (2013),

    2013

  35. [2014]

    InISSTA 2014, Proceedings of the 2014 International Symposium on Software Testing and Analysis

    Defects4J: A Database of existing faults to enable controlled testing studies for Java programs. InISSTA 2014, Proceedings of the 2014 International Symposium on Software Testing and Analysis. San Jose, CA, USA, 437–440. doi:10.1145/2610384.2628055 Tool demo. Christian Kästner

    2014

  36. [2015]

    In2015 IEEE/ACM 12th Working Conference on Mining Software Repositories

    Dataset of developer-labeled commit messages. In2015 IEEE/ACM 12th Working Conference on Mining Software Repositories. IEEE, 490–493. doi:10.1109/MSR. 2015.71 Yisroel Mirsky, George Macon, Michael Brown, Carter Yagemann, Matthew Pruett, Evan Downing, Sukarno Mertoguno, and Wenke Lee

    2015 doi

  37. [2016]

    In2016 IEEE 16th International Working Conference on Source Code Analysis and Manipulation (SCAM)

    Vulnerability Prediction Models: A Case Study on the Linux Kernel. In2016 IEEE 16th International Working Conference on Source Code Analysis and Manipulation (SCAM). 1–10. doi:10.1109/SCAM.2016.15 Matthieu Jimenez, Renaud Rwemalika, Mike Papadakis, Federica Sarro, Yves Le Trao...

    2016 doi

  38. [2017]

    InProceedings of the 13th International Conference on Predictive Models and Data Analytics in Software Engineering

    Boosting automatic commit classification into maintenance activities by utilizing source code changes. InProceedings of the 13th International Conference on Predictive Models and Data Analytics in Software Engineering. 97–106. doi:10.1145/3127005.3127016 Zhen Li, Ning Wang, De...

  39. [2018]

    In2018 IEEE 18th International Working Conference on Source Code Analysis and Manipulation (SCAM)

    [Engineering Paper] Enabling the Continuous Analysis of Security Vulnerabilities with VulData7. In2018 IEEE 18th International Working Conference on Source Code Analysis and Manipulation (SCAM). IEEE, 56–61. doi:10.1109/SCAM.2018.00014 Matthieu Jimenez, Mike Papadakis, and Yve...

  40. [2019]

    Proceedings of the ACM on Programming Languages3, POPL (2019), 1–29

    code2vec: Learning distributed representations of code. Proceedings of the ACM on Programming Languages3, POPL (2019), 1–29. doi:10.1145/3290353 Daniel Arp, Erwin Quiring, Feargus Pendlebury, Alexander Warnecke, Fabio Pierazzi, Christian Wressnegger, Lorenzo Cavallaro, and Kon...

    2019 doi

  41. [2020]

    InProceedings of the 17th International Conference on Mining Software Repositories

    A C/C++ code vulnerability dataset with code changes and CVE summaries. InProceedings of the 17th International Conference on Mining Software Repositories. 508–512. doi:10.1145/3379597.3387501 Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou...

  42. [2021]

    Unified Pre-training for Program Understanding and Generation. InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-...

    2021 doi

  43. [2022]

    In31st USENIX Security Symposium (USENIX Security 22)

    Dos and don’ts of machine learning in computer security. In31st USENIX Security Symposium (USENIX Security 22). 3971–3988. doi:10.48550/arXiv.2010.09470 Guru Bhandari, Amara Naseer, and Leon Moonen

    2010 doi arXiv

  44. [2023]

    InProceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses(Hong Kong, China)(RAID ’23)

    DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection. InProceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses(Hong Kong, China)(RAID ’23). Association for Computing Machinery, New York, NY, U...

  45. [2025]

    Association for Computational Linguistics, Online, 1536–1547

    FSE016:24 Ranindya Paramitha, Yuan Feng, and Fabio Massacci the Association for Computational Linguistics: EMNLP 2020, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, Online, 1536–1547. doi:10.18653/v1/2020.findings-emnlp.139 Michael Fu a...

    2020 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.