Pith. sign in

REVIEW 5 major objections 6 minor 4 cited by

SecVulEval: Benchmarking LLMs for Real-World C/C++ Vulnerability Detection

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces SecVulEval, a statement-level C/C++ vulnerability benchmark, and reports that the best tested LLM reaches only 23.83% F1 on locating the vulnerable statement with correct reasoning.

desk verdict A potentially useful statement-level C/C++ vulnerability benchmark, but the ground-truth labels are built from deleted fix lines with no validation study, so the headline F1 numbers should be read as provisional. read the letter →

arxiv 2505.19828 v1 pith:YW6LLBWC submitted 2025-05-26 cs.SE

Md Basim Uddin Ahmed , Nima Shiri Harzevili , Jiho Shin , Hung Viet Pham , Song Wang This is my paper
classification cs.SE
keywords vulnerabilitydetectionstatement-levellabelingC/C++largelanguagemodelsbenchmarkCVEcontextextractionmulti-agentLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SecVulEval is a benchmark of 25,440 C/C++ functions spanning 5,867 real CVEs, built to judge vulnerability detection at the level of individual statements rather than whole functions. The authors argue that function-level datasets inflate apparent model ability and omit program context that real vulnerabilities depend on, so each function is annotated with external context in five categories. Using a multi-agent pipeline to evaluate five large language models, the benchmark reports that the best model, Claude-3.7-Sonnet, reaches 23.83% F1 for flagging the correct vulnerable statement with sound reasoning. The paper's central message is that current LLMs remain far from reliable tools for real-world C/C++ vulnerability localization, and that statement-level evaluation is needed to see this.

What carries the argument

The central machinery is the benchmark's labeling rule paired with its context schema. The rule upgrades raw deleted lines from fixing commits into complete syntactic statements, which matters because a C/C++ statement routinely spans multiple lines, so line fragments alone are not meaningful units. The context schema, adapted from prior work, distinguishes five kinds of external information, and an LLM-assisted extractor fills it in automatically with an estimated 82.98% accuracy on a 100-sample check. This two-part design lets scoring require both the correct statement and the correct reason, rather than a binary guess about the function.

What would settle it

Independently annotate a random sample of, say, 200 SecVulEval functions to separate vulnerability-causing statements from deleted lines that are refactoring or collateral edits, then recompute the 23.83% F1 and the per-model rankings; if agreement with the commit-derived labels is low, the benchmark's headline result measures label noise rather than model ability.

Watch

Extended reading notes

Core claim

SecVulEval's central claim is that statement-level labels plus rich context change what vulnerability detection benchmarks can honestly claim. The dataset marks a statement as vulnerable when it appears among the deleted lines of a CVE's single fixing commit, keeps vulnerable and fixed versions of every function, strips tangled and duplicate functions, and attaches CWE, CVE, and commit metadata. It also records, for each function, which of five context types — function arguments, external functions, type or execution declarations, globals, and execution environments — are needed to understand the flaw. Evaluated with a five-agent pipeline, the tested models detect the vulnerable statement with correct reasoning at best 23.83% F1, with closed models over-flagging and open models under-flagging. The authors read this as evidence that function-level scores overstate LLM ability and that fine-grained localization is the still-unsolved core of the task.

Load-bearing premise

The load-bearing premise is that the deleted lines in each fixing commit are exactly the vulnerable statements; that mapping is borrowed from commit data and never independently validated, so if fixes include refactoring, partial repairs, or unrelated edits, every statement-level score inherits those errors.

Editorial extensions

If this is right

  • Statement-level F1 with verified reasoning becomes the metric that matters; function-level accuracy alone no longer counts as evidence that a model detects real vulnerabilities.
  • The zero-duplicate design closes a known data-leakage channel, so future scores on SecVulEval can be attributed to model capability rather than to overlap with training data.
  • The benchmark supports training and evaluating models that must name the source statement and explain the cause, not just classify functions, which is the granularity a human reviewer or auto-fixing tool needs.
  • The high-recall, low-precision pattern of the closed models implies that using them as review aids would produce many false alarms unless precision is explicitly addressed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: because the ground-truth statements come from deleted lines, the 23.83% F1 is measured against a label set that may contain refactoring or partial-fix noise; a manually corrected label set could move the number in either direction.
  • Beyond the paper: the context extractor's 82.98% accuracy suggests a direct controlled run — human-verified contexts instead of LLM-extracted ones on the same sampled functions — to separate model weakness from missing context.
  • Beyond the paper: the systematic over-flagging of null dereference, use-after-free, and integer overflow invites an ablation that mutates only the null checks and return statements and observes whether verdicts flip, which would reveal shallow-cue reliance.
  • Beyond the paper: a single-agent version of the same models on the same 300 samples would quantify how much of the reported ceiling is due to the multi-agent consensus design rather than to the underlying LLMs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces SecVulEval, a C/C++ vulnerability benchmark with statement-level vulnerability labels, contextual information extracted by GPT-4.1, and claimed de-duplication. It reports 25,440 function samples from 5,867 CVEs across 707 projects. The authors evaluate five LLMs (Qwen2.5-Coder-32B, Deepseek-Coder-33B, Codestral-22B, GPT-4.1, Claude-3.7-Sonnet) using a five-agent pipeline. The headline result is that Claude-3.7-Sonnet achieves only 23.83% F1-score for statement-level vulnerability detection with correct reasoning, with GPT-4.1 close behind. The paper also evaluates context identification accuracy across the five context categories.

Significance. If the statement-level labels are reliable, SecVulEval would be a valuable resource: it is large, covers diverse real-world C/C++ projects and CWEs, includes rich metadata and contexts, and applies rigorous de-duplication. The release of dataset and code supports reproducibility. The finding that strong LLMs achieve low statement-level F1 scores is a useful, falsifiable result that aligns with recent concerns about overestimated vulnerability-detection performance. However, the central contribution is the statement-level ground truth, and that ground truth is derived automatically from fix-commit changed lines with no validation study. The paper's main claims stand or fall on the correctness of these labels, so the missing label validation is a load-bearing issue, not a presentation concern.

major comments (5)
  1. [§3.2, §3.3] The statement-level ground-truth labels are never validated. Section 3.2 derives labels from changed lines in fixing commits, and Section 3.3 applies heuristics to filter tangled files/functions, but there is no manual validation of whether deleted (or added) lines actually correspond to vulnerable statements. Fix commits often contain refactoring, reformatting, or partial fixes, and the filtering criteria do not guarantee that all such noise is removed. The only manual validations reported are for GPT-4.1 context extraction (§3.4, 100 samples) and for LLM outputs (§5.1, 300 samples). Appendix A lists limitations but omits any caveat about label correctness. Without a label-validation study, the headline F1 scores (e.g., 23.83% for Claude-3.7-Sonnet) may reflect agreement with noisy annotations rather than true detection ability.
  2. [§3.2, §3.3, Figure 1] The transformation from changed lines to 'statements' is not described. C/C++ statements often span multiple lines, which the paper itself notes as a motivation for statement-level labels, yet the paper never explains how added/deleted lines are grouped into statements, how multi-line statements are handled, or whether the ground-truth vulnerable statements are the deleted lines, the added lines, or both. The evaluation in §5.1 compares model outputs to 'changed statements,' but without a precise definition of this mapping, the statement-level metric is ambiguous and not reproducible.
  3. [§3.3] The construction of the non-vulnerable function set is not described. The paper reports 14,442 non-vulnerable functions, but the pipeline in Figure 1 and Sections 3.1–3.3 only explains how vulnerable functions are collected and filtered. No section describes where non-vulnerable functions come from, how they are sampled, or how they are validated as truly non-vulnerable. Because function-level precision and recall in Table 3 depend on the negative set, the sampling of these 14,442 functions directly affects the reported metrics, including the 300-sample manual evaluation that contains 120 vulnerable and 180 non-vulnerable functions.
  4. [§5.1] The statement-level evaluation is based on manual validation of only 300 samples drawn from the Top-25 CWE types, and no inter-annotator agreement is reported. The acceptance criteria in §5.1 (exact statements, correct reasoning, sink vs. source, allowance of at most two unrelated statements) involve subjective judgment, and a single annotator's decisions may not be reliable. The paper should report at least two annotators and an agreement measure (e.g., Cohen's kappa) for a random subset to support the correctness of the statement-level TP/FP classifications that underlie all headline F1 scores.
  5. [§4.2, §5.2] The context-identification ground truth is generated by GPT-4.1, and GPT-4.1 is also one of the evaluated models. Section 4.2 says the dataset's context annotations were extracted by GPT-4.1, and Section 5.2 compares each model's extracted symbols against those annotations. Evaluating GPT-4.1 against a GPT-4.1-generated gold standard is circular, and even for other models, the gold standard is only 82.98% accurate on a 100-sample validation. The accuracy is reported as an overall percentage and permits up to one superfluous symbol per category, which may mask per-category errors. This weakens the conclusions of Experiment 2, such as the claim that Claude-3.7-Sonnet is better at identifying type definitions and globals.
minor comments (6)
  1. [Table 1] The '0.0' duplicate rate is based on md5 hashing of whitespace-normalized function strings. Semantic duplicates, such as renamed variables or reformatted functions, would not be detected, so the claim of zero duplication is stronger than the method supports.
  2. [§4.3] The statement-level True Positive criterion 3 in §5.1 allows a prediction to be a True Positive even when it includes up to two unrelated statements alongside the correct one. This permissive criterion should be stated in the metrics section, since it directly affects the reported F1 scores.
  3. [Table 3] The table reports point estimates without confidence intervals or significance tests. The difference between Claude-3.7-Sonnet (23.83%) and GPT-4.1 (22.38%) at statement level is small relative to the 300-sample evaluation, so it is unclear whether the ranking is meaningful.
  4. [§4.4] The paper uses pass@1 with temperature = 0.1. At such a low temperature, outputs are nearly deterministic, so 'pass@1' is an unusual term; this should be clarified or replaced with a standard single-sample evaluation description.
  5. [§4.1] The statement that this is 'the first time that an LLM-based multi-agent pipeline is applied for the vulnerability detection task' is a strong novelty claim and should be supported with a more thorough comparison to existing multi-agent approaches, or qualified.
  6. [§3.4] The context-extraction validation reports '82.98 ± 9.68% (with 99% Confidence Interval)'. The notation is unconventional: it would be clearer to state the estimated accuracy and the confidence interval explicitly, and to clarify that accuracy is the proportion of samples with correct context identification as defined by the permissive criterion.

Circularity Check

1 steps flagged · score 2.0 of 10

GPT-4.1's context-identification score is measured against contexts that GPT-4.1 itself generated; the headline vulnerability-detection claim is independent.

  1. other [Section 3.4 (Contextual Information Collection) and Section 4.2 / Table 4 (Experiment 2)]
    "We prompt GPT-4.1 with all the available information to identify the context required to understand the vulnerability in this function, and categorize them according to the five definitions. ... we compare the LLM-generated symbols against the ground-truth dependencies annotated in our dataset."

    The dataset's context annotations are generated by GPT-4.1 in Section 3.4 and are then treated as the 'ground-truth dependencies' in Section 4.2. For the GPT-4.1 row in Table 4, the evaluated model's context extraction is compared with symbols that the same model itself produced during dataset construction. Thus GPT-4.1's context-identification accuracy on the full dataset is a measure of self-consistency rather than independent correctness; only 100 samples received human-validated ground truth (82.98% accuracy), so the full comparison lacks an independent reference. This is a partial circularity confined to Experiment 2.

full rationale

The main derivation chain—converting fixing commits into statement-level labels and benchmarking LLM vulnerability detection—does not collapse into its inputs. Vulnerable statements are defined from deleted/changed lines in fix commits, and the multi-agent detectors receive only function code, ASTs, and context, not the patch; the reported 23.83% F1 is therefore an independent outcome, not a fitted or renamed input. The low scores also show the models do not trivially reproduce the labels. No load-bearing self-citation or imported uniqueness theorem is present; reference [21] is a background citation. The one mild circularity is confined to Experiment 2: GPT-4.1 both generated the dataset's context annotations and is one of the models scored against those annotations, so Table 4's absolute accuracies (especially GPT-4.1's) are partly self-referential. The paper's own limitation appendix acknowledges that 'relevant context' is not an objective quantity, supporting the concern. The unvalidated statement-level labels are a data-quality/correctness risk rather than a circularity, and they do not raise the circularity score beyond the noted context-evaluation issue.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The benchmark relies on several domain assumptions about the source data and automated labeling, but introduces no new free parameters or invented entities. The key assumption is the validity of deleted lines as vulnerability labels, which is untested.

assumptions (4)
  • domain assumption CVE records and patch references in NVD are accurate and map to the correct fixing commits.
    Section 3.1 relies on NVD API data and the 'Patch' tag to identify vulnerabilities; inaccuracies would propagate into the dataset.
  • domain assumption Statements deleted in a fixing commit correspond to the vulnerable statements.
    Section 3.2/3.3 extracts changed lines and uses deleted lines as ground-truth vulnerable statements; this mapping is never validated.
  • domain assumption The five context categories from Risse et al. are sufficient to represent all relevant contextual information for a vulnerability.
    Section 3.4 adopts these categories; if context types outside these five matter, the benchmark's context annotations would be incomplete.
  • domain assumption GPT-4.1's context extraction is accurate enough (82.98% on 96 samples) to serve as the dataset's context ground truth.
    Section 3.4 reports a small manual validation; the accuracy may not generalize to the full dataset.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SecVulEval: Benchmarking LLMs for Real-World C/C++ Vulnerability Detection." pith.science (2026). https://pith.science/paper/YW6LLBWC

@misc{pith2026250519828,
  author       = {Pith},
  title        = {Pith review of: SecVulEval: Benchmarking LLMs for Real-World C/C++ Vulnerability Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YW6LLBWC}},
  note         = {Machine review of arXiv:2505.19828}
}
read the original abstract

Large Language Models (LLMs) have shown promise in software engineering tasks, but evaluating their effectiveness in vulnerability detection is challenging due to the lack of high-quality datasets. Most existing datasets are limited to function-level labels, ignoring finer-grained vulnerability patterns and crucial contextual information. Also, poor data quality such as mislabeling, inconsistent annotations, and duplicates can lead to inflated performance and weak generalization. Moreover, by including only the functions, these datasets miss broader program context, like data/control dependencies and interprocedural interactions, that are essential for accurately understanding real-world security flaws. Without this context, detection models are evaluated under unrealistic assumptions. To address these limitations, this paper introduces SecVulEval, a benchmark designed to support fine-grained evaluation of LLMs and other detection methods with rich contextual information. SecVulEval focuses on real-world C/C++ vulnerabilities at the statement level. This granularity enables more precise evaluation of a model's ability to localize vulnerabilities, beyond simple binary classification at the function level. By incorporating rich contextual information, SecVulEval sets a new standard for vulnerability detection benchmarks in realistic scenarios. This benchmark includes 25,440 function samples covering 5,867 unique CVEs in C/C++ projects from 1999 to 2024. We evaluated the SOTA LLMs with a multi-agent-based approach. The evaluation on our dataset shows that the models are still far from accurately predicting vulnerable statements in a given function. The best-performing Claude-3.7-Sonnet model achieves 23.83% F1-score for detecting vulnerable statements with correct reasoning. Finally, we analyze the LLM outputs and provide insights into their behavior in vulnerability detection for C/C++.

Figures

Figures reproduced from arXiv: 2505.19828 by the authors.

Figure 1
Figure 1. Data collection pipeline of SECVULEVAL In our work, we address these shortcomings and challenges by including vulnerable and non￾vulnerable functions along with statement-level vulnerability labels, along with contextual information for the vulnerability. These are accompanied by other metadata for varied analysis, along with rigorous de-duplication and filtering to maintain data quality [PITH_FULL_IMAGE:figures/fu… view at source ↗
Figure 2
Figure 2. Distribution of five context cate [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Number of vulnerable functions in Top-20 CWE [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The multi-agent vulnerability detection pipeline. Each agent, except the first one, is driven [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: GPT-4.1 marking line-10 as Null Dereference but [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DREA: Decoupled Reasoning and Exploration Agents for Repository-Level Vulnerability Detection

    cs.CR 2026-07 conditional novelty 6.0 of 10

    DREA improves repository-level vulnerability detection by coupling an LLM planner that forms hypotheses with a cheap local explorer that gathers cross-file evidence, lifting paired accuracy from 19-26% to 30-42% at mu...

  2. Neuro-Symbolic Reasoning for Vulnerability Detection

    cs.SE 2026-07 conditional novelty 6.0 of 10

    Separating LLM fact filtering from Lean 4 obligation discharge improves vulnerability-detection F1 in all fifteen CWE×backend settings, especially doubling recall on double-free.

  3. VULPO: Context-Aware Vulnerability Detection via On-Policy LLM Optimization

    cs.CR 2025-11 conditional novelty 6.0 of 10

    A 4B LLM trained with judge-scored, difficulty-scaled on-policy RL reaches F1 70.45 on context-aware C/C++ vulnerability detection, comparable to DeepSeek-V3.1, with the caveat that the same judge provided the trainin...

  4. VulnRepairEval: An Exploit-Based Evaluation Framework for Assessing Large Language Model Vulnerability Repair Capabilities

    cs.SE 2025-09 conditional novelty 6.0 of 10

    A benchmark that requires a patch to block a working exploit finds the best LLM repairs only 21.7% of 23 real CVEs, with most failures caused by missed localization and malformed patches.

Reference graph

Works this paper leans on

25 extracted references · 15 canonical work pages · cited by 4 Pith papers

  1. [1]

    Cvefixes: automated collection of vulner- abilities and their fixes from open-source software

    Guru Bhandari, Amara Naseer, and Leon Moonen. Cvefixes: automated collection of vulner- abilities and their fixes from open-source software. In Proceedings of the 17th International Conference on Predictive Models and Data Analytics in Software Engineering , pages 30–39, 2021

    work page 2021

  2. [2]

    Deep learning based vulnerability detection: Are we there yet? IEEE Transactions on Software Engineering , 48(9):3280–3296, 2021

    Saikat Chakraborty, Rahul Krishna, Yangruibo Ding, and Baishakhi Ray. Deep learning based vulnerability detection: Are we there yet? IEEE Transactions on Software Engineering , 48(9):3280–3296, 2021

    work page 2021

  3. [3]

    Diversevul: A new vulnerable source code dataset for deep learning based vulnerability detection

    Yizheng Chen, Zhoujie Ding, Lamya Alowain, Xinyun Chen, and David Wagner. Diversevul: A new vulnerable source code dataset for deep learning based vulnerability detection. In Proceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses, pages 654–668, 2023

    work page 2023

  4. [4]

    CreativEval: Evaluating Creativity of LLM-Based Hardware Code Generation

    Matthew DeLorenzo, Vasudev Gohil, and Jeyavijayan Rajendran. Creativeval: Evaluating creativity of llm-based hardware code generation. arXiv preprint arXiv:2404.08806, 2024

    work page Pith review arXiv 2024

  5. [5]

    Vulnerability detection with code language models: How far are we? arXiv preprint arXiv:2403.18624, 2024

    Yangruibo Ding, Yanjun Fu, Omniyyah Ibrahim, Chawin Sitawarin, Xinyun Chen, Basel Alomair, David Wagner, Baishakhi Ray, and Yizheng Chen. Vulnerability detection with code language models: How far are we? arXiv preprint arXiv:2403.18624, 2024

    arXiv 2024

  6. [6]

    Ac/c++ code vulnerability dataset with code changes and cve summaries

    Jiahao Fan, Yi Li, Shaohua Wang, and Tien N Nguyen. Ac/c++ code vulnerability dataset with code changes and cve summaries. In Proceedings of the 17th International Conference on Mining Software Repositories, pages 508–512, 2020

    work page 2020

  7. [7]

    Linevul: A transformer-based line-level vulner- ability prediction

    Michael Fu and Chakkrit Tantithamthavorn. Linevul: A transformer-based line-level vulner- ability prediction. In Proceedings of the 19th International Conference on Mining Software Repositories, pages 608–620, 2022

    work page 2022

  8. [8]

    Binaiv: Semantic-enhanced vulnerability detection for linux x86 binaries

    Yeming Gu, Hui Shu, and Fei Kang. Binaiv: Semantic-enhanced vulnerability detection for linux x86 binaries. C&S, 135:103508, 2023

    work page 2023

Show all 25 references
  1. [9]

    Deepseek-coder: When the large language model meets programming–the rise of code intelligence

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming–the rise of code intelligence. arXiv preprint arXiv:2401.14196, 2024

    2024 arXiv

  2. [10]

    Outside the comfort zone: Analysing llm capabilities in software vulnerability detection

    Yuejun Guo, Constantinos Patsakis, Qiang Hu, Qiang Tang, and Fran Casino. Outside the comfort zone: Analysing llm capabilities in software vulnerability detection. In European symposium on research in computer security , pages 271–289. Springer, 2024

    2024

  3. [11]

    Large language models for code: Security hardening and adversarial testing

    Jingxuan He and Martin Vechev. Large language models for code: Security hardening and adversarial testing. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pages 1865–1879, 2023

    2023

  4. [12]

    Linevd: Statement-level vulnerability detection using graph neural networks

    David Hin, Andrey Kan, Huaming Chen, and M Ali Babar. Linevd: Statement-level vulnerability detection using graph neural networks. In Proceedings of the 19th international conference on mining software repositories, pages 596–607, 2022

    2022

  5. [13]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jia- jun Zhang, Bowen Yu, Kai Dang, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186, 2024

    2024 arXiv

  6. [14]

    Llm-assisted code cleaning for training accurate code generators

    Naman Jain, Tianjun Zhang, Wei-Lin Chiang, Joseph E Gonzalez, Koushik Sen, and Ion Stoica. Llm-assisted code cleaning for training accurate code generators. arXiv preprint arXiv:2311.14904, 2023. 10

    2023 arXiv

  7. [15]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    arXiv

  8. [16]

    More agents is all you need

    Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu, and Deheng Ye. More agents is all you need. arXiv preprint arXiv:2402.05120, 2024

    2024 arXiv

  9. [17]

    Vulnerability management in linux distributions: An empirical study on debian and fedora

    Jiahuei Lin, Haoxiang Zhang, Bram Adams, and Ahmed E Hassan. Vulnerability management in linux distributions: An empirical study on debian and fedora. Empirical Software Engineering, 28(2):47, 2023

    2023

  10. [18]

    Megavul: Ac/c++ vulnerability dataset with comprehensive code representations

    Chao Ni, Liyu Shen, Xiaohu Yang, Yan Zhu, and Shaohua Wang. Megavul: Ac/c++ vulnerability dataset with comprehensive code representations. In 2024 IEEE/ACM 21st International Conference on Mining Software Repositories (MSR) , pages 738–742. IEEE, 2024

    2024

  11. [19]

    Top score on the wrong exam: On benchmarking in machine learning for vulnerability detection

    Niklas Risse and Marcel Böhme. Top score on the wrong exam: On benchmarking in machine learning for vulnerability detection. arXiv preprint arXiv:2408.12986, 2024

    2024 arXiv

  12. [20]

    Lprotector: An llm-driven vulnerability detection system

    Ze Sheng, Fenghua Wu, Xiangwu Zuo, Chao Li, Yuxin Qiao, and Lei Hang. Lprotector: An llm-driven vulnerability detection system. arXiv preprint arXiv:2411.06493, 2024

    2024 arXiv

  13. [21]

    A systematic literature review on automated software vulnerability detection using machine learning

    Nima Shiri Harzevili, Alvine Boaye Belle, Junjie Wang, Song Wang, Zhen Ming Jiang, and Nachiappan Nagappan. A systematic literature review on automated software vulnerability detection using machine learning. ACM Computing Surveys, 57(3):1–36, 2024

    2024

  14. [22]

    Simulating strategic reasoning: Comparing the ability of single llms and multi-agent systems to replicate human behavior

    Karthik Sreedhar and Lydia Chilton. Simulating strategic reasoning: Comparing the ability of single llms and multi-agent systems to replicate human behavior. 2025

    2025

  15. [23]

    Multi-agent collaboration mechanisms: A survey of llms

    Khanh-Tung Tran, Dung Dao, Minh-Duong Nguyen, Quoc-Viet Pham, Barry O’Sullivan, and Hoang D Nguyen. Multi-agent collaboration mechanisms: A survey of llms. arXiv preprint arXiv:2501.06322, 2025

    2025 arXiv

  16. [24]

    Large language model for vulnerability detection: Emerging results and future directions

    Xin Zhou, Ting Zhang, and David Lo. Large language model for vulnerability detection: Emerging results and future directions. InProceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results , pages 47–51, 2024

    2024

  17. [25]

    relevant context

    Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks. Advances in neural information processing systems , 32, 2019. 11 Appendices The appendix in...

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.