REVIEW 5 cited by
How Far Have We Gone in Vulnerability Detection Using Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
As software becomes increasingly complex and prone to vulnerabilities, automated vulnerability detection is critically important, yet challenging. Given the significant successes of large language models (LLMs) in various tasks, there is growing anticipation of their efficacy in vulnerability detection. However, a quantitative understanding of their potential in vulnerability detection is still missing. To bridge this gap, we introduce a comprehensive vulnerability benchmark VulBench. This benchmark aggregates high-quality data from a wide range of CTF (Capture-the-Flag) challenges and real-world applications, with annotations for each vulnerable function detailing the vulnerability type and its root cause. Through our experiments encompassing 16 LLMs and 6 state-of-the-art (SOTA) deep learning-based models and static analyzers, we find that several LLMs outperform traditional deep learning approaches in vulnerability detection, revealing an untapped potential in LLMs. This work contributes to the understanding and utilization of LLMs for enhanced software security.
Forward citations
Cited by 5 Pith papers
-
Measuring and Augmenting Large Language Models for Solving Capture-the-Flag Challenges
A benchmark and agent for CTF solving, but the agent's retrieval database appears to contain the answers to the test challenges, undermining the reported improvements.
-
Mono: Is Your "Clean" Vulnerability Dataset Really Solvable? Exposing and Trapping Undecidable Patches and Beyond
Mono reports that 31% of MegaVul patches are non-security and about 16.7% of CVEs are 'undecidable', while its added context raises LLM vulnerability detection F1 by up to 15%.
-
ReCopilot: Reverse Engineering Copilot in Binary Analysis
ReCopilot is a 7B binary-analysis LLM that reports 13% higher average scores than existing tools on a new six-task benchmark, but the benchmark and artifacts are not released.
-
LPASS: Linear Probes as Stepping Stones for vulnerability detection using compressed LLMs
Linear probe accuracy on simple code metrics can guide layer pruning and roughly predict post-fine-tuning vulnerability detection performance, but several headline numbers in the abstract do not match the paper's own tables.
-
LLMs in Software Security: A Survey of Vulnerability Detection Techniques and Insights
A survey of LLM-based vulnerability detection covering 58 papers, with a taxonomy, dataset overview, and gap analysis, but limited by non-transparent selection and unsupported quantitative claims.
Discussion (0). Continue with ORCID to comment.