Pith. sign in

REVIEW 3 major objections 6 minor 42 references

Can LLM Prompting Serve as a Proxy for Static Analysis in Vulnerability Detection

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Security-aware LLM prompting, which augments CWE descriptions with contrastive vulnerable/fixed examples, outperforms default static analysis on partial-code vulnerability detection across four CWE types, lifting pairwise accuracy by up…

desk verdict A serious empirical exploration of CWE-specific LLM prompting for partial-code vulnerability detection, but the headline gains over static analyzers rest on a baseline that may be crippled by the snippet setting. read the letter →

arxiv 2412.12039 v3 pith:WP4MUD2M submitted 2024-12-16 cs.CR cs.AIcs.CLcs.SE

Ira Ceka , Feitong Qiao , Anik Dey , Aastha Valecha , Gail Kaiser , Baishakhi Ray This is my paper · ORCID
classification cs.CRcs.AIcs.CLcs.SE
keywords vulnerabilitydetectionstaticanalysisLLMpromptingchain-of-thoughtpartialcodeCWEcontrastiveexamplesDeepSeek-R1
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that carefully designed LLM prompting can replace, or at least complement, hand-written static-analysis rules for vulnerability detection when only partial code is available. The authors build CWE-specific prompts that pair natural-language descriptions of a vulnerability class with contrastive chain-of-thought examples (a vulnerable snippet and its fixed counterpart from the synthetic Juliet suite), then test these prompts on real-world samples from SVEN and CVEFixes across four CWE categories. Their central evidence is a head-to-head comparison with four static analyzers run with default settings on the same partial-code snippets: with the reasoning model DeepSeek-R1, every prompting strategy outperforms the static-analyzer baseline, and the best strategies improve accuracy by up to 31.6 percentage points, F1 by up to 71.7 percentage points, and pairwise accuracy (the fraction of vulnerable/fixed pairs where both snippets are correctly labeled) by up to 60.4 percentage points while reducing false negatives by up to 37.6 percentage points. If the comparison holds, snippet-level code review—common in pull requests, code reuse, and AI-assisted generation—can be made substantially more effective without training a model or building the project.

What carries the argument

The load-bearing mechanism is a contrastive chain-of-thought prompt: a CWE-specific instruction set phrased as checks, followed by a vulnerable example with reasoning for why it fails the checks, its fixed counterpart with reasoning for why it passes, and then the new snippet to be classified. The instruction sets come from four sources—the bare CWE name, LLM-generated instructions, LLM-generated instructions from few-shot samples, and human-authored security documentation—and the contrastive examples come from synthetic Juliet C/C++ vulnerable/fixed pairs. The mechanism's work is to make vulnerability detection a guided reasoning task rather than a raw binary classification: it gives the model a concrete pattern to match, a positive and negative reference, and a template that forces per-instruction justification before the final verdict.

What would settle it

Take the same SVEN and CVEFixes snippets and run CodeGuru, Semgrep, SonarQube, and CodeQL after restoring the full enclosing function and its project or build context (or with their intended code-analysis configurations rather than defaults), then compare recall and pairwise accuracy with the reported LLM numbers. If any tool's pairwise accuracy rises from near zero to the 26–60% range, the claimed advantage is an artifact of giving the analyzers a crippled setup.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that domain-informed prompting can elicit reliable vulnerability judgments from LLMs even in the hard partial-code setting where hand-written static-analysis rules fail. The paper evaluates prompts on 520 SVEN samples and 1,784 CVEFixes samples across CWE-078 (OS command injection), CWE-190 (integer overflow), CWE-416 (use after free), and CWE-476 (null pointer dereference). Static analyzers in the same setting score near-zero pairwise accuracy—the highest is about 3%—and often detect no true vulnerabilities at all: across CWE-190 and CWE-416 every tool reports zero true positives. Against that baseline, the paper finds that for every CWE there is at least one prompting strategy that outperforms all four analyzers, with the best gains coming from reasoning models such as DeepSeek-R1. The authors conclude that security-property-aware prompting can serve as a complementary proxy for static-analysis rules in snippet review, not that it replaces deep program analysis in full-project settings.

Load-bearing premise

The comparison assumes that running the four static analyzers with default settings on isolated snippets is a fair proxy for static-analysis capability; if those tools are handicapped by the snippet setting and would do substantially better with build context or tuned rules, the reported gains mostly show the baseline is weak rather than showing LLMs are inherently better.

Editorial extensions

If this is right

  • Partial-code vulnerability detection can be improved without model training or a full project build: for each CWE at least one prompting strategy outperforms every default static analyzer, so snippet review can be made substantially safer with prompt design alone.
  • The gains are not tied to one universal prompt; the best strategy varies by CWE and model (o1 with natural-language instructions for CWE-078, DeepSeek-R1 with generic chain-of-thought for CWE-190), so deployment needs per-CWE strategy selection.
  • The prompting advantage carries over to the noisier CVEFixes dataset, where accuracy improves by up to 12 percentage points over the prior GPT-3.5/GPT-4 baseline, suggesting the effect is not an artifact of the curated SVEN set.
  • Because static analyzers in this setting have extremely low recall, using these prompts as a complement would cut false negatives at review time, catching vulnerabilities that default SA rules miss.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the static analyzers were run with default settings on isolated snippets, the headline comparison does not show that LLMs are inherently better analysts—only that they beat default snippet-mode configurations. A hybrid pipeline that uses LLM prompts to triage snippets and static analysis to confirm on full builds is the natural next experiment.
  • Editorial inference: the contrastive-prompt construction (instructions plus vulnerable/fixed pairs plus reasoning) is largely automatic once a synthetic suite exists, so scaling the approach to the rest of the CWE Top 25 is plausible; the paper itself only demonstrates four CWE classes.
  • Editorial inference: the manual response analysis suggests many misses come from missing context and selective reasoning, so prompting the model to enumerate all data-flow paths or inspect each sink individually before deciding is a testable way to push pairwise accuracy higher.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper proposes a set of CWE-specific prompting strategies for LLM-based vulnerability detection on partial code snippets. The strategies combine a basic prompt with natural-language vulnerability instructions, chain-of-thought reasoning, and contrastive synthetic examples from the Juliet C/C++ suite. The authors evaluate the strategies on SVEN (four CWEs, 520 samples) and CVEFixes (1,784 samples) using o1, Claude 3.5 Sonnet v2, DeepSeek-R1, and several other models, and compare them against CodeGuru, Semgrep, SonarQube, and CodeQL run with default settings on the same partial snippets. The paper reports that for each CWE some prompting strategy exceeds the static-analyzer baseline, with pairwise-accuracy gains up to 60.4 points, and also compares against Khare et al. on CVEFixes. The central claim is that LLM prompting can serve as a proxy for static analysis in the partial-code setting.

Significance. If the comparison were fair and the results reproducible, this would be a practically useful result: partial-code review is common in pull requests and AI-assisted coding, and a drop-in prompt-based detector that beats static analyzers at snippet level would be valuable. The paper's strengths include the use of a cleaner real-world dataset (SVEN), a clearly described multi-phase prompt-generation pipeline with concrete templates, a manual error analysis of 414 responses, a robustness check on tag-based formatting, a cost analysis, and a publicly announced replication package. However, the headline comparative claim depends on the fairness of the static-analyzer baseline and on the integrity of the SVEN split for instruction generation; both are currently under-specified. In addition, the per-CWE best results are selected post hoc from many strategy-model combinations without significance testing, so the quantitative claim of "significantly improves" is not yet supported.

major comments (3)
  1. [§IV-B, Table V] The static-analyzer baseline is constructed by running CodeGuru, Semgrep, SonarQube, and CodeQL with default settings on isolated partial snippets and applying only queries "capable of operating on partial code." This is not the intended operating regime for CodeQL and SonarQube, which rely on build/project context and interprocedural dataflow; CodeQL is in fact reported only for CWE-078. Consequently, every static-analyzer row in Table V has recall below 6.3% and pairwise accuracy below 3%, so the baseline behaves like a near-zero detector. The headline improvements in Table IV (F1 +71.7, pairwise +60.4, accuracy +31.6) are measured against a baseline that may be crippled by the snippet harness rather than a fair representative of static-analysis capability. Please re-run the same tools on the full functions or repositories from which the SVEN snippets are drawn, using their standard CWE-relevant security query suites, and report whether the conclusions change; if the full-context setting is infeasible, state this explicitly and restrict the claims to the partial-code setting without the "proxy for static analysis" phrasing.
  2. [§III-B, §IV-C] The NL(S2) instructions are generated by prompting the LLM with three vulnerable/fixed pairs "extracted from the SVEN validation set for that particular CWE," while the evaluation in Section V is reported on SVEN. The paper never documents a split that excludes these three pairs from the evaluation set. If any of those pairs appear in the test data, the NL(S2) setting, which is the top-performing setting for CWE-416 in Table IV, is evaluated with instruction-generation examples that are also in the test set, inflating its results. Please specify the exact SVEN split, confirm that the three selected pairs per CWE are excluded from all evaluation, and, if they are not, re-run the affected results and adjust the conclusions.
  3. [Table IV, §V-A] For each CWE, the headline winner is selected post hoc as the best combination of setting and model (e.g., NL (S1) with o1 for CWE-078, NL+CoT (S0) with DeepSeek-R1 for CWE-190), and the table reports only point estimates averaged over three trials with no confidence intervals or significance tests. With CWE-level sample sizes of 76 to 190 and pairwise-accuracy differences as small as +8.8 points (CWE-190), the claim that the best strategy "significantly improves" over the baselines is not supported by the data. Please add per-strategy variance and, at minimum, a paired bootstrap or permutation test against the vanilla prompt and static-analyzer baselines, or explicitly soften the "significantly improves" language to "improves in this sample."
minor comments (6)
  1. [§VII-B] The text "Table VII-B provides the average token count" appears to refer to the cost-analysis table, which is labeled Table IX; please fix the cross-reference.
  2. [Table VIII] The caption uses "S1" and "S2" without defining them; clarify that S1 is the basic prompt with the verbosity instruction and S2 is the tag-based format without the verbosity instruction.
  3. [§III-D, Phase 1] The phase description says the authors "manually select pairwise samples" but also "randomly select one such scenario"; please clarify whether the selection is manual, random, or a combination.
  4. [Table IV] The FNR/FPR columns mix two metrics into a single delta value in the "FNR / FPR" header; separate the FNR and FPR columns or precisely define what the single reported number represents.
  5. [Appendix] The appendix prompt text contains stray characters (e.g., "if (data ¡ (LLONG MAX/2))"); please proofread the appendix for encoding and typographic errors.
  6. [Figure 5] The caption does not specify what the plotted values are (accuracy, F1, or pairwise accuracy); clarify the caption and legend so the figure is self-contained.

Circularity Check

1 steps flagged · score 4.0 of 10

One evaluation strategy is seeded from the test set: NL(S2) instructions are generated from three SVEN pairs with no documented hold-out, so the CWE-416 best result is partly fit to the evaluation data; the rest of the evaluation is self-contained.

  1. fitted input called prediction [Section III-B (NL (S2)); Section IV-C (Datasets); Table IV (CWE-416 row)]
    "NL (S2): We utilize LLM-generated instructions from few-shot samples. We ask the LLM to culminate a set of instructions based on 3 pairwise (both vulnerable and fixed) samples extracted from the SVEN validation set for that particular CWE. ... SVEN (real-world): a manually-labeled, balanced dataset known to have 94% label accuracy. This dataset originally comprises of 803 vulnerable/non-vulnerable pairs (1.6k samples total). We filter for the 4 CWE's, extracting 520 samples total (the full dataset for the 4 CWEs)."

    The NL(S2) prompt is, by construction, a function of the labels of three vulnerable/fixed SVEN pairs. The paper never states that these three pairs were removed from the 520 SVEN samples on which Table IV reports accuracy/F1/pairwise accuracy; it calls SVEN 'the full dataset for the 4 CWEs.' Under that description the seed pairs are in the evaluation set, so for those pairs the LLM is not predicting but recalling examples used to build the prompt. Table IV makes NL(S2) the best strategy for CWE-416, so the per-CWE-best claim rests in part on this leaked setting. Headline gains for CWE-078/190/476 come from NL(S1), NL+CoT(S0), and NL(S0), which do not use SVEN examples, so the circularity is partial.

full rationale

The derivation chain for the prompting strategies is otherwise self-contained: contrastive examples come from the synthetic Juliet C/C++ dataset, instructions for S1/S3 and all NL+CoT variants come from LLM generation or MITRE text, and evaluation is on SVEN/CVEFixes, which are external to the prompt-construction inputs. The static-analyzer comparison is a baseline-fairness concern (tools run in partial-snippet mode with only partial-code-capable queries), not a circularity of the paper's derivation, and I do not score it here. There is no load-bearing self-citation chain: the cited prior works of the authors are used for context and metrics, not to justify the central claim. The only concrete circularity signal is the NL(S2) construction: because the instruction is generated from three SVEN pairs and the paper documents no split excluding those pairs from the 520-sample SVEN evaluation, the CWE-416 best-setting result is partly fitted to the data it is said to predict. This affects one strategy and one CWE of the four, while the headline accuracy/F1/pairwise/FNR improvement numbers come from non-leaking strategies, so a moderate score of 4 is appropriate rather than 6-8.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

This is an empirical benchmark paper, so there are no fitted equations or invented physical entities. The central comparison rests on three domain assumptions: the static analyzer baseline is representative, SVEN labels are reliable, and prompt construction does not leak evaluation signal.

assumptions (3)
  • domain assumption Default partial-code execution of CodeGuru, Semgrep, SonarQube, and CodeQL is a fair proxy for static analysis capability.
    Section IV-B and Table V: all comparisons use default settings and only queries that run on partial code. If these tools require build context or configured rules, the baseline is weakened.
  • domain assumption SVEN ground truth labels are accurate enough to score the models.
    Section IV-C states SVEN has 94 percent label accuracy, and all accuracy, F1, FNR, and pairwise metrics treat these labels as correct.
  • domain assumption Prompt-generation examples from SVEN in NL(S2) do not leak evaluation signal.
    Section III-B uses SVEN validation pairs to build instructions, while Section IV-C evaluates on SVEN; the paper does not document a split that excludes those pairs from the tested set.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can LLM Prompting Serve as a Proxy for Static Analysis in Vulnerability Detection." pith.science (2026). https://pith.science/paper/WP4MUD2M

@misc{pith2026241212039,
  author       = {Pith},
  title        = {Pith review of: Can LLM Prompting Serve as a Proxy for Static Analysis in Vulnerability Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WP4MUD2M}},
  note         = {Machine review of arXiv:2412.12039}
}
read the original abstract

Despite their remarkable success, large language models (LLMs) have shown limited ability on safety-critical code tasks such as vulnerability detection. Typically, static analysis (SA) tools, like CodeQL, CodeGuru Security, etc., are used for vulnerability detection. SA relies on predefined, manually-crafted rules for flagging various vulnerabilities. Thus, effectiveness of SA in detecting vulnerabilities depends on human experts and is known to report high error rates. In this study we investigate whether LLM prompting can be an effective alternative to these static analyzers in the partial code setting. We propose prompting strategies that integrate natural language instructions of vulnerabilities with contrastive chain-of-thought reasoning, augmented using contrastive samples from a synthetic dataset. Our findings demonstrate that security-aware prompting techniques can be effective alternatives to the laborious, hand-crafted rules of static analyzers, which often result in high false negative rates in the partial code setting. When leveraging SOTA reasoning models such as DeepSeek-R1, each of our prompting strategies exceeds the static analyzer baseline, with the best strategies improving accuracy by as much as 31.6%, F1-scores by 71.7%, pairwise accuracies by 60.4%, and reducing FNR by as much as 37.6%.

Figures

Figures reproduced from arXiv: 2412.12039 by the authors.

Figure 1
Figure 1. Basic Prompt (B) B. Natural Language Instructions In the natural language instruction setup, we add to [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Prompting strategy with NL instructions 5https://cwe.mitre.org/about/index.html [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. NL+CoT Prompt. An example prompt provided by us during the prompt construction. In the second phase, we prompt the LLM (ChatGPT-4) to generate a set of instructions specific to each CWE. The process begins with a predefined prompt, which includes the CWE-ID and a consistent format for instruction generation. For example, the prompt for CWE-78: OS Command Injection is structured as follows: Phase 2 Prompt You are a s… view at source ↗
Figures from the paper (14 more)
Figure 5
Figure 5. Figure 5: Overall performance of our techniques across all [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 4
Figure 4. Figure 4: Heatmap illustrating how often each prompting strategy [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 6
Figure 6. Figure 6: CWE-078 — OpenAI o1 [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: CWE-078 — DeepSeek-R1 [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: CWE-078 — Claude 3.5 Sonnet v2 [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: CWE-190 — OpenAI o1 [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: CWE-190 — DeepSeek-R1 [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: CWE-190 — Claude 3.5 Sonnet v2 [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: CWE-476 — OpenAI o1 [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 13
Figure 13. Figure 13: CWE-476 — DeepSeek-R1 [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]
Figure 14
Figure 14. Figure 14: CWE-476 — Claude 3.5 Sonnet v2 [PITH_FULL_IMAGE:figures/full_fig_p021_14.png]
Figure 15
Figure 15. Figure 15: CWE-416 — OpenAI o1 [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]
Figure 16
Figure 16. Figure 16: CWE-416 — DeepSeek-R1 [PITH_FULL_IMAGE:figures/full_fig_p022_16.png]
Figure 17
Figure 17. Figure 17: CWE-416 — Claude 3.5 Sonnet v2 [PITH_FULL_IMAGE:figures/full_fig_p022_17.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 22 canonical work pages

  1. [1]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, and F. A. et al., “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  2. [2]

    Starcoder: May the source be with you!

    R. Li, L. B. Allal, Y . Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, and J. C. et al., “Starcoder: May the source be with you!” arXiv preprint arXiv:2305.06161, 2023

    arXiv 2023

  3. [3]

    Deep learning based vulnerability detection: Are we there yet?

    S. Chakraborty, R. Krishna, Y . Ding, and B. Ray, “Deep learning based vulnerability detection: Are we there yet?”IEEE Transactions on Software Engineering, 2021

    work page 2021

  4. [4]

    Detecting false alarms from automatic static analysis tools: How far are we?

    H. J. Kang, K. L. Aw, and D. Lo, “Detecting false alarms from automatic static analysis tools: How far are we?” inProceedings of the 44th International Conference on Software Engineering (ICSE ’22). ACM, May 2022, pp. 698–709. [Online]. Available: http://dx.doi.org/10.1145/3510003.3510214

    arXiv 2022

  5. [5]

    Stack Overflow in Github: Any Snippets There?

    D. Yang, P. Martins, V . Saini, and C. Lopes, “Stack overflow in github: Any snippets there?” arXiv preprint arXiv:1705.01198, 2017. [Online]. Available: https://arxiv.org/abs/1705.01198

    work page Pith review arXiv 2017

  6. [6]

    Beyond dependencies: The role of copy-based reuse in open source software development,

    M. Jahanshahi, D. Reid, and A. Mockus, “Beyond dependencies: The role of copy-based reuse in open source software development,”ACM Transactions on Software Engineering and Methodology, January 2025. [Online]. Available: http://dx.doi.org/10.1145/3715907 11https://github.com/SoftwEngLab/partial-code-analysis-llm-cwe 12

    doi:10.1145/3715907 2025

  7. [7]

    How do developers reuse stackoverflow answers in their github projects?

    J. Chen, Y . Zhao, and N. Meng, “How do developers reuse stackoverflow answers in their github projects?” inProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering Work- shops, 2024, pp. 146–155

    work page 2024

  8. [8]

    Amazon Nova AI Challenge -- Trusted AI: Advancing secure, AI-assisted software development

    S. Sahai, P. Goyal, M. Johnston, A. Gottardi, Y . Lu, L. Hu, L. Dai, S. Liu, S. Sagi, H. Shiet al., “Amazon nova ai challenge–trusted ai: Advancing secure, ai-assisted software development,”arXiv preprint arXiv:2508.10108, 2025

    work page Pith review arXiv 2025

Show all 42 references
  1. [9]

    AI is already writing about 30% of code at Microsoft and Google-here’s what it means for software engineers,

    MSN, “AI is already writing about 30% of code at Microsoft and Google-here’s what it means for software engineers,” https://www.msn.com/en-us/money/news/ai-is-already-writing-about- 30-of-code-at-microsoft-and-google-here-s-what-it-means-for-software- engineers/ar-AA1DWyrq, April 2025

    2025

  2. [10]

    Vulnerabilities in ai code generators: Exploring targeted data poisoning attacks,

    D. Cotroneo, C. Improta, P. Liguori, and R. Natella, “Vulnerabilities in ai code generators: Exploring targeted data poisoning attacks,” inPro- ceedings of the 32nd IEEE/ACM International Conference on Program Comprehension, 2024, pp. 280–292

    2024

  3. [11]

    Just another copy and paste? comparing the security vulnerabilities of chatgpt generated code and stackoverflow answers,

    S. Hamer, M. d’Amorim, and L. Williams, “Just another copy and paste? comparing the security vulnerabilities of chatgpt generated code and stackoverflow answers,” in2024 IEEE Security and Privacy Workshops (SPW). IEEE, 2024, pp. 87–94

    2024

  4. [12]

    How secure is code generated by chatgpt?

    R. Khoury, A. R. Avila, J. Brunelle, and B. M. Camara, “How secure is code generated by chatgpt?” in2023 IEEE international conference on systems, man, and cybernetics (SMC). IEEE, 2023, pp. 2445–2451

    2023

  5. [13]

    Asleep at the keyboard? assessing the security of github copilot’s code con- tributions,

    H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri, “Asleep at the keyboard? assessing the security of github copilot’s code con- tributions,”Communications of the ACM, vol. 68, no. 2, pp. 96–105, 2025

    2025

  6. [14]

    Devign: Effective vul- nerability identification by learning comprehensive program semantics via graph neural networks,

    Y . Zhou, S. Liu, J. Siow, X. Du, and Y . Liu, “Devign: Effective vul- nerability identification by learning comprehensive program semantics via graph neural networks,”Advances in Neural Information Processing Systems, vol. 32, 2019

    2019

  7. [15]

    Vulnerability detection with fine- grained interpretations,

    Y . Li, S. Wang, and T. N. Nguyen, “Vulnerability detection with fine- grained interpretations,” inProceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2021, pp. 292–303

    2021

  8. [16]

    Linevd: Statement-level vulnerability detection using graph neural networks,

    D. Hin, A. Kan, H. Chen, and M. A. Babar, “Linevd: Statement-level vulnerability detection using graph neural networks,” inProceedings of the 19th International Conference on Mining Software Repositories, 2022, pp. 596–607

    2022

  9. [17]

    A comprehensive study of the capabilities of large language models for vulnerability detection,

    B. Steenhoek, M. M. Rahman, M. K. Roy, M. S. Alam, E. T. Barr, and W. Le, “A comprehensive study of the capabilities of large language models for vulnerability detection,” arXiv preprint arXiv:2403.17218, 2024

    2024 arXiv

  10. [18]

    Vulnerability detection with code language models: How far are we?

    Y . Ding, Y . Fu, O. Ibrahim, C. Sitawarin, X. Chen, B. Alomair, D. Wagner, B. Ray, and Y . Chen, “Vulnerability detection with code language models: How far are we?” arXiv preprint arXiv:2403.18624, 2024

    2024 arXiv

  11. [19]

    Llms cannot reliably identify and reason about security vulnerabilities (yet?): A comprehensive evaluation, framework, and benchmarks,

    S. Ullah, M. Han, S. Pujar, H. Pearce, A. K. Coskun, and G. Stringhini, “Llms cannot reliably identify and reason about security vulnerabilities (yet?): A comprehensive evaluation, framework, and benchmarks,” in Proceedings of the IEEE Symposium on Security and Privacy, 2024

    2024

  12. [20]

    Understanding the effectiveness of large language models in detecting security vulnerabilities,

    A. Khare, S. Dutta, Z. Li, A. Solko-Breslin, R. Alur, and M. Naik, “Understanding the effectiveness of large language models in detecting security vulnerabilities,” arXiv preprint arXiv:2311.16169, 2023

    2023 arXiv

  13. [21]

    Finetuning large language models for vulnerability detection,

    A. Shestov, R. Levichev, R. Mussabayev, E. Maslov, P. Zadorozhny, A. Cheshkov, R. Mussabayev, A. Toleu, G. Tolegen, and A. Krasso- vitskiy, “Finetuning large language models for vulnerability detection,” IEEE Access, 2025

    2025

  14. [22]

    Grace: Empowering llm-based software vulnerability detection with graph structure and in- context learning,

    G. Lu, X. Ju, X. Chen, W. Pei, and Z. Cai, “Grace: Empowering llm-based software vulnerability detection with graph structure and in- context learning,”Journal of Systems and Software, vol. 212, p. 112031, 2024

    2024

  15. [23]

    Large language models for code: Security hardening and adversarial testing,

    J. He and M. Vechev, “Large language models for code: Security hardening and adversarial testing,” inProceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, 2023, pp. 1865–1879

    2023

  16. [24]

    Cvefixes: Automated collec- tion of vulnerabilities and their fixes from open-source software,

    G. Bhandari, A. Naseer, and L. Moonen, “Cvefixes: Automated collec- tion of vulnerabilities and their fixes from open-source software,” in Proceedings of the 17th International Conference on Predictive Models and Data Analytics in Software Engineering, 2021, pp. 30–39

    2021

  17. [25]

    Symbolic security predicates: Hunt program weaknesses,

    A. Vishnyakov, V . Logunova, E. Kobrin, D. Kuts, D. Parygina, and A. Fedotov, “Symbolic security predicates: Hunt program weaknesses,” in2021 Ivannikov ISPRAS Open Conference (ISPRAS). IEEE, 2021. [Online]. Available: https://arxiv.org/abs/2111.05770

    2021 arXiv

  18. [26]

    Amazon codeguru security,

    Amazon Web Services, Inc., “Amazon codeguru security,” https://aws. amazon.com/codeguru/, 2025

    2025

  19. [27]

    Semgrep: Lightweight static analysis for many languages,

    r2c, Inc., “Semgrep: Lightweight static analysis for many languages,” https://semgrep.dev, 2025

    2025

  20. [28]

    Codeql: Semantic code analysis engine,

    GitHub, “Codeql: Semantic code analysis engine,” https://codeql.github. com/, 2025

    2025

  21. [29]

    Sonarqube: Continuous inspection of code quality,

    SonarSource, “Sonarqube: Continuous inspection of code quality,” https: //www.sonarqube.org/, 2025

    2025

  22. [30]

    Automated vulnerability detection in source code using deep representation learning,

    R. Russell, L. Kim, L. Hamilton, T. Lazovich, J. Harer, O. Ozdemir, P. Ellingwood, and M. McConley, “Automated vulnerability detection in source code using deep representation learning,” inProceedings of the 2018 17th IEEE International Conference on Machine Learning and Appli...

    2018

  23. [31]

    Codebert: A pre-trained model for programming and natural languages,

    Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, and D. J. et al., “Codebert: A pre-trained model for programming and natural languages,” arXiv preprint arXiv:2002.08155, 2020

    2002 arXiv

  24. [32]

    Linevul: A transformer-based line- level vulnerability prediction,

    M. Fu and C. Tantithamthavorn, “Linevul: A transformer-based line- level vulnerability prediction,” inProceedings of the 19th International Conference on Mining Software Repositories, 2022, pp. 608–620

    2022

  25. [33]

    Unixcoder: Unified cross-modal pre-training for code representation,

    D. Guo, S. Lu, N. Duan, Y . Wang, M. Zhou, and J. Yin, “Unixcoder: Unified cross-modal pre-training for code representation,” arXiv preprint arXiv:2203.03850, 2022

    2022 arXiv

  26. [34]

    Generalization-enhanced code vulnerability detection via multi-task instruction fine-tuning,

    X. Du, M. Wen, J. Zhu, Z. Xie, B. Ji, H. Liu, X. Shi, and H. Jin, “Generalization-enhanced code vulnerability detection via multi-task instruction fine-tuning,” 2024. [Online]. Available: https://arxiv.org/abs/2406.03718

    2024 arXiv

  27. [35]

    Llmxcpg: Context-aware vulnerability detection through code property graph-guided large language models,

    A. Lekssays, H. Mouhcine, K. Tran, T. Yu, and I. Khalil, “Llmxcpg: Context-aware vulnerability detection through code property graph-guided large language models,” 2025. [Online]. Available: https://arxiv.org/abs/2507.16585

    2025 arXiv

  28. [36]

    How far have we gone in vulnerability detection using large language models,

    Z. Gao, H. Wang, Y . Zhou, W. Zhu, and C. Zhang, “How far have we gone in vulnerability detection using large language models,” arXiv preprint arXiv:2311.12420, 2023

    2023 arXiv

  29. [37]

    Realvul: Can we detect vulnerabilities in web applications with llm?

    D. Cao, Y . Liao, and X. Shang, “Realvul: Can we detect vulnerabilities in web applications with llm?” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, Florida, USA, November 2024, pp. 8268–8282

    2024

  30. [38]

    From generalist to specialist: Exploring cwe-specific vulnerability detection,

    S. A. Atiiq, C. Gehrmann, K. Dahl ´en, and K. Khalil, “From generalist to specialist: Exploring cwe-specific vulnerability detection,” 2024. [Online]. Available: https://arxiv.org/abs/2408.02329

    2024 arXiv

  31. [39]

    One-for-all does not work! enhancing vulnerability detection by mixture-of-experts (moe),

    X. Yang, S. Wang, J. Zhou, and W. Zhu, “One-for-all does not work! enhancing vulnerability detection by mixture-of-experts (moe),”Proc. ACM Softw. Eng., vol. 2, no. FSE, Jun. 2025. [Online]. Available: https://doi.org/10.1145/3715736

    2025 doi

  32. [40]

    Applying contrastive learning to code vulnerability type classification,

    C. Ji, S. Yang, H. Sun, and Y . Zhang, “Applying contrastive learning to code vulnerability type classification,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y . Al-Onaizan, M. Bansal, and Y .-N. Chen, Eds. Miami, Florida, USA: Ass...

    2024

  33. [41]

    Chatgpt for vulnerability detection, classification, and repair: How far are we?

    M. Fu, C. Tantithamthavorn, V . Nguyen, and T. Le, “Chatgpt for vulnerability detection, classification, and repair: How far are we?” arXiv preprint arXiv:2310.09810, 2023

    2023 arXiv

  34. [42]

    Llm-assisted static analysis for detecting security vulnerabilities,

    Z. Li, S. Dutta, and M. Naik, “Llm-assisted static analysis for detecting security vulnerabilities,” arXiv preprint arXiv:2405.17238, 2024. X. APPENDIX A. NL+CoT (S0) Sample Prompt for (CWE-190) You are a security expert. You will analyze new examples for vulnerabilities. Use ...

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.