{"id":"06886a2b-1856-4e6e-bc48-15ab52e361e1","arxiv_id":"2412.12039","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"CWE-specific natural language instructions with contrastive chain-of-thought prompts let LLMs beat default static analyzers on partial-code vulnerability detection for four CWEs.","lead":"This paper tests whether telling an LLM what a vulnerability looks like, plus showing it good and bad code examples, lets it find security flaws in snippets better than static analysis tools. It reports large gains on four vulnerability types, but usually only for the best prompt chosen per type.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Baseline fairness is the load-bearing issue: the static analyzers are run outside their intended whole-program mode, so their near-zero recall may be an artifact of the snippet harness; a full-context rerun would settle whether the headline LLM gains are real.","rationale":"The reader's weakest assumption identifies the same load-bearing concern, and I agree with it. The central empirical claim is a comparison: LLM prompting outperforms static analysis in the partial-code setting. The comparison is only meaningful if the static-analyzer baseline is representative of what those tools can do. Running CodeQL without a database, SonarQube without project context, and all tools on isolated snippets removes the interprocedural information that these analyzers depend on, so the observed recall near zero may reflect the evaluation harness rather than inherent limitations of static analysis. I considered two other candidate concerns: the post-hoc selection of the best prompting strategy per CWE, and the use of SVEN validation examples in the NL (S2) prompt-generation setting. These are real secondary risks, but the baseline-fairness issue is more load-bearing because it affects every headline delta, not just one CWE or one strategy. The proposed full-context rerun is a discrete, feasible check: if static analyzers still show near-zero recall and pairwise accuracy on full repositories, the paper's central comparison holds in the partial-code setting; if not, the claim must be narrowed to out-of-the-box static analyzers on isolated snippets. The verdict should remain CONDITIONAL because the concern is addressable and the paper otherwise provides a substantial empirical study with reproducible prompts and an explicit evaluation design.","tokens_in":20423,"tokens_out":6128,"duration_ms":58103,"concrete_test":"Select all 520 SVEN samples, use SVEN metadata to check out the original vulnerable and fixed repository commits, build CodeQL databases and run the default CWE-relevant security suites for all four CWEs; run Semgrep, CodeGuru, and SonarQube with their standard security rules on the same full projects. Recompute recall, F1, and pairwise accuracy exactly as in Table V. If full-context pairwise accuracy exceeds roughly 10% or recall exceeds roughly 30% for any CWE, the Table V baseline is not a faithful proxy and the reported deltas must be recomputed against the full-context baseline.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"All headline improvements are measured against the static-analyzer baseline constructed in Section IV-B: CodeGuru, Semgrep, SonarQube, and CodeQL are run with default settings on isolated partial snippets, using only queries \"capable of operating on partial code.\" This is not the operating regime of these tools. CodeQL and SonarQube rely on build/project context and interprocedural dataflow; CodeQL is only reported for CWE-078 in Table V, and every static-analyzer row shows recall at or below 6.3% with pairwise accuracy at or below 3%. Under those conditions the baseline is effectively a near-zero detector, so large relative gains (accuracy +31.6%, F1 +71.7%, pairwise +60.4%) can occur even when the LLM is only moderately competent. The paper's conclusion that prompting can \"serve as a proxy for static analysis\" therefore rests on a comparison that may measure the snippet harness rather than static-analysis capability. The concern is not that the partial-code setting is irrelevant; it is that the baseline tools were not given a fair chance to do what they are designed to do. The fix is to validate the baseline by running the same tools on the full functions or repositories from which the SVEN snippets were drawn, with their standard CWE-relevant security query suites enabled.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a set of CWE-specific prompting strategies for LLM-based vulnerability detection on partial code snippets. The strategies combine a basic prompt with natural-language vulnerability instructions, chain-of-thought reasoning, and contrastive synthetic examples from the Juliet C/C++ suite. The authors evaluate the strategies on SVEN (four CWEs, 520 samples) and CVEFixes (1,784 samples) using o1, Claude 3.5 Sonnet v2, DeepSeek-R1, and several other models, and compare them against CodeGuru, Semgrep, SonarQube, and CodeQL run with default settings on the same partial snippets. The paper reports that for each CWE some prompting strategy exceeds the static-analyzer baseline, with pairwise-accuracy gains up to 60.4 points, and also compares against Khare et al. on CVEFixes. The central claim is that LLM prompting can serve as a proxy for static analysis in the partial-code setting.","tokens_in":20645,"tokens_out":5000,"duration_ms":42297,"significance":"If the comparison were fair and the results reproducible, this would be a practically useful result: partial-code review is common in pull requests and AI-assisted coding, and a drop-in prompt-based detector that beats static analyzers at snippet level would be valuable. The paper's strengths include the use of a cleaner real-world dataset (SVEN), a clearly described multi-phase prompt-generation pipeline with concrete templates, a manual error analysis of 414 responses, a robustness check on tag-based formatting, a cost analysis, and a publicly announced replication package. However, the headline comparative claim depends on the fairness of the static-analyzer baseline and on the integrity of the SVEN split for instruction generation; both are currently under-specified. In addition, the per-CWE best results are selected post hoc from many strategy-model combinations without significance testing, so the quantitative claim of \"significantly improves\" is not yet supported.","major_comments":[{"comment":"The static-analyzer baseline is constructed by running CodeGuru, Semgrep, SonarQube, and CodeQL with default settings on isolated partial snippets and applying only queries \"capable of operating on partial code.\" This is not the intended operating regime for CodeQL and SonarQube, which rely on build/project context and interprocedural dataflow; CodeQL is in fact reported only for CWE-078. Consequently, every static-analyzer row in Table V has recall below 6.3% and pairwise accuracy below 3%, so the baseline behaves like a near-zero detector. The headline improvements in Table IV (F1 +71.7, pairwise +60.4, accuracy +31.6) are measured against a baseline that may be crippled by the snippet harness rather than a fair representative of static-analysis capability. Please re-run the same tools on the full functions or repositories from which the SVEN snippets are drawn, using their standard CWE-relevant security query suites, and report whether the conclusions change; if the full-context setting is infeasible, state this explicitly and restrict the claims to the partial-code setting without the \"proxy for static analysis\" phrasing.","section":"§IV-B, Table V"},{"comment":"The NL(S2) instructions are generated by prompting the LLM with three vulnerable/fixed pairs \"extracted from the SVEN validation set for that particular CWE,\" while the evaluation in Section V is reported on SVEN. The paper never documents a split that excludes these three pairs from the evaluation set. If any of those pairs appear in the test data, the NL(S2) setting, which is the top-performing setting for CWE-416 in Table IV, is evaluated with instruction-generation examples that are also in the test set, inflating its results. Please specify the exact SVEN split, confirm that the three selected pairs per CWE are excluded from all evaluation, and, if they are not, re-run the affected results and adjust the conclusions.","section":"§III-B, §IV-C"},{"comment":"For each CWE, the headline winner is selected post hoc as the best combination of setting and model (e.g., NL (S1) with o1 for CWE-078, NL+CoT (S0) with DeepSeek-R1 for CWE-190), and the table reports only point estimates averaged over three trials with no confidence intervals or significance tests. With CWE-level sample sizes of 76 to 190 and pairwise-accuracy differences as small as +8.8 points (CWE-190), the claim that the best strategy \"significantly improves\" over the baselines is not supported by the data. Please add per-strategy variance and, at minimum, a paired bootstrap or permutation test against the vanilla prompt and static-analyzer baselines, or explicitly soften the \"significantly improves\" language to \"improves in this sample.\"","section":"Table IV, §V-A"}],"minor_comments":[{"comment":"The text \"Table VII-B provides the average token count\" appears to refer to the cost-analysis table, which is labeled Table IX; please fix the cross-reference.","section":"§VII-B"},{"comment":"The caption uses \"S1\" and \"S2\" without defining them; clarify that S1 is the basic prompt with the verbosity instruction and S2 is the tag-based format without the verbosity instruction.","section":"Table VIII"},{"comment":"The phase description says the authors \"manually select pairwise samples\" but also \"randomly select one such scenario\"; please clarify whether the selection is manual, random, or a combination.","section":"§III-D, Phase 1"},{"comment":"The FNR/FPR columns mix two metrics into a single delta value in the \"FNR / FPR\" header; separate the FNR and FPR columns or precisely define what the single reported number represents.","section":"Table IV"},{"comment":"The appendix prompt text contains stray characters (e.g., \"if (data ¡ (LLONG MAX/2))\"); please proofread the appendix for encoding and typographic errors.","section":"Appendix"},{"comment":"The caption does not specify what the plotted values are (accuracy, F1, or pairwise accuracy); clarify the caption and legend so the figure is self-contained.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an important problem and the prompting pipeline is clearly presented, but the current evidence does not support the strong comparative claim in the abstract. The static-analyzer baseline is run in a snippet-only mode that is not the intended operating regime for most of the tools, and the SVEN split for NL(S2) instruction generation is not documented. I recommend the editor ask for (1) a full-context static-analyzer rerun or an explicit scope restriction, (2) a documented split excluding the instruction-generation pairs, and (3) significance testing or softened language for the per-CWE wins. The replication package should also be checked to confirm the code and data support the reported tables."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this paper is a serious empirical attempt at a real problem, but its headline claims rest on a static-analyzer baseline that may be too weak to support them. The idea—CWE-specific natural-language instructions plus contrastive chain-of-thought from synthetic pairs—is not entirely new, but the combination and the partial-code evaluation on SVEN and CVEFixes are a genuine addition, and the manual review of 414 LLM responses is the kind of qualitative work that gives the numbers texture.\n\nWhat it does well: the prompting strategies are clearly specified, the dataset filtering is transparent, and the replication package is a plus. The comparison to Khare et al. on CVEFixes is useful, and the gains there (up to 12% accuracy) are more believable than the SVEN-vs-static-analyzer numbers.\n\nThe soft spots are exactly where the reader put them. The static analyzers are run on isolated snippets with default settings and only partial-code queries; for CodeQL and SonarQube that is not their operating regime. Their near-zero recall and pairwise accuracy in Table V suggest the baseline is effectively a no-detector for this harness, so the 31.6% accuracy and 71.7% F1 improvements over static analysis could be an artifact of the snippet setting rather than a real weakness in static analysis. The paper should either rerun the analyzers on full functions or repositories, or clearly delimit the claim to 'default, snippet-mode static analysis.' The post hoc selection of the best strategy and model per CWE is also a concern; no significance tests or error bars are reported. Finally, NL(S2) generates instructions from three SVEN validation pairs and then evaluates on SVEN without documenting a split. That is a minor leak but should be fixed.\n\nNone of this kills the paper. The central finding—that some LLM prompting strategies can beat default snippet-mode static analyzers on these four CWEs—is plausible and worth reporting, but the magnitude of the headline gains is not yet established.\n\nRecommendation: send to peer review, but the reviewers should push for a fairer static-analyzer baseline and a non-selective strategy evaluation.","headline":"A serious empirical exploration of CWE-specific LLM prompting for partial-code vulnerability detection, but the headline gains over static analyzers rest on a baseline that may be crippled by the snippet setting.","tokens_in":21216,"tokens_out":2309,"would_cite":false,"duration_ms":20900,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Security-aware LLM prompting, which augments CWE descriptions with contrastive vulnerable/fixed examples, outperforms default static analysis on partial-code vulnerability detection across four CWE types, lifting pairwise accuracy by up…","keywords":["vulnerability detection","static analysis","LLM prompting","chain-of-thought","partial code","CWE","contrastive examples","DeepSeek-R1"],"falsifier":"Take the same SVEN and CVEFixes snippets and run CodeGuru, Semgrep, SonarQube, and CodeQL after restoring the full enclosing function and its project or build context (or with their intended code-analysis configurations rather than defaults), then compare recall and pairwise accuracy with the reported LLM numbers. If any tool's pairwise accuracy rises from near zero to the 26–60% range, the claimed advantage is an artifact of giving the analyzers a crippled setup.","tokens_in":20216,"feed_emoji":"🛡️","tokens_out":9473,"duration_ms":80116,"temperature":0.7,"pith_summary":"This paper tries to establish that carefully designed LLM prompting can replace, or at least complement, hand-written static-analysis rules for vulnerability detection when only partial code is available. The authors build CWE-specific prompts that pair natural-language descriptions of a vulnerability class with contrastive chain-of-thought examples (a vulnerable snippet and its fixed counterpart from the synthetic Juliet suite), then test these prompts on real-world samples from SVEN and CVEFixes across four CWE categories. Their central evidence is a head-to-head comparison with four static analyzers run with default settings on the same partial-code snippets: with the reasoning model DeepSeek-R1, every prompting strategy outperforms the static-analyzer baseline, and the best strategies improve accuracy by up to 31.6 percentage points, F1 by up to 71.7 percentage points, and pairwise accuracy (the fraction of vulnerable/fixed pairs where both snippets are correctly labeled) by up to 60.4 percentage points while reducing false negatives by up to 37.6 percentage points. If the comparison holds, snippet-level code review—common in pull requests, code reuse, and AI-assisted generation—can be made substantially more effective without training a model or building the project.","feed_headline":"LLM prompts beat static analyzers on snippet vulnerabilities","feed_subtitle":"Best strategy lifts F1 by 71.7 points and pairwise accuracy by 60.4 points on partial code.","key_machinery":"The load-bearing mechanism is a contrastive chain-of-thought prompt: a CWE-specific instruction set phrased as checks, followed by a vulnerable example with reasoning for why it fails the checks, its fixed counterpart with reasoning for why it passes, and then the new snippet to be classified. The instruction sets come from four sources—the bare CWE name, LLM-generated instructions, LLM-generated instructions from few-shot samples, and human-authored security documentation—and the contrastive examples come from synthetic Juliet C/C++ vulnerable/fixed pairs. The mechanism's work is to make vulnerability detection a guided reasoning task rather than a raw binary classification: it gives the model a concrete pattern to match, a positive and negative reference, and a template that forces per-instruction justification before the final verdict.","core_discovery":"On the paper's own terms, the discovery is that domain-informed prompting can elicit reliable vulnerability judgments from LLMs even in the hard partial-code setting where hand-written static-analysis rules fail. The paper evaluates prompts on 520 SVEN samples and 1,784 CVEFixes samples across CWE-078 (OS command injection), CWE-190 (integer overflow), CWE-416 (use after free), and CWE-476 (null pointer dereference). Static analyzers in the same setting score near-zero pairwise accuracy—the highest is about 3%—and often detect no true vulnerabilities at all: across CWE-190 and CWE-416 every tool reports zero true positives. Against that baseline, the paper finds that for every CWE there is at least one prompting strategy that outperforms all four analyzers, with the best gains coming from reasoning models such as DeepSeek-R1. The authors conclude that security-property-aware prompting can serve as a complementary proxy for static-analysis rules in snippet review, not that it replaces deep program analysis in full-project settings.","pith_inferences":["Editorial inference: because the static analyzers were run with default settings on isolated snippets, the headline comparison does not show that LLMs are inherently better analysts—only that they beat default snippet-mode configurations. A hybrid pipeline that uses LLM prompts to triage snippets and static analysis to confirm on full builds is the natural next experiment.","Editorial inference: the contrastive-prompt construction (instructions plus vulnerable/fixed pairs plus reasoning) is largely automatic once a synthetic suite exists, so scaling the approach to the rest of the CWE Top 25 is plausible; the paper itself only demonstrates four CWE classes.","Editorial inference: the manual response analysis suggests many misses come from missing context and selective reasoning, so prompting the model to enumerate all data-flow paths or inspect each sink individually before deciding is a testable way to push pairwise accuracy higher."],"forward_implications":["Partial-code vulnerability detection can be improved without model training or a full project build: for each CWE at least one prompting strategy outperforms every default static analyzer, so snippet review can be made substantially safer with prompt design alone.","The gains are not tied to one universal prompt; the best strategy varies by CWE and model (o1 with natural-language instructions for CWE-078, DeepSeek-R1 with generic chain-of-thought for CWE-190), so deployment needs per-CWE strategy selection.","The prompting advantage carries over to the noisier CVEFixes dataset, where accuracy improves by up to 12 percentage points over the prior GPT-3.5/GPT-4 baseline, suggesting the effect is not an artifact of the curated SVEN set.","Because static analyzers in this setting have extremely low recall, using these prompts as a complement would cut false negatives at review time, catching vulnerabilities that default SA rules miss."],"supporting_citations":[{"why":"Supplies the real-world SVEN dataset of manually labeled vulnerable/fixed pairs used as the primary evaluation set.","marker":"[23]"},{"why":"Supplies CVEFixes, the real-world dataset used for the comparison with prior LLM baselines on C/C++ samples.","marker":"[24]"},{"why":"Supplies the synthetic Juliet C/C++ vulnerable/fixed pairs used to build contrastive chain-of-thought examples.","marker":"[25]"},{"why":"One of the four static analyzers whose default partial-code results form the baseline that prompting must beat.","marker":"[26]"},{"why":"One of the four static analyzers providing the baseline.","marker":"[27]"},{"why":"One of the four static analyzers providing the baseline.","marker":"[28]"},{"why":"One of the four static analyzers providing the baseline.","marker":"[29]"},{"why":"Supplies the pairwise-accuracy metric and the argument that accuracy and F1 alone are misleading for imbalanced vulnerability detection.","marker":"[18]"},{"why":"Supplies the prior GPT-3.5/GPT-4 baseline on CVEFixes C/C++ that the paper's prompting strategies are compared against.","marker":"[20]"}],"fun_headline_variants":["LLM prompts outshine static analyzers on partial code","Prompting LLMs finds more vulnerabilities than static tools","Security-aware LLM prompts beat static analyzers on snippets","LLM prompting rivals static analysis for snippet flaws","For snippet review, LLM prompts surpass static rules"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that running the four static analyzers with default settings on isolated snippets is a fair proxy for static-analysis capability; if those tools are handicapped by the snippet setting and would do substantially better with build context or tuned rules, the reported gains mostly show the baseline is weak rather than showing LLMs are inherently better.","fun_headline_variants_meta":{"raw":{"variants":["LLM prompts outshine static analyzers on partial code","Prompting LLMs finds more vulnerabilities than static tools","Security-aware LLM prompts beat static analyzers on snippets","LLM prompting rivals static analysis for snippet flaws","For snippet review, LLM prompts surpass static rules"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00025,"raw_usage":{"total_tokens":1573,"prompt_tokens":987,"completion_tokens":586,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":509}},"tokens_in":603,"tokens_out":586,"duration_ms":5786,"temperature":1.0,"reasoning_tokens":509,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:20:29.058562+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same SVEN and CVEFixes snippets and run CodeGuru, Semgrep, SonarQube, and CodeQL after restoring the full enclosing function and its project or build context (or with their intended code-analysis configurations rather than defaults), then compare recall and pairwise accuracy with the reported LLM numbers. If any tool's pairwise accuracy rises from near zero to the 26–60% range, the claimed advantage is an artifact of giving the analyzers a crippled setup.","supporting_citations":[{"cited_title":"Cvefixes: Automated collec- tion of vulnerabilities and their fixes from open-source software,","cited_arxiv_id":null,"evidence_quote":"Supplies CVEFixes, the real-world dataset used for the comparison with prior LLM baselines on C/C++ samples."},{"cited_title":"Symbolic Security Predicates: Hunt Program Weaknesses","cited_arxiv_id":"2111.05770","evidence_quote":"Supplies the synthetic Juliet C/C++ vulnerable/fixed pairs used to build contrastive chain-of-thought examples."},{"cited_title":"Amazon codeguru security,","cited_arxiv_id":null,"evidence_quote":"One of the four static analyzers whose default partial-code results form the baseline that prompting must beat."},{"cited_title":"Semgrep: Lightweight static analysis for many languages,","cited_arxiv_id":null,"evidence_quote":"One of the four static analyzers providing the baseline."},{"cited_title":"Codeql: Semantic code analysis engine,","cited_arxiv_id":null,"evidence_quote":"One of the four static analyzers providing the baseline."},{"cited_title":"Sonarqube: Continuous inspection of code quality,","cited_arxiv_id":null,"evidence_quote":"One of the four static analyzers providing the baseline."}],"review_version":1}