{"id":"f12a01fc-10b6-4738-904f-383d8b78a772","arxiv_id":"2607.25225","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Sector-conditioned prompt framing does not significantly change static-analysis vulnerability rates in AI-generated code; the apparent effect is a composition artifact of two CWE categories.","lead":"This paper tests whether adding critical-infrastructure context to coding prompts changes how often LLMs produce insecure code. It finds no significant sector-specific effect and shows model choice matters more than prompt framing.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The composition story—that the apparent protective drift is a CWE-specific effect—is untested because template and CWE are perfectly confounded; leave-one-CWE-out is leave-one-template-out, and a 100%-baseline CWE-502 template can only drift downward.","rationale":"The reader's weakest assumption is exactly the load-bearing concern I identify: one hand-authored template per CWE confounds template identity with vulnerability class. I agree with that identification. The concern matters because the paper's strongest claim is not merely 'we failed to reject the null' but a substantive explanation of why the null arose—that the apparent protective effect is a composition artifact of CWE-502 and CWE-22. That explanation drives both the discussion (Section 7.2) and the practical takeaway that model selection, not prompt framing, is the lever. Because the two categories that carry the entire negative drift are extreme templates (100% and 16% baseline rates, n=25 per category), the magnitudes and even the signs of the per-CWE effects are highly sensitive to the chosen template. The mixed-model conditional result—the only statistically significant condition effect in the paper (OR 0.38)—is also carried by these same two templates, so it inherits the same confound. The paper honestly lists Limitation 1, but does not let that limitation constrain the CWE-level conclusions. Therefore I would not reject the paper; the pooled null and model-selection findings are well supported. However, I would condition acceptance on either (a) adding a multi-template robustness study for the event-bearing CWEs, or (b) revising the conclusions to state that the composition effect is template-bound, not CWE-bound. This is a moderate adjustment: it does not overturn the no-sector-drift null, but it changes the strength and scope of the central explanatory claim.","tokens_in":16301,"tokens_out":7257,"duration_ms":78302,"concrete_test":"Add a robustness arm with at least 3–5 independently authored baseline/industry template pairs for each event-bearing CWE (CWE-502, CWE-22, and CWE-78 as a control), keeping the rest of the SecDrift protocol fixed (6 models, 5 replicates, same matched-baseline logic). Recompute Table 6 row (c), the leave-one-CWE-out table, and the mixed-model/permutation condition estimates on this multi-template corpus. If the −11.7pp CWE-502 and −14.6pp CWE-22 drifts are not consistently reproduced—for example, any template shows a significantly different baseline flag rate or the pooled conditional odds ratio changes sign or loses significance—then the composition artifact is a template effect, not a CWE effect, and the paper must be reframed as template-specific rather than CWE-specific.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central explanatory claim that the pooled null is a composition artifact of CWE-502 and CWE-22 (Section 6.1.1, Table 6) rests entirely on a design in which each CWE category is represented by exactly one hand-authored baseline template (Section 7.4, Limitation 1). Thus template and CWE are perfectly confounded; Table 7's leave-one-CWE-out analysis is literally a leave-one-template-out analysis. The two event-bearing categories happen to be extreme templates: CWE-502 baselines are 100% flagged—a ceiling, so drift can only be non-positive—and CWE-22 baselines are 16% on 25 samples. The mixed-model conditional effect (OR 0.38, CrI [0.28, 0.51]) and the CWE-stratified permutation finding (p = 0.044) are likewise driven by these same two templates. If a different CWE-502 template were less saturated, or a different CWE-22 template more so, both the per-category drift magnitudes and the composition-bound 'genuine association' could vanish or reverse. The paper acknowledges this as a limitation, but the claim that the apparent protective effect is a 'genuine but composition-bound within-task association, not a general or sector-conditioned effect' is presented as a main finding, not merely a caveat. The pooled null itself is a statement about these prompts; the paper's headline generalization to CWE categories is not supported. This is the soft spot: the raw p-values are internally consistent, but the causal/explanatory decomposition that gives the paper its practical message depends on untested template invariance.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"SecDrift is a benchmark and empirical study asking whether conditioning code-generation prompts on CISA critical-infrastructure sector contexts changes the SAST-flag vulnerability rate of LLM-generated Python code. The design covers 7 LLMs (6 with analyzable code), 8 CISA sectors, 9 CWE categories, 5 replicates, totaling 5,355 evaluations, with a baseline, a matched-baseline condition that substitutes terminology only, and full industry prompts. The headline finding is a null result: the raw baseline-to-industry gap of -2.7pp (14.0% vs. 11.4%) is not significant (Fisher's exact p = 0.24), no sector shows significant drift after correction, and the terminology-only matched-baseline contrast is clearly null. The paper further argues that the apparent protective gap is a composition artifact concentrated in CWE-502 (insecure deserialization) and CWE-22 (path traversal), that a non-CISA placebo reproduces the industry rate, and that model selection is a more consistent and larger driver of vulnerability rates. The authors release the framework, prompts, code, verdicts, and analysis scripts.","tokens_in":16675,"tokens_out":4878,"duration_ms":50524,"significance":"If the result stands, SecDrift is a useful negative result for prompt-engineering security: it suggests that merely inserting sector terminology and operational context does not reliably alter SAST-flag rates, and that model choice is a more dependable lever. The paper's strengths are its reproducible benchmarking framework, the matched-baseline condition, the placebo control, the human adjudication of zero-detection categories, and the unusually large battery of sensitivity analyses (CWE exclusion, leave-one-out, mixed-effects, Firth, permutation, aggregation-rule, length adjustment). The empirical null for the pooled and per-sector comparisons is well supported. However, the paper's interpretive claim that the conditional effect is a 'genuine but composition-bound within-task association' of two CWE categories is not supported by the design, because each CWE is represented by exactly one hand-authored baseline template; the causal/explanatory decomposition is therefore weaker than the headline null.","major_comments":[{"comment":"The central explanatory claim—that the apparent protective drift is a composition artifact concentrated in CWE-502 and CWE-22, and that the conditional effect is a 'genuine but composition-bound within-task association'—cannot be identified from this design. Each CWE has exactly one hand-authored baseline template, so template and CWE are perfectly confounded; Table 7 is literally a leave-one-template-out analysis, not a leave-one-CWE-out analysis. The two event-bearing categories are extreme templates: CWE-502 baselines are at the 100% ceiling and CWE-22 baselines are 16% on 25 samples. The mixed-model OR (0.38) and the stratified permutation (p = 0.044) are driven by these same two templates. A different CWE-502 template, or a different CWE-22 template, could plausibly change both the per-category drift and the composition story. The paper acknowledges the confound in §7.4, but the abs","section":"§6.1.1, Table 7; §7.4 Limitation 1"},{"comment":"The mixed-effects model reports a significant industry condition coefficient (OR 0.38, CrI [0.28, 0.51]), and §6.6 calls this a 'task-specific association' that is 'genuine.' But the industry prompts are not a pure framing manipulation: they drop the explicit signature and worked example in all nine tasks, and change parameters, arity, or the operation itself for several tasks—including the two event-bearing templates (CWE-502 changes from single-format deserialization to multi-format load-by-extension; CWE-22 changes arity from 2 to 1 with a fixed base directory). Section 4.2 explicitly states that 'the raw baseline-vs-industry contrast confounds framing with interface/task changes' and that 'where a residual difference appears we cannot attribute it to framing.' The mixed-model finding is therefore a within-scenario statistical association that includes interface drift, not an estimate","section":"§4.2, Table 1; §6.6, Table 12"},{"comment":"The paper reports that CWE-502 has a 100% baseline flag rate (25/25) and an 88.3% industry rate, and notes that because the baseline is at the ceiling, the drift can only be non-positive and part of it is mechanical regression. Yet the CWE-502 result is then used as a main component of the 'composition artifact' explanation and of the conditional mixed-model effect. Given the ceiling and the interface change in the CWE-502 industry prompt, the −11.7pp movement cannot be attributed to sector framing or to a property of the CWE category. Please either report the matched-baseline rate for CWE-502 (which would at least separate terminology from interface change) or explicitly exclude CWE-502 from any causal reading, treating it only as a saturated-template descriptive result.","section":"§6.5, CWE-502 analysis"}],"minor_comments":[{"comment":"The abstract says the apparent effect is 'a genuine but composition-bound within-task association'; given the template-CWE confound, 'genuine' is too strong. Consider 'a statistically detectable but template-bound association' or similar.","section":"Abstract and §7.4"},{"comment":"The human adjudication is a valuable addition, but the paper should state more prominently that the 26.7% false-negative rate applies only to the six zero-detection categories, not to the whole corpus, and that the per-condition missed-vulnerability counts (8/29, 10/29, 6/32) are too unbalanced in sample size for the claim that 'the industry condition is not elevated—if anything lower' to carry inferential weight.","section":"§4.4, Table 3"},{"comment":"The paper correctly reports no sector as significant, but the minimum detectable effect of h≈0.33 at 80% power means the null can only rule out large effects. This is acknowledged in the text, but the phrase 'substantive null' in Section 6.2 could mislead readers; consider 'null relative to the effect sizes the design can detect.'","section":"§6.2, Table 8"},{"comment":"The placebo comparison reports that the non-CISA controls reproduce the CISA industry rate (p = 0.63). Since both conditions share the same interface drift, this is a comparison of two confounded conditions; the conclusion that the pooled pattern reflects 'generic framing specificity' is reasonable descriptively, but should be worded as a consistency check, not an independent identification of the source.","section":"§6.3, Table 9"},{"comment":"Reference [15] (OWASP Top 10) appears in the bibliography but does not seem to be cited in the text. Please add a citation or remove the reference.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The empirical null is well supported and the benchmark is a useful contribution. My concern is specifically with the causal/explanatory decomposition: the abstract and conclusions assert that the apparent effect is a 'genuine' CWE-category composition artifact, but the one-template-per-CWE design makes this claim untestable. This is fixable by softening the interpretation and/or adding one or two counterfactual templates for the event-bearing CWEs. I would be willing to accept after such a revision; I do not see a need to rerun the full experiment if the authors reframe the claims to match the design."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, the short version: this paper's central null—that sector-conditioned prompt framing does not produce detectable security drift—survives a careful read. The paper is well executed and unusually transparent. But the explanation for why the null looks the way it does, the \"composition artifact\" story, rests on a design where template and CWE are perfectly confounded, and that part of the paper is weaker than the abstract suggests.\n\nWhat's new: it's the first benchmark I know of that systematically varies implicit sector framing while holding the task fixed via a matched baseline, and it adds a non-CISA placebo. The protocol is reusable, the data and scripts are released, and the human adjudication of the zero-detection categories is a real contribution: it shows the SAST proxy misses 26.7% of true vulnerabilities concentrated in XSS and weak crypto. The negative result is backed by multiple pre-specified analyses—Fisher, Wilson intervals, mixed-effects model, Firth, permutation, placebo, length adjustment. The effect sizes are small and consistent across sectors. Model choice being the stronger lever is a practical, well-supported takeaway.\n\nThe soft spot is the composition explanation. Each CWE has exactly one hand-authored template, so the leave-one-CWE-out analysis is literally leave-one-template-out. CWE-502 sits at a 100% baseline flag rate—a ceiling, so drift can only be non-positive—and CWE-22 is 16% on 25 samples. The conditional OR of 0.38 in the mixed model and the stratified permutation p=0.044 are carried by those same two templates. So the claim that the apparent protective effect is a \"genuine but composition-bound within-task association\" is not supported by the data; it's a plausible post hoc reading. To their credit, the authors acknowledge this in Section 7.4 and in the Table 7 caption, but the abstract and conclusion state the composition story as a finding, not a caveat. That should be reined in or tested with multiple templates per CWE.\n\nMinor concerns: the baseline is small (235 code-only samples, 25 per CWE), the raw industry-vs-baseline contrast carries interface drift (a fact the paper quantifies honestly), and the matched-baseline analysis is the clean comparison. None of these undercut the null; they just limit how strongly you can interpret the pattern.\n\nWho's this for: anyone building secure-code benchmarks or deciding where to spend governance effort on LLM-based development. It deserves a serious referee; I'd send it out and ask for a revision that softens the composition claim. The measurement protocol and the null itself are worth publishing on their own.","headline":"The null result—no sector-conditioned security drift—holds up under scrutiny, but the paper's composition-artifact explanation is untested because template and CWE are perfectly confounded.","tokens_in":17126,"tokens_out":2266,"would_cite":true,"duration_ms":22493,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that conditioning code-generation prompts on critical-infrastructure sector context does not produce statistically significant changes in vulnerability rates; the apparent protective effect is an artifact of two vulnerabili","keywords":["security drift","LLM code generation","prompt engineering","critical infrastructure","CWE","static analysis","vulnerability rates","benchmark"],"falsifier":"A replication that assigns multiple independently written baseline templates per CWE (e.g., three to five templates for CWE-502 with baseline flag rates below 100%, and several for CWE-22) would settle whether the drift is category-specific or template-specific. If a new CWE-502 template's industry rate still drops while another does not, or if the pooled baseline-vs-industry gap becomes significant under any template set, the paper's composition-artifact claim would need revision.","tokens_in":16213,"feed_emoji":"🛡️","tokens_out":2535,"duration_ms":28519,"temperature":0.7,"pith_summary":"SecDrift asks whether adding industry context to code-generation prompts changes the security of the code LLMs produce. Across 5,355 evaluations with 6 models, 8 CISA sectors, and 9 vulnerability categories, the answer is no: the apparent 2.7-percentage-point drop in vulnerability rates under industry framing is not statistically significant and disappears when two CWE categories are removed. No single sector shows drift distinguishable from baseline, and a placebo using non-CISA industries reproduces the same small pattern, suggesting generic domain wording rather than critical-infrastructure identity. The paper's positive finding is that model choice moves security outcomes consistently, whereas prompt framing does not. The authors use a matched-baseline design to isolate terminology from task changes and a human-validation study to bound the static-analysis false-negative rate.","feed_headline":"Model choice, not sector prompts, drives AI code security","feed_subtitle":"5,355 evaluations find no significant security drift from industry framing; vulnerability rates track the model, not the prompt.","key_machinery":"The SecDrift benchmark uses a 5-dimension transformation (context injection, terminology mapping, stakeholder framing, use-case grounding, requirement preservation) to produce industry prompts from neutral baselines, plus a matched-baseline condition that substitutes only domain terminology while preserving the task interface. A mixed-effects logistic regression separates marginal from conditional effects; a separate non-CISA placebo controls for generic industry specificity; and a two-tier functional check plus human adjudication of zero-detection categories bounds detector blind spots. Together these controls let the authors attribute the small pooled drift to composition rather than to se","core_discovery":"The central claim is that sector-conditioned prompt framing does not produce statistically significant security drift in LLM-generated code. The paper states that the apparent protective effect of industry framing is a composition artifact of two CWE categories, not a general or sector-conditioned effect, while model selection is the variable that most consistently moves security outcomes. The baseline-to-industry gap of -2.7 percentage points has Fisher's exact p = 0.24 and Cohen's h = -0.08; excluding CWE-502 and CWE-22 reverses the sign (+0.4pp, p = 1.00). None of eight sectors shows drift distinguishable from baseline, and a non-CISA placebo reproduces the CISA industry rate almost exact","pith_inferences":["If a future replication uses multiple independently authored templates per CWE, the composition story may shift: the within-task association for CWE-502 and CWE-22 could prove to be template-specific rather than category-specific.","The placebo result invites a broader test: any domain framing, not just critical-infrastructure wording, may produce a small non-significant shift, suggesting prompt specificity itself is weak as a security lever.","The human-validation finding implies that benchmark improvements should focus on detector coverage for XSS and weak cryptography before relying on flag rates to rank models or prompts.","Because model choice is the consistent driver, security-evaluation suites for code LLMs could be designed to report model-conditioned profiles rather than aggregate prompt-conditioned rates."],"forward_implications":["Developers should not treat prompt phrasing, including industry context, as a security control: the data show no significant effect in either direction.","Organizations should prioritize model selection over sector-specific prompt design, since model differences persist across conditions while sector differences do not.","Aggregate vulnerability rates should be reported with per-CWE breakdowns, because a few categories can dominate and create composition artifacts.","The static-analysis flag rate understates true vulnerability for XSS and weak cryptography, so absolute rates in those categories should be read as detector lower bounds.","Abstention-prone models can earn deceptively low vulnerability rates by generating code for only a fraction of prompts, so refusal behavior must be accounted for in security benchmarks."],"fun_headline_variants":["AI code security hinges on model, not sector prompts","Industry framing doesn't shift AI code vulnerability rates","Model choice, not sector framing, determines AI code security","Sector-specific prompts don't affect AI code security: model does","AI code security: model choice outweighs sector prompt effects"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"Each CWE category is tested with exactly one hand-authored baseline template, so the vulnerability class and the template are perfectly entangled; if different templates for the same CWE were used, the composition artifact and even the pooled null could change.","fun_headline_variants_meta":{"raw":{"variants":["AI code security hinges on model, not sector prompts","Industry framing doesn't shift AI code vulnerability rates","Model choice, not sector framing, determines AI code security","Sector-specific prompts don't affect AI code security: model does","AI code security: model choice outweighs sector prompt effects"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000722,"raw_usage":{"total_tokens":3161,"prompt_tokens":913,"completion_tokens":2248,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":657,"completion_tokens_details":{"reasoning_tokens":2168}},"tokens_in":657,"tokens_out":2248,"duration_ms":15049,"temperature":1.0,"reasoning_tokens":2168,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T03:02:46.154769+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A replication that assigns multiple independently written baseline templates per CWE (e.g., three to five templates for CWE-502 with baseline flag rates below 100%, and several for CWE-22) would settle whether the drift is category-specific or template-specific. If a new CWE-502 template's industry rate still drops while another does not, or if the pooled baseline-vs-industry gap becomes significant under any template set, the paper's composition-artifact claim would need revision.","supporting_citations":[],"review_version":1}