{"id":"6906c8dd-fc0d-4f94-ae31-cf5c58f5736d","arxiv_id":"2505.11857","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"F2SRD retrieves relevant OWASP ASVS verification requirements for each functional requirement and uses them to prompt GPT-4, producing security requirements judged more inspired, diverse, and specific than two baselines.","lead":"This paper presents F2SRD, a system that reads a software's functional requirements, looks up matching security checks from the OWASP standard, and asks GPT-4 to write security requirements using those checks as guidance. The authors report that the resulting security requirements score higher than two existing methods on measures of inspiration, diversity, and specificity when tested on three software requirement datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The comparison to GPT-4 does not isolate relevance: any ASVS VR context, random or retrieved, might explain the observed SR quality gains. A random-VR ablation is needed before the central attribution to relevant retrieval is supported.","rationale":"Good-faith reading: F2SRD is a practical pipeline, and the paper supplies real evidence for it: retriever ablations (w/o training and w/o data filtering in Section 4.2), a blinded human study with ICC = 0.82, public artifacts, and explicit failure cases. The system almost certainly changes GPT-4's output. The unresolved question is whether the change is caused by retrieving the most relevant VRs or simply by adding ASVS-style text to the prompt. The GPT-4 baseline is not an adequate control for that question because it lacks all standard guidance. The reader's weakest assumption (synthetic FR-VR data quality and the ICC = 0.47 relevance annotations) is a real and admitted limitation, but it targets the reliability of the retriever's relevance signal. The missing random-VR control targets a more fundamental link: even a perfect retriever would not establish the paper's mechanism unless the relevance of the retrieved VRs, rather than their ASVS origin or prompt length, is shown to drive SR quality. A randomized or permuted VR condition is inexpensive with the released code and would settle this directly. If random VRs match F2SRD, the contribution should be reframed as standard-context prompting rather than relevant retrieval; if F2SRD wins, the current CONDITIONAL verdict can be upgraded. I therefore recommend keeping the reader's CONDITIONAL verdict unchanged.","tokens_in":22792,"tokens_out":8153,"duration_ms":86523,"concrete_test":"Use the released replication package to run a randomized-VR control on the 323 ePurse/CPN/GPS FRs. Keep the exact Fig. 5 prompt and the one-SR-per-pair rule, but replace the retriever's top-5 VRs with five VRs sampled uniformly at random from the 241 valid ASVS VRs (or, equivalently, with VRs permuted across FRs so the VR distribution is unchanged). Compute the three quantitative metrics and run the blinded human specificity/inspiration rating on a size-213 stratified sample for both conditions. If random/permuted VRs produce human scores and acceptance rates statistically indistinguishable from F2SRD, the relevant-retrieval attribution is unsupported; if F2SRD clearly beats the random-VR condition, the central claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is causal: retrieving the most relevant ASVS verification requirements is what makes F2SRD generate more inspired, diverse, and specific SRs. The evaluation does not test that causal link. In Section 4.3.2, the GPT-4 baseline receives only an FR-to-SR prompt, while F2SRD receives the same FR plus five ASVS VRs and a longer two-example prompt (Fig. 5). The two conditions therefore differ in context length, domain vocabulary, and instruction complexity, not only in VR relevance. The qualitative examples in Fig. 7 show F2SRD outputs containing ASVS-specific terms such as subdomain takeover, DNS expiry, and LFI/RFI; injecting any ASVS text into the prompt could produce exactly this effect, whether or not the VR was relevant to the FR. The retriever accuracy results in Section 4.2 do not close this gap: they measure agreement with a human consensus whose initial ICC was only 0.47, and Section 5 explicitly admits that the synthetic FR-VR training data were not quality-assessed. A missing random-VR condition is therefore the most load-bearing gap: it leaves open the possibility that the paper's headline result is driven by the presence of standard-derived context rather than by relevant retrieval.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes F2SRD, a two-phase approach that generates security requirements (SRs) from functional requirements (FRs) by first retrieving relevant verification requirements (VRs) from the OWASP ASVS standard and then prompting GPT-4 with the FR plus the top-5 VRs. Phase I synthesizes FR-VR pairs with GPT-4, filters them with ColBERTv2, and trains a ColBERTv2-based retriever augmented with a learnable token-weight embedding initialized from TF-IDF. Phase II uses the retriever to select the top-5 VRs for each FR and asks GPT-4 to generate an SR when the VR is deemed relevant. The authors evaluate the retriever on three RE'17 datasets (ePurse, CPN, GPS), report 80.4% accuracy against a human-consensus gold standard, and compare the generated SRs against RelGAN and a standalone GPT-4 baseline on self-information, self-BLEU, vocabulary size, and human-rated inspiration and specificity. The central claim is that retrieval of relevant ASVS VRs, rather than generic prompting, is what makes the generated SRs more inspired, diverse, and specific.","tokens_in":23010,"tokens_out":9175,"duration_ms":80971,"significance":"If the central claim holds, the paper makes a useful contribution: it demonstrates a concrete way to ground LLM-based security requirements generation in established verification standards, with an evaluation framework that combines quantitative metrics, human scoring, and public artifacts. The RE'17 benchmark datasets are external, the human evaluation for SR quality shows strong inter-rater reliability (ICC=0.82), and the reported differences versus both baselines are substantial and statistically significant. The authors also make data and scripts publicly available via figshare, which aids replication. However, the evaluation does not currently isolate the effect of relevant retrieval from the effect of simply providing any ASVS-style context, and the retriever's training data and gold standard are acknowledged to be weakly validated. These issues are fixable and do not undermine the value of the framework, but they must be addressed before the causal claim about relevance can be accepted.","major_comments":[{"comment":"The central claim that relevant VR retrieval is the cause of the quality gains is not tested. The F2SRD prompt in Fig. 5 includes the FR, five ASVS VRs, and two worked examples, while the GPT-4 baseline prompt (Fig. 10 in the replication package) contains no VR context. The comparison therefore varies at least three factors—presence of standard-derived domain vocabulary, prompt length, and example count—in addition to VR relevance. The qualitative examples in Fig. 7 are consistent with any injection of ASVS-specific context (e.g., subdomain takeover, DNS expiry, LFI/RFI). A random-VR ablation, in which the same prompt template is used with randomly selected ASVS VRs, is necessary to attribute the gains in Table 4 and Table 6 to relevant retrieval. Without it, the abstract's claim that relevant VRs are what make the SRs 'more focused' is not supported by the data.","section":"Section 4.3.2, Tables 4 and 6"},{"comment":"The paper acknowledges in Section 5 that 'We did not assess the quality of this synthetically generated data.' This is load-bearing: the 1,536 synthetic FR-VR pairs are the only training data for the retriever, and the retriever's accuracy in Table 2 is measured on FRs from the same source type. If GPT-4's synthetic FRs share template-specific patterns with the prompts used to create them, the retriever may learn those patterns rather than genuine FR-VR relevance. The filtering by zero-shot ColBERTv2 (top-30% threshold) mitigates but does not eliminate this risk, because the filter is itself an automatic model with no human validation. Please add a sample-based human quality assessment of the synthesized pairs, or validate the retriever on independently labeled FR-VR pairs.","section":"Section 3, Step 1 and Section 5 (Construct Validity)"},{"comment":"The gold standard for retriever evaluation is a consensus derived from annotators whose independent agreement was only ICC(2,k)=0.47, which is generally considered moderate. The consensus procedure may improve label quality, but the low initial agreement shows that the relevance construct is difficult even for humans; the reported accuracy rates (80.4% vs 85.1%) should therefore be interpreted with caution. Please report per-annotator accuracy or expert-validated labels, and report an agreement measure on the final consensus labels (e.g., Cohen's kappa for the two-raters-after-discussion step). This matters because RQ1's conclusion about the retriever's utility rests entirely on this gold standard.","section":"Section 4.2"},{"comment":"The self-information metric as defined is not normalized for sequence length. If p(SR) is the joint probability of the SR under Davinci-002, longer SRs will tend to have lower p and hence higher SI, independent of 'unexpectedness.' Since F2SRD's SRs are visibly longer and more detailed (e.g., Fig. 7), the SI advantage in Table 4 may partly reflect verbosity rather than inspiration. Please report a length-normalized variant (e.g., average per-token negative log-likelihood) or include SR length as a covariate/an effect-size control. This does not undermine the human inspiration results in Table 6, but it is needed for the quantitative claim.","section":"Section 4.3.4, Eq. (5)"}],"minor_comments":[{"comment":"The phrase 'achieved average accuracy gains of 34.45% and 8.94%, respectively' is inconsistent with the preceding sentence; these are relative improvements when the steps are included, not gains when they are removed.","section":"Section 4.2, paragraph after Table 2"},{"comment":"The caption contains a duplicated word: 'Relevance Between Between FR and VR.'","section":"Figure 6 caption"},{"comment":"The sentence beginning 'The higher Self-Information exception in CPN, as observed with GPT-4' is confusing because in Table 4 the CPN exception is RelGAN's higher self-information (160.70 vs GPT-4's 141.17), not GPT-4's; please rephrase.","section":"Section 4.3.6"},{"comment":"Please specify the exact computation of p(SR) from Davinci-002 (e.g., joint token probability, temperature, decoding parameters) and the tokenization/BLEU settings used for Self-BLEU.","section":"Equations (5) and (6)"},{"comment":"The text 'does not make any sence' should read 'does not make any sense.'","section":"Section 4.3.4"},{"comment":"The description 'the dialogue interface of GPT-4' should identify the model version and sampling parameters (temperature, max tokens) for reproducibility; the same applies to the GPT-4 calls used for F2SRD and for synthetic data generation.","section":"Section 4.3.2"},{"comment":"References [24] and [25] are the same ICSE 2023 paper and should be consolidated.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The main barrier is the missing random-VR ablation, which is a standard control and should be feasible with the authors' existing code. The synthetic-data limitation is honestly acknowledged, but it is central to RQ1 and should be addressed with at least a sample-based human check. I see no novelty or scope concern; the paper is a reasonable fit for a software-engineering venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: F2SRD is a genuinely useful two-phase pipeline that retrieves OWASP ASVS verification requirements for a functional requirement and uses them to condition GPT-4 for security requirement generation. It is a real contribution, evaluated more thoroughly than most papers in this corner, and the authors are honest about limitations. The biggest issue is that the headline comparison does not isolate the role of relevant retrieval.\n\nWhat's new: the specific combination of a trained FR-VR retriever (ColBERTv2 backbone plus a learnable TF-IDF-initialized token-weighting layer) and LLM generation conditioned on standards text. The synthetic data pipeline (GPT-4 generates FRs per VR, filtered by ColBERTv2) is also new, and so is the evaluation framework with inspiration, diversity, and specificity. They report retriever accuracy 80.4% against human consensus, and F2SRD outperforms both RelGAN and standalone GPT-4 on self-information, self-BLEU, vocabulary size, and human ratings with ICC 0.82. Data and scripts are public on figshare, and the paper explicitly flags the synthetic-data quality and external validity concerns.\n\nThe soft spots are real but fixable. Most important: the GPT-4 baseline is confounded. GPT-4 gets a short FR-to-SR prompt; F2SRD gets the FR, five VRs, and two worked examples. The observed gains could come from any ASVS-text context or just a longer prompt, not from the retriever picking relevant VRs. A random-VR ablation would settle this, and I don't see one. This matters because the paper's contribution is exactly the retrieval of relevant standards, not merely the use of standards text.\n\nSecond, the retriever ground truth rests on consensus annotations after initial ICC 0.47—moderate agreement. The consensus process helps, but it means the 80.4% accuracy number is softer than it looks. Third, the training data is synthesized by GPT-4 and filtered by ColBERTv2, which is also the retriever's backbone. The authors use real test FRs, so the circularity is limited, but it's worth flagging.\n\nNone of this is fatal. The practical claim—that standards-derived context makes generated SRs more specific and diverse—is likely true. The causal claim that relevant retrieval is what drives the gain is not yet proven. A revision with a random-VR condition and a length-matched GPT-4 prompt would substantially strengthen it.\n\nThis paper is for researchers in requirements engineering, LLM-based software engineering, and security automation. It provides a usable baseline and an honest discussion of evaluation pitfalls. It absolutely deserves a serious peer-review round rather than a desk reject; I would accept it conditionally and require the random-VR ablation before final acceptance.","headline":"A useful, honest pipeline for LLM-based security requirement generation whose headline comparison leaves the role of relevant retrieval unproven.","tokens_in":23618,"tokens_out":4179,"would_cite":true,"duration_ms":41472,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"F2SRD generates security requirements by first retrieving the most relevant OWASP ASVS verification checks and then prompting GPT-4 with them, beating standalone GPT-4 and RelGAN on inspiration, diversity, and specificity.","keywords":["security requirements generation","functional requirements","OWASP ASVS","verification requirements retrieval","retrieval-augmented generation","ColBERTv2","large language models","requirements engineering"],"falsifier":"Randomize the VR condition: feed GPT-4 randomly selected ASVS verification requirements instead of the retriever's top-five for the same functional requirements; if the inspiration, diversity, and specificity scores stay at F2SRD levels, the claimed benefit comes from adding standard text rather than from retrieving the relevant checks.","tokens_in":22529,"feed_emoji":"🔐","tokens_out":11925,"duration_ms":107335,"temperature":0.7,"pith_summary":"Security requirements are often written late and incompletely because engineers lack security expertise. F2SRD is a two-phase method that turns functional requirements (FRs) into security requirements (SRs) without that expertise: a retriever trained on synthetic FR-verification-requirement (VR) pairs selects the most relevant OWASP ASVS verification checks for a given FR, and those VRs are placed into structured prompts that direct GPT-4 to write concrete SRs, or to decline when the VR does not apply. The authors report that F2SRD outperforms standalone GPT-4 and RelGAN on inspiration, diversity, and specificity across all three project datasets, and that human raters score its SRs as more specific and more inspiring. The practical claim is that anchoring generation to standard verification checks is what turns generic security advice into actionable requirements.","feed_headline":"Retrieved OWASP checks make GPT-4 security specs more specific","feed_subtitle":"F2SRD beats standalone GPT-4 and RelGAN on inspiration, diversity, and specificity across three project corpora.","key_machinery":"The load-bearing component is the weighted late-interaction retriever. An FR and a VR are encoded as token-level vectors by shared encoders; a learnable weight embedding layer assigns each VR token a weight initialized from TF-IDF; and the final similarity score is computed as $$\\mathrm{Sim}_{FR,VR} = \\sum_{i \\in m} \\max_{j \\in n} \\left( W_{VR} \\cdot \\mathrm{CosSim}(E_{FR}, E_{VR}) \\right).$$ This lets a single meaningful token, such as \"registration,\" drive a match even when the FR and VR share little overall wording. In the generation phase the retrieved VRs become the steering content of the GPT-4 prompt, which also contains worked positive and negative examples and an instruction to emit no SR when the VR is irrelevant.","core_discovery":"The paper's central claim is that relevant security verification standards are the key conditioning signal for generating high-quality security requirements. F2SRD learns to match functional requirements to verification requirements from the OWASP ASVS using a ColBERTv2-style late-interaction retriever augmented with a learnable token-weighting layer initialized from TF-IDF statistics over the ASVS corpus; after GPT-4-synthesized pairs are filtered by a zero-shot retriever, 1,536 FR-VR pairs form the training set. At inference the retriever recommends five VRs per FR, and GPT-4 is instructed to judge whether the FR truly needs verification by that VR and, when it does, to write a targeted SR. The paper reports retriever accuracy of 80.4% against consensus human labels, with human labelers reaching 85.1% agreement with the consensus, and both quantitative metrics and human-subject ratings in which F2SRD scores above RelGAN and standalone GPT-4 on inspiration, diversity, and specificity.","pith_inferences":["The retriever was trained on GPT-4-generated FRs, so the reported gains may depend on how closely real-world FRs resemble the prompting style used to create the synthetic pairs; this can be checked by testing on human-written FRs from other domains.","A control experiment that feeds GPT-4 randomly chosen VRs rather than retrieved ones would isolate whether the quality gain comes from retrieval relevance or merely from adding standard-sounding text to the prompt; the paper does not run this control.","The human evaluation only compared F2SRD with GPT-4, so the apparent superiority over RelGAN rests on quantitative metrics; a head-to-head human reading of RelGAN outputs after filtering could give a fuller picture.","Since the pipeline is standard-agnostic at the architecture level, adapting it to ISO/IEC 15408 or FIPS 200 would test whether verification standards generalize as generation guidance, a step the paper leaves for future work."],"forward_implications":["If the reported results hold, requirements engineers can draft security requirements immediately from functional specifications, with the retriever surfacing the ASVS checks most likely to apply and the LLM turning them into concrete statements.","The generated SRs remain traceable to a recognized verification standard, so the same pipeline can double as an early ASVS compliance check rather than producing free-floating security advice.","The retriever's 80.4% accuracy indicates that automated FR-to-VR mapping is close to human-level on these projects, meaning the main bottleneck shifts from finding relevant checks to judging whether an SR is actually implementable.","Because the method is built on a generic retrieve-then-prompt recipe, the same design can in principle be retargeted to other security standards once a retriever is trained on the corresponding VR corpus."],"supporting_citations":[{"why":"Supplies the 241 ASVS verification requirements that anchor both retrieval and generation.","marker":"[47]"},{"why":"Provides the ColBERTv2 late-interaction retrieval model used as the retriever backbone and training initialization.","marker":"[40]"},{"why":"Contributes the LLM-prompting technique used to synthesize ten functional requirements per VR.","marker":"[20]"},{"why":"Provides the zero-shot retrieval filtering strategy used to vet the synthetic FR-VR pairs down to 1,536.","marker":"[39]"},{"why":"Supplies the RelGAN text generation model used as one of the two comparison baselines.","marker":"[37]"},{"why":"Establishes RelGAN as a prior method for security-requirement synthesis and documents its known syntactic and relevance limitations.","marker":"[24]"},{"why":"Is the GPT-4 technical report, the LLM backbone used both for data synthesis and for final SR generation.","marker":"[2]"},{"why":"Offers the Self-BLEU metric used to quantify diversity of the generated security requirements.","marker":"[58]"},{"why":"Defines self-information, the information-theoretic measure used for the unexpectedness component of inspiration.","marker":"[21]"}],"fun_headline_variants":["OWASP retrieval sharpens GPT-4 security requirement generation","F2SRD: OWASP standards power more inspired security specs","Verification standards lift GPT-4 security requirements above baseline","Retrieved OWASP checks make GPT-4 specs more specific and diverse","Use OWASP verification to get better GPT-4 security requirements"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The approach stands on the assumption that the GPT-4-synthesized functional-requirement/verification-requirement pairs used to train the retriever encode genuine relevance rather than prompt-template artifacts, and that the consensus labels used as ground truth are trustworthy despite annotators' initial inter-rater reliability of only 0.47.","fun_headline_variants_meta":{"raw":{"variants":["OWASP retrieval sharpens GPT-4 security requirement generation","F2SRD: OWASP standards power more inspired security specs","Verification standards lift GPT-4 security requirements above baseline","Retrieved OWASP checks make GPT-4 specs more specific and diverse","Use OWASP verification to get better GPT-4 security requirements"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000394,"raw_usage":{"total_tokens":2094,"prompt_tokens":999,"completion_tokens":1095,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":1005}},"tokens_in":615,"tokens_out":1095,"duration_ms":10369,"temperature":1.0,"reasoning_tokens":1005,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:46:44.928661+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Randomize the VR condition: feed GPT-4 randomly selected ASVS verification requirements instead of the retriever's top-five for the same functional requirements; if the inspiration, diversity, and specificity scores stay at F2SRD levels, the claimed benefit comes from adding standard text rather than from retrieving the relevant checks.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 241 ASVS verification requirements that anchor both retrieval and generation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the LLM-prompting technique used to synthesize ten functional requirements per VR."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the RelGAN text generation model used as one of the two comparison baselines."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes RelGAN as a prior method for security-requirement synthesis and documents its known syntactic and relevance limitations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines self-information, the information-theoretic measure used for the unexpectedness component of inspiration."}],"review_version":1}