{"id":"8923056a-a2b9-4837-9306-fd49c662e232","arxiv_id":"2505.10494","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new four-task benchmark and a fine-tuned judge model show that leading AI coding assistants still generate vulnerable code, even though many can spot vulnerabilities when prompted.","lead":"The paper introduces CoV-Eval, a benchmark that tests AI coding assistants on four security tasks: writing safe code, fixing vulnerabilities, and spotting and naming them. It also trains a judge model, VC-Judge, to grade the AI outputs, and finds that today's models are better at detecting dangerous code than at avoiding or repairing it.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline finding that 'most LLMs identify vulnerable codes well' is not supported: the detection test set is 89.8% vulnerable, so an always-positive model reaches 94.6% F1, matching several Table 1 entries.","rationale":"The reader identified VC-Judge train/test contamination as the weakest assumption. That concern is real and already warrants the conditional verdict. However, the judge-contamination issue mainly affects the generative SR@1 numbers, which are scored by VC-Judge, and the SC-IFT filtering in Section 6.3. The vulnerability detection metrics in Table 1 are computed directly from LLM outputs using regular matching (Section 3.4), so they are independent of VC-Judge. The more load-bearing problem is the structure of the detection test set itself: with 477 positive versus 54 negative examples, high F1 and recall are achievable without any discrimination ability. A model that always outputs 'vulnerable' reaches the exact recall and near-exact F1 values reported for several leading models, making the headline claim 'most LLMs identify vulnerable codes well' unsupported as stated. This is not a question of fine-tuning or alignment; it is a measurement-design flaw that would survive even a perfectly unbiased judge. Fixing it requires balanced evaluation and reporting precision/balanced accuracy. The paper's other central findings, such as low classification F1 and moderate SR@1 rates, are not undermined by this specific concern, and the benchmark still has value, so conditional acceptance with a required re-analysis is the appropriate verdict.","tokens_in":22843,"tokens_out":6965,"duration_ms":70439,"concrete_test":"Recompute vulnerability detection metrics on the 531-program test set after balancing the classes: take the 54 non-vulnerable programs, randomly sample 54 of the 477 vulnerable programs, and repeat this 100 times. Report mean F1, precision, recall, and balanced accuracy for each model in Table 1, and compare with the always-vulnerable classifier on the same balanced subsample. If model F1 drops to near 50% or does not exceed the trivial baseline, the claim that most LLMs identify vulnerable codes well should be withdrawn or substantially qualified.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's first main finding depends on binary vulnerability detection F1/recall in Table 1. The detection test set (Appendix A.1) contains 477 vulnerable and only 54 non-vulnerable programs, so an always-positive classifier achieves recall 100% and positive-class F1 = 2*(477/531)*1/(477/531+1) = 94.6%; the weighted-F1 trivial baseline is 85.0%. Several Table 1 entries sit at or near this ceiling: GPT-4o has detection F1 94.62 and recall 99.58, and Qwen1.5-14B-chat has F1 94.64 and recall 100.00, exactly the always-positive baseline. Most of the 'good detection' entries above 80% F1 are only a few points above a classifier that needs zero discrimination, because precision is never reported and detection accuracy is omitted from the table. Consequently, the abstract's claim that 'most LLMs identify vulnerable codes well' may merely reflect the extreme positive skew of the test set. This also explains the otherwise striking contrast with low classification F1: a model that labels everything as vulnerable has high detection recall and near-zero classification precision, which is precisely the pattern in Table 1. The self-detection recall numbers in Table 3 inherit the same baseline issue because they use the same highly imbalanced detection test set.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CoV-Eval, a multi-task benchmark for evaluating the code security of large language models. It covers four tasks: code completion, vulnerability repair, vulnerability detection, and vulnerability classification, built on the Copilot-CWE seed set and a new Vul-Evol synthesis framework. The authors also introduce VC-Judge, a fine-tuned LLAMA3-8B-Instruct model designed to automatically judge whether generated code is vulnerable, and report a study of 20 proprietary and open-source LLMs. The main empirical findings are that most LLMs detect vulnerabilities with relatively high recall but still generate insecure code, classify vulnerability types poorly, repair vulnerabilities with limited success, and that fine-tuning on secure code data can improve both security and usability.","tokens_in":23139,"tokens_out":6626,"duration_ms":59892,"significance":"If the results were fully robust, CoV-Eval would be a useful multi-task benchmark for code security evaluation, and VC-Judge could offer a scalable alternative to human review. The paper has concrete strengths: the benchmark construction is transparent, the dataset and code are released, the human-alignment comparison in Table 5 is informative, and the fine-tuning experiments in Section 6.3 are a valuable addition. However, two load-bearing issues — the extreme class imbalance in the detection test set and the circular training/evaluation use of VC-Judge — currently undermine the headline claim that LLMs 'identify vulnerable codes well' and the reliability of the reported generative scores. The significance of the paper therefore depends on addressing these issues.","major_comments":[{"comment":"The vulnerability detection test set contains 477 vulnerable and only 54 non-vulnerable programs (89.8% positive). A trivial always-positive classifier achieves positive-class F1 94.6%, recall 100%, and weighted F1 85.0%. Several entries in Table 1 are effectively at this ceiling: Qwen1.5-14B-chat (94.64/100.00), ChatGLM3-6B (94.64/100.00), CodeShell-7B-chat (94.54/99.79), and GPT-4o (94.62/99.58). Since precision and accuracy are not reported, the abstract's claim that 'most LLMs identify vulnerable codes well' is not supported: a model that labels everything as vulnerable would produce numbers very close to the reported ones. The authors should re-report detection results with precision, balanced accuracy, MCC, or F1 relative to a majority-class baseline, and revise the qualitative conclusions accordingly.","section":"§3.2, Appendix A.1, Table 1"},{"comment":"VC-Judge's instruction-tuning dataset includes 216 LLM-generated CoV-Eval programs and the 531-program CoV-Eval detection test set. The same VC-Judge is then used as the primary scorer for code completion and repair in Tables 1 and 3, and as the filter for the SC-IFT data whose effect is later measured with the same judge in Section 6.3. This creates a leak: VC-Judge has seen programs from the same 54 seed scenarios it later grades, so the human-alignment numbers in Table 5 and the SR@1 values in Tables 1, 3, and 4 may be optimistically biased. The authors should train VC-Judge without any CoV-Eval-origin data, evaluate on a held-out subset of scenarios or vulnerability types disjoint from the training data, or validate a sample of judged completions against fresh human annotations.","section":"§4.1, Table 7, §5, §6.3"},{"comment":"Each LLM is evaluated with a single inference per test instance (temperature 0.6, top-p 0.9 for open models), and every reported number is a point estimate from one run, with no confidence intervals, bootstrap estimates, or multiple seeds. LLM sampling is stochastic, so the close rankings in Table 1 (e.g., CodeLLAMA-13B vs CodeLLAMA-7B in classification F1) and the comparative findings in Section 5.2 may reflect noise. The authors should provide variance estimates, run multiple seeds for at least the main tables, or explicitly justify why a single draw is sufficient for the paper's comparative claims.","section":"§5.1, Tables 1–3"}],"minor_comments":[{"comment":"The reference 'Antropic. 2024. Claude 3.5 sonnet' contains a typo; it should be 'Anthropic'.","section":"References"},{"comment":"The description for CWE-306 is listed as 'Insufficient Processing of Invalid or Unintended Input', but Figure 8 gives the correct standard name 'Missing Authentication for Critical Function' for CWE-306. Please reconcile the two.","section":"Appendix B"},{"comment":"The row 'CodeLLAMA-7B-Instructt' has a typo and is set without a clear line break from the preceding row, making the table hard to parse.","section":"Table 4"},{"comment":"Figure 4 uses 'Index of LLMs' on the x-axis but does not provide a legend mapping indices to model names; please add a mapping or label the axis with model names.","section":"Figure 4"},{"comment":"The phrase 'From LLAMA2 to CodeLLAMA and then to LLAMA3 and LLAMA3.1' suggests a linear model lineage; CodeLLAMA is a separate code fine-tune of LLAMA2, not a successor in the LLAMA family. Please clarify.","section":"§5.2 finding (4)"},{"comment":"The text says discrimination task outputs are evaluated with 'regular matching' but does not specify the exact extraction rules; please state how keywords are extracted and normalized (e.g., case, leading zeros in CWE identifiers).","section":"§3.4"}],"recommendation":"major_revision","confidential_remarks":"The two main concerns — the heavily imbalanced detection set and the circular training/evaluation of VC-Judge — are directly fixable through additional experiments and re-analysis, so I do not view them as grounds for rejection. I would recommend that the editor ask for a revised version that re-baselines the detection results against trivial classifiers and retrains or revalidates VC-Judge on data disjoint from CoV-Eval."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: CoV-Eval is a real attempt at a multi-task code-security harness, and it deserves a serious referee. But the abstract's claim that 'most LLMs identify vulnerable codes well' is not supported by the evidence. The detection test set contains 477 vulnerable and 54 non-vulnerable programs. An always-positive classifier scores 94.6% F1 on the positive class and 85% weighted F1. Several models in Table 1 sit right at that ceiling—Qwen1.5-14B-chat and ChatGLM3-6B both hit 94.64 F1 with 100% recall, which is exactly the always-positive behavior. The rest of the 'good detection' results are only a few points above a trivial baseline that needs zero discrimination.\n\nWhat is genuinely new: the four-task combination, the Vul-Evol scenario synthesis, the VC-Judge fine-tuned evaluator, and the head-to-head comparison across 20 models. The paper is transparent—appendices give dataset statistics, prompt templates, and human annotation details, and the code and data are released. The per-CWE generation analysis in Table 2 is useful, and the instruction-tuning study in Section 6.3, while entangled with VC-Judge, points in a plausible direction.\n\nThe soft spots are real. First, the test-set imbalance above. Second, circularity: VC-Judge is trained on 216 CoV-Eval-generated codes and the 531-program CoV-Eval detection set, then used to score CoV-Eval results and even to filter the SC-IFT data whose effect is later measured with the same judge. That is fixable by holding out evaluation data from judge training, but as it stands the absolute SR@1 numbers are not fully independent. Third, each model is scored from a single inference per scenario with no error bars; on 54 scenarios the SR@1 differences in Table 2 are likely noise.\n\nThe generation and repair findings are probably robust—low classification F1 and low repair rates are consistent with earlier work and don't depend on the detection ceiling. The headline detection finding is the one that needs to be reworked.\n\nI'd send this to peer review, but the authors should be asked to report precision, add a balanced or stratified detection set, and separate binary detection from type naming. After that, CoV-Eval could become a useful community benchmark.","headline":"The multi-task benchmark and VC-Judge are genuinely useful, but the headline 'LLMs detect vulnerabilities well' finding is an artifact of a 90%-vulnerable test set.","tokens_in":23689,"tokens_out":3974,"would_cite":false,"duration_ms":35734,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Most LLMs can tell when code is vulnerable but still produce insecure completions at a 25-50% rate, according to a new four-task benchmark and an automated judge.","keywords":["code security","large language models","vulnerability benchmark","multi-task evaluation","vulnerability detection","vulnerability repair","secure code generation","instruction tuning"],"falsifier":"Have independent security experts re-annotate a random sample of the completion outputs from the 20 models, or the 531-program detection set, and compare their labels with VC-Judge's; if human-versus-judge agreement is low or the disagreements cluster on specific models or CWE types, the reported security rates and rankings are artifacts of the judge rather than properties of the models.","tokens_in":22626,"feed_emoji":"🔐","tokens_out":8380,"duration_ms":77425,"temperature":0.7,"pith_summary":"The paper's central claim is that large language models are far better at recognizing vulnerable code than at writing secure code: across 20 models, most detect binary vulnerability accurately, yet 25-50 percent of their single-attempt code completions are insecure. To make this measurable, the paper builds CoV-Eval, a four-task benchmark covering code completion, vulnerability repair, vulnerability detection, and vulnerability classification across 18 common weakness types in C and Python. It also trains VC-Judge, an automated vulnerability reviewer fine-tuned to align with human experts, so the whole benchmark can be scored without hand inspection. If the claim is right, tool builders can use CoV-Eval to compare models on security directly, and training researchers get a concrete target: the weak links are vulnerability-type classification and repair, not basic detection.","feed_headline":"Most AI coders ship insecure code even when they spot it","feed_subtitle":"Four-task benchmark of 20 models finds high vulnerability detection but weak repair and 25-50% insecure completions.","key_machinery":"The engine of the evaluation is a pair of instruments. CoV-Eval is a multi-task benchmark built from 54 seed code scenarios in C and Python, expanded by Vul-Evol, an instruction-evolution-and-filtering pipeline that produces 270 more complex scenarios; the test sets contain 324 completion cases, 477 repair cases, and 531 detection and classification cases across 18 CWE vulnerability types. VC-Judge is a fine-tuned LLM built from an 8-billion-parameter open model, trained on 14,952 instructional examples drawn from CoV-Eval outputs, the 531-program detection set, and a public vulnerability dataset; it is used as the automated grader of generated programs. The load-bearing mechanism is that VC-Judge's judgment-style prompts, which ask whether a code has a specific named vulnerability rather than leaving the type open, bring automated grading close enough to human reviewers to make the security rate (SR@1), the fraction of single-attempt completions judged non-vulnerable, comparable across models.","core_discovery":"The paper's discovery is a split between what LLMs can see and what they produce. In binary vulnerability detection, most of the 20 evaluated models score above 80 percent F1 and many recall nearly all vulnerable programs; two models that the paper tests even achieve 100 percent recall yet still generate insecure completions in about 26-31 percent of cases. The same models do poorly when asked to name the weakness: proprietary models reach only modest classification F1, and open-source models mostly stay below 27 percent. Repair is also limited, with open-source models fixing between roughly 24 and 60 percent of vulnerable programs even when told the vulnerability type. Across the completion benchmark, security rates by vulnerability type show that command injection, dangerous file upload, and integer overflow are the recurring failure modes. The paper presents CoV-Eval and VC-Judge as the reliable instruments that surface these patterns.","pith_inferences":["Another implication the paper leaves implicit is that the 25-50 percent single-draw vulnerability rate likely understates real risk in interactive assistants: when a user sees several sampled completions, the chance that at least one insecure snippet appears grows quickly, so the paper's SR@1 measure could be extended to SR@k.","Because VC-Judge's training data overlaps with the benchmark it grades, a natural next step is to publish per-instance judge labels and run a holdout human audit; without that, the fine-grained CWE-type statistics are the least certain numbers in the paper.","The judge's prompt design, testing one specific vulnerability type per call instead of open multi-class naming, looks transferable: feeding such judgment labels back into training, rather than raw code, could become a cheap security filter for other code-generation benchmarks.","The benchmark's task structure also suggests a diagnostic: comparing a model's security rate on completions against its own recall on self-generated code would directly measure whether security knowledge learned through detection transfers to generation time."],"forward_implications":["A model that detects vulnerabilities well can still generate them often, so detection accuracy alone is not evidence that a code assistant is secure to deploy.","Proprietary models remain more secure than open-source models on average, but code-specific fine-tuning closes part of the gap: specialized code models beat their general counterparts in completion security and overall security score.","Training on secure, vulnerability-free code data can improve generation security without harming functional usability, while vulnerability-detection instruction data specifically improves repair ability.","The recurring weak types, command injection, dangerous file upload, and integer overflow, offer concrete targets for security hardening in the next generation of models.","A fine-tuned judge with judgment-style prompts can grade generated code at scale with human-aligned rankings, replacing manual review for benchmark scoring."],"supporting_citations":[{"why":"Supplies the seed scenarios and the labeled vulnerable programs that become the completion, repair, and detection test sets.","marker":"(Pearce et al., 2022)"},{"why":"A prior single-task security benchmark whose limitation motivates the multi-task design.","marker":"(Siddiq and Santos, 2022)"},{"why":"Provides a prior large-scale completion benchmark used as the out-of-domain security test in the fine-tuning comparison.","marker":"(Bhatt et al., 2023)"},{"why":"Establishes that LLM vulnerability detectors still trail human experts, motivating a judge model trained for alignment.","marker":"(Steenhoek et al., 2024)"},{"why":"Supplies a large public C/C++ vulnerability dataset used to build VC-Judge's training samples and the detection instruction data.","marker":"(Fan et al., 2020a)"},{"why":"Contributes the instruction-evolution strategy that Vul-Evol adapts to generate harder vulnerable-code scenarios.","marker":"(Luo et al., 2023)"},{"why":"Provides the HumanEval pass@1 usability scores reported alongside security rates in the main comparison.","marker":"(Chen et al., 2021)"},{"why":"Supplies general code instruction data whose security-blind fine-tuning is compared with secure-code fine-tuning.","marker":"(Chaudhary, 2023)"}],"fun_headline_variants":["AI coders flag bugs but can't fix them","Why code copilots spot flaws yet ship them","20 LLMs: alert to vulnerabilities, weak on repair","Code models detect bugs but still generate insecure code","The AI coder blind spot: naming and fixing flaws"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"VC-Judge's labels are treated as ground truth for grading all 20 models, but the judge was itself trained partly on CoV-Eval programs, including 216 LLM-generated codes and the same 531-program detection set it later helps grade, so any systematic bias in VC-Judge shifts every security rate, repair score, and model ranking reported in the tables.","fun_headline_variants_meta":{"raw":{"variants":["AI coders flag bugs but can't fix them","Why code copilots spot flaws yet ship them","20 LLMs: alert to vulnerabilities, weak on repair","Code models detect bugs but still generate insecure code","The AI coder blind spot: naming and fixing flaws"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000109,"raw_usage":{"total_tokens":1028,"prompt_tokens":898,"completion_tokens":130,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":53}},"tokens_in":514,"tokens_out":130,"duration_ms":2070,"temperature":1.0,"reasoning_tokens":53,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:07:58.143112+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent security experts re-annotate a random sample of the completion outputs from the 20 models, or the 531-program detection set, and compare their labels with VC-Judge's; if human-versus-judge agreement is low or the disagreements cluster on specific models or CWE types, the reported security rates and rankings are artifacts of the judge rather than properties of the models.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"A prior single-task security benchmark whose limitation motivates the multi-task design."}],"review_version":1}