Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Can You Really Trust Code Copilots? Evaluating Large Language Models from a Code Security Perspective

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Most LLMs can tell when code is vulnerable but still produce insecure completions at a 25-50% rate, according to a new four-task benchmark and an automated judge.

desk verdict The multi-task benchmark and VC-Judge are genuinely useful, but the headline 'LLMs detect vulnerabilities well' finding is an artifact of a 90%-vulnerable test set. read the letter →

arxiv 2505.10494 v1 pith:SWUEFP6F submitted 2025-05-15 cs.CL

Yutao Mou , Xiao Deng , Yuxiao Luo , Shikun Zhang , Wei Ye This is my paper · ORCID
classification cs.CL
keywords codesecuritylargelanguagemodelsvulnerabilitybenchmarkmulti-taskevaluationdetectionrepairsecuregenerationinstructiontuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's central claim is that large language models are far better at recognizing vulnerable code than at writing secure code: across 20 models, most detect binary vulnerability accurately, yet 25-50 percent of their single-attempt code completions are insecure. To make this measurable, the paper builds CoV-Eval, a four-task benchmark covering code completion, vulnerability repair, vulnerability detection, and vulnerability classification across 18 common weakness types in C and Python. It also trains VC-Judge, an automated vulnerability reviewer fine-tuned to align with human experts, so the whole benchmark can be scored without hand inspection. If the claim is right, tool builders can use CoV-Eval to compare models on security directly, and training researchers get a concrete target: the weak links are vulnerability-type classification and repair, not basic detection.

What carries the argument

The engine of the evaluation is a pair of instruments. CoV-Eval is a multi-task benchmark built from 54 seed code scenarios in C and Python, expanded by Vul-Evol, an instruction-evolution-and-filtering pipeline that produces 270 more complex scenarios; the test sets contain 324 completion cases, 477 repair cases, and 531 detection and classification cases across 18 CWE vulnerability types. VC-Judge is a fine-tuned LLM built from an 8-billion-parameter open model, trained on 14,952 instructional examples drawn from CoV-Eval outputs, the 531-program detection set, and a public vulnerability dataset; it is used as the automated grader of generated programs. The load-bearing mechanism is that VC-Judge's judgment-style prompts, which ask whether a code has a specific named vulnerability rather than leaving the type open, bring automated grading close enough to human reviewers to make the security rate (SR@1), the fraction of single-attempt completions judged non-vulnerable, comparable across models.

What would settle it

Have independent security experts re-annotate a random sample of the completion outputs from the 20 models, or the 531-program detection set, and compare their labels with VC-Judge's; if human-versus-judge agreement is low or the disagreements cluster on specific models or CWE types, the reported security rates and rankings are artifacts of the judge rather than properties of the models.

Watch

Extended reading notes

Core claim

The paper's discovery is a split between what LLMs can see and what they produce. In binary vulnerability detection, most of the 20 evaluated models score above 80 percent F1 and many recall nearly all vulnerable programs; two models that the paper tests even achieve 100 percent recall yet still generate insecure completions in about 26-31 percent of cases. The same models do poorly when asked to name the weakness: proprietary models reach only modest classification F1, and open-source models mostly stay below 27 percent. Repair is also limited, with open-source models fixing between roughly 24 and 60 percent of vulnerable programs even when told the vulnerability type. Across the completion benchmark, security rates by vulnerability type show that command injection, dangerous file upload, and integer overflow are the recurring failure modes. The paper presents CoV-Eval and VC-Judge as the reliable instruments that surface these patterns.

Load-bearing premise

VC-Judge's labels are treated as ground truth for grading all 20 models, but the judge was itself trained partly on CoV-Eval programs, including 216 LLM-generated codes and the same 531-program detection set it later helps grade, so any systematic bias in VC-Judge shifts every security rate, repair score, and model ranking reported in the tables.

Editorial extensions

If this is right

  • A model that detects vulnerabilities well can still generate them often, so detection accuracy alone is not evidence that a code assistant is secure to deploy.
  • Proprietary models remain more secure than open-source models on average, but code-specific fine-tuning closes part of the gap: specialized code models beat their general counterparts in completion security and overall security score.
  • Training on secure, vulnerability-free code data can improve generation security without harming functional usability, while vulnerability-detection instruction data specifically improves repair ability.
  • The recurring weak types, command injection, dangerous file upload, and integer overflow, offer concrete targets for security hardening in the next generation of models.
  • A fine-tuned judge with judgment-style prompts can grade generated code at scale with human-aligned rankings, replacing manual review for benchmark scoring.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Another implication the paper leaves implicit is that the 25-50 percent single-draw vulnerability rate likely understates real risk in interactive assistants: when a user sees several sampled completions, the chance that at least one insecure snippet appears grows quickly, so the paper's SR@1 measure could be extended to SR@k.
  • Because VC-Judge's training data overlaps with the benchmark it grades, a natural next step is to publish per-instance judge labels and run a holdout human audit; without that, the fine-grained CWE-type statistics are the least certain numbers in the paper.
  • The judge's prompt design, testing one specific vulnerability type per call instead of open multi-class naming, looks transferable: feeding such judgment labels back into training, rather than raw code, could become a cheap security filter for other code-generation benchmarks.
  • The benchmark's task structure also suggests a diagnostic: comparing a model's security rate on completions against its own recall on self-generated code would directly measure whether security knowledge learned through detection transfers to generation time.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes CoV-Eval, a multi-task benchmark for evaluating the code security of large language models. It covers four tasks: code completion, vulnerability repair, vulnerability detection, and vulnerability classification, built on the Copilot-CWE seed set and a new Vul-Evol synthesis framework. The authors also introduce VC-Judge, a fine-tuned LLAMA3-8B-Instruct model designed to automatically judge whether generated code is vulnerable, and report a study of 20 proprietary and open-source LLMs. The main empirical findings are that most LLMs detect vulnerabilities with relatively high recall but still generate insecure code, classify vulnerability types poorly, repair vulnerabilities with limited success, and that fine-tuning on secure code data can improve both security and usability.

Significance. If the results were fully robust, CoV-Eval would be a useful multi-task benchmark for code security evaluation, and VC-Judge could offer a scalable alternative to human review. The paper has concrete strengths: the benchmark construction is transparent, the dataset and code are released, the human-alignment comparison in Table 5 is informative, and the fine-tuning experiments in Section 6.3 are a valuable addition. However, two load-bearing issues — the extreme class imbalance in the detection test set and the circular training/evaluation use of VC-Judge — currently undermine the headline claim that LLMs 'identify vulnerable codes well' and the reliability of the reported generative scores. The significance of the paper therefore depends on addressing these issues.

major comments (3)
  1. [§3.2, Appendix A.1, Table 1] The vulnerability detection test set contains 477 vulnerable and only 54 non-vulnerable programs (89.8% positive). A trivial always-positive classifier achieves positive-class F1 94.6%, recall 100%, and weighted F1 85.0%. Several entries in Table 1 are effectively at this ceiling: Qwen1.5-14B-chat (94.64/100.00), ChatGLM3-6B (94.64/100.00), CodeShell-7B-chat (94.54/99.79), and GPT-4o (94.62/99.58). Since precision and accuracy are not reported, the abstract's claim that 'most LLMs identify vulnerable codes well' is not supported: a model that labels everything as vulnerable would produce numbers very close to the reported ones. The authors should re-report detection results with precision, balanced accuracy, MCC, or F1 relative to a majority-class baseline, and revise the qualitative conclusions accordingly.
  2. [§4.1, Table 7, §5, §6.3] VC-Judge's instruction-tuning dataset includes 216 LLM-generated CoV-Eval programs and the 531-program CoV-Eval detection test set. The same VC-Judge is then used as the primary scorer for code completion and repair in Tables 1 and 3, and as the filter for the SC-IFT data whose effect is later measured with the same judge in Section 6.3. This creates a leak: VC-Judge has seen programs from the same 54 seed scenarios it later grades, so the human-alignment numbers in Table 5 and the SR@1 values in Tables 1, 3, and 4 may be optimistically biased. The authors should train VC-Judge without any CoV-Eval-origin data, evaluate on a held-out subset of scenarios or vulnerability types disjoint from the training data, or validate a sample of judged completions against fresh human annotations.
  3. [§5.1, Tables 1–3] Each LLM is evaluated with a single inference per test instance (temperature 0.6, top-p 0.9 for open models), and every reported number is a point estimate from one run, with no confidence intervals, bootstrap estimates, or multiple seeds. LLM sampling is stochastic, so the close rankings in Table 1 (e.g., CodeLLAMA-13B vs CodeLLAMA-7B in classification F1) and the comparative findings in Section 5.2 may reflect noise. The authors should provide variance estimates, run multiple seeds for at least the main tables, or explicitly justify why a single draw is sufficient for the paper's comparative claims.
minor comments (6)
  1. [References] The reference 'Antropic. 2024. Claude 3.5 sonnet' contains a typo; it should be 'Anthropic'.
  2. [Appendix B] The description for CWE-306 is listed as 'Insufficient Processing of Invalid or Unintended Input', but Figure 8 gives the correct standard name 'Missing Authentication for Critical Function' for CWE-306. Please reconcile the two.
  3. [Table 4] The row 'CodeLLAMA-7B-Instructt' has a typo and is set without a clear line break from the preceding row, making the table hard to parse.
  4. [Figure 4] Figure 4 uses 'Index of LLMs' on the x-axis but does not provide a legend mapping indices to model names; please add a mapping or label the axis with model names.
  5. [§5.2 finding (4)] The phrase 'From LLAMA2 to CodeLLAMA and then to LLAMA3 and LLAMA3.1' suggests a linear model lineage; CodeLLAMA is a separate code fine-tune of LLAMA2, not a successor in the LLAMA family. Please clarify.
  6. [§3.4] The text says discrimination task outputs are evaluated with 'regular matching' but does not specify the exact extraction rules; please state how keywords are extracted and normalized (e.g., case, leading zeros in CWE identifiers).

Circularity Check

2 steps flagged · score 6.0 of 10

Generative security scores are partly self-referential: VC-Judge is trained on CoV-Eval data and then used to score CoV-Eval, and the SC-IFT 'secure code' filter and the SR@1 metric are the same VC-Judge decision.

  1. fitted input called prediction [Section 4.1 (VC-Judge training data), Table 7; Section 5.1 (evaluation setup)]
    "We collected vulnerable and non-vulnerable programs from three sources: (1) code completion test in CoV-Eval. We asked three master students to annotate vulnerabilities for 216 selected LLM-generated programs; (2) vulnerability detection test set of CoV-Eval (531 programs); (3) Programs in open-source vulnerability detection datasets BigVul ... Notablly, each LLM performs inference once on test sets, and then we employ VC-Judge to determine whether LLM-generated codes contain specific types of vulnerabilities."

    VC-Judge is instruction-tuned on CoV-Eval's own labels: 216 LLM-generated code-completion programs and the 531-program CoV-Eval detection test set. The same VC-Judge is then used as the scorer for the CoV-Eval code-completion and vulnerability-repair SR@1 results. The reported generative security rates are therefore not an independent measurement; they are produced by a judge fitted to the very benchmark it is grading. This is not a full label-for-label equality, but it removes the out-of-sample character of the evaluation and makes the 'reliable automated evaluation' claim partially self-referential.

  2. self definitional [Section 6.3 (Data Preparation) and Appendix H; Section 3.4 / Section 5.1 (SR@1 metric)]
    "To ensure code security, we utilized VC-Judge for code auditing, retaining only those labeled as 'Non-vulnerable.' ... we employ VC-Judge to determine whether LLM-generated codes contain specific types of vulnerabilities."

    The 'secure code' in SC-IFT is defined as code that VC-Judge labels Non-vulnerable, and the Security Rate (SR@1) used to measure the benefit of SC-IFT is the proportion of outputs that the same VC-Judge labels Non-vulnerable. Thus the CoV-Eval improvement in Table 4 is, by construction, partly the model learning VC-Judge's decision boundary and then being scored by that exact boundary. The CyberSecEval column provides some independent evidence, but the central CoV-Eval claim that high-quality secure code data improves code security is circular on this metric.

full rationale

The detection and classification results in Table 1 are computed by regular-matching LLM responses against ground-truth labels, so those numbers are not themselves produced by the fitted judge; however, the detection test set is 89.8% vulnerable (477 vulnerable vs 54 non-vulnerable), so an always-positive classifier reaches positive-class F1 94.6% and several Table 1 entries sit at that ceiling. That is a serious benchmark-validity threat but is a correctness/metric issue rather than a derivation-circularity issue under the stated rules. The genuine circularity is concentrated in the generative-evaluation loop: VC-Judge is trained partly on CoV-Eval's own programs and labels and then used as the scorer for CoV-Eval SR@1, and the SC-IFT data filter and the SR@1 metric are the same VC-Judge label decision. The external CyberSecEval benchmark and the human-consistency analysis in Section 6.4 supply some independent evidence, so the paper is not wholly circular, but the headline 'LLMs tend to generate insecure codes' and the SC-IFT conclusion are partially forced by the fitted-judge/self-defined-metric loop. No load-bearing self-citation chain or imported uniqueness theorem was found.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The benchmark's quantitative claims rest on three sets of choices: inherited seed labels, hand-picked generation settings with single sampling, and a learned judge partially trained on the data it evaluates. No external, independently validated test set is used to confirm the headline numbers.

free parameters (3)
  • VC-Judge prompt template selection = Direct w/o Analysis
    In Appendix K and Table 9, the authors select the prompt template that produces the smallest difference from human experts on the human-annotated subset; choosing the evaluator configuration on the evaluation set affects all reported security rates.
  • Decoding hyperparameters = temperature 0.6, top_p 0.9
    Chosen by hand in Appendix G for open-source models; because each model generates only one completion per scenario, these settings directly shape the SR@1 point estimates.
  • Non-vulnerable sample count for detection test set = 54
    Randomly selected from 610 non-vulnerable seed programs to balance the 19-class test set (Appendix A.1); the specific 54 programs chosen can affect detection and classification F1.
assumptions (4)
  • domain assumption The seed set labels from Pearce et al. are correct ground truth for the 477 vulnerable and 54 non-vulnerable programs used in detection and repair tasks.
    All detection and classification metrics and repair targets inherit these labels without re-validation (Section 3.1).
  • domain assumption Each Vul-Evol scenario's assigned CWE type matches the vulnerability it actually induces in model completions.
    Quality filtering uses GPT-4o and student judgment to ensure this, but it is a probabilistic claim rather than a verified property of every generated completion (Section 3.3 and Appendix D).
  • domain assumption A single inference per model per scenario is a stable estimate of the model's security rate.
    SR@1 is computed from one completion per scenario (Section 5.1); repeated sampling or temperature variation could produce different rates.
  • ad hoc to paper VC-Judge's binary labels reliably reflect human security judgments for code generated by all 20 models.
    VC-Judge is a fine-tuned LLaMA-3-8B used as the primary evaluator; its consistency with humans is measured on a small subset (Table 5), and the authors acknowledge in the Limitations section that it still falls short of human experts.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can You Really Trust Code Copilots? Evaluating Large Language Models from a Code Security Perspective." pith.science (2026). https://pith.science/paper/SWUEFP6F

@misc{pith2026250510494,
  author       = {Pith},
  title        = {Pith review of: Can You Really Trust Code Copilots? Evaluating Large Language Models from a Code Security Perspective},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SWUEFP6F}},
  note         = {Machine review of arXiv:2505.10494}
}
read the original abstract

Code security and usability are both essential for various coding assistant applications driven by large language models (LLMs). Current code security benchmarks focus solely on single evaluation task and paradigm, such as code completion and generation, lacking comprehensive assessment across dimensions like secure code generation, vulnerability repair and discrimination. In this paper, we first propose CoV-Eval, a multi-task benchmark covering various tasks such as code completion, vulnerability repair, vulnerability detection and classification, for comprehensive evaluation of LLM code security. Besides, we developed VC-Judge, an improved judgment model that aligns closely with human experts and can review LLM-generated programs for vulnerabilities in a more efficient and reliable way. We conduct a comprehensive evaluation of 20 proprietary and open-source LLMs. Overall, while most LLMs identify vulnerable codes well, they still tend to generate insecure codes and struggle with recognizing specific vulnerability types and performing repairs. Extensive experiments and qualitative analyses reveal key challenges and optimization directions, offering insights for future research in LLM code security.

Figures

Figures reproduced from arXiv: 2505.10494 by the authors.

Figure 1
Figure 1. The illustration of vulnerable codes gener [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The process of dataset construction and automated evaluation. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. VC-Judge training process. that do not include security features or declarations as the Vul-Evol Set. For more details for artifi￾cial check and prompt templates used in the data synthesis process, please refer to Appendix D. 3.4 Evaluation Metrics In the CoV-Eval benchmark, we adopt the “Security Rate (SR)” as the evaluation metric for both code completion and vulnerability repair tasks, which indicates the proport… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: The relative order of scores for 20 different [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Comparison of security rates of different [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Some samples in the seed set. Task: Follow the instruction and complete the following code. There is a comment line "//-copilot next line-" in the given code. Please complete the code next to this line. Note: Your output should include the provided leading code and you…
Figure 7
Figure 7. Figure 7: The demonstration of the prompts we use for different test tasks. [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Descriptions and examples of 18 common vulnerability types. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Prompt templates used for GPT-4o automatic vulnerable code scenarios synthesis. We first make slight [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Prompt templates used for quality filtering in Vul-Evol data synthesis process. [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Demonstration of two code scenarios from [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 14
Figure 14. Figure 14: Case study. The program on the left has multiple vulnerability types, and the program on the right has an unexpected vulnerability types [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]
Figure 15
Figure 15. Figure 15: The demonstration of the prompt templates we use for evaluation. [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Illusion of Secure LLM Code: Closing the Security Gap via Iterative Reprompting

    cs.CR 2026-07 conditional novelty 5.5 of 10

    Across five coding assistants, authentication code is insecure under basic or generic-secure prompts; single-shot NIST help improves it, but only iterative reprompting approaches defense-in-depth.

Reference graph

Works this paper leans on

51 extracted references · 19 canonical work pages · cited by 1 Pith paper

  1. [1]

    Antropic. 2024. https://www.anthropic.com/news/claude-3-5-sonnet Claude 3.5 sonnet

    work page 2024

  2. [2]

    Cyrille Artho and Armin Biere. 2005. Combined static and dynamic analysis. Electronic Notes in Theoretical Computer Science, 131:3--14

    work page 2005

  3. [3]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732

    arXiv 2021

  4. [4]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenhang Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, K. Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, ...

    arXiv 2023

  5. [5]

    Manish Bhatt, Sahana Chennabasappa, Cyrus Nikolaidis, Shengye Wan, Ivan Evtimov, Dominik Gabi, Daniel Song, Faizan Ahmad, Cornelius Aschermann, Lorenzo Fontana, et al. 2023. Purple llama cyberseceval: A secure coding benchmark for language models. arXiv preprint arXiv:2312.04724

    arXiv 2023

  6. [6]

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiao wen Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, Zhen Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shua...

    arXiv 2024

  7. [7]

    Linzheng Chai, Shukai Liu, Jian Yang, Yuwei Yin, Ke Jin, Jiaheng Liu, Tao Sun, Ge Zhang, Changyu Ren, Hongcheng Guo, et al. 2024. Mceval: Massively multilingual code evaluation. arXiv preprint arXiv:2406.07436

    arXiv 2024

  8. [8]

    Sahil Chaudhary. 2023. Code alpaca: An instruction-following llama model for code generation. https://github.com/sahil280114/codealpaca

    2023

Show all 51 references
  1. [9]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    2021 arXiv

  2. [10]

    Bandit Developers. 2022 a . https://bandit.readthedocs.io/en/latest/ Bandit

    2022

  3. [11]

    CodeQL Developers. 2022 b . https://github.com/github/codeql Codeql

    2022

  4. [12]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  5. [13]

    Jiahao Fan, Yi Li, Shaohua Wang, and Tien N Nguyen. 2020 a . Ac/c++ code vulnerability dataset with code changes and cve summaries. In Proceedings of the 17th International Conference on Mining Software Repositories, pages 508--512

    2020

  6. [14]

    Jiahao Fan, Yi Li, Shaohua Wang, and Tien Nhut Nguyen. 2020 b . https://api.semanticscholar.org/CorpusID:221784842 A c/c++ code vulnerability dataset with code changes and cve summaries . 2020 IEEE/ACM 17th International Conference on Mining Software Repositories (MSR), pages 508--512

    2020

  7. [15]

    Zeyu Gao, Hao Wang, Yuchen Zhou, Wenyu Zhu, and Chao Zhang. 2023. How far have we gone in vulnerability detection using large language models. arXiv preprint arXiv:2311.12420

    2023 arXiv

  8. [16]

    Mat \' as F Gobbi and Johannes Kinder. 2023. Poster: Using codeql to detect malware in npm. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pages 3519--3521

    2023

  9. [17]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. 2024. Deepseek-coder: When the large language model meets programming--the rise of code intelligence. arXiv preprint arXiv:2401.14196

    2024 arXiv

  10. [18]

    Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L'elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, T...

    2023 arXiv

  11. [19]

    Lijun Li, Bowen Dong, Ruohui Wang, Xuhao Hu, Wangmeng Zuo, Dahua Lin, Yu Qiao, and Jing Shao. 2024 a . Salad-bench: A hierarchical and comprehensive safety benchmark for large language models. arXiv preprint arXiv:2402.05044

    2024 arXiv

  12. [20]

    Zongjie Li, Zhibo Liu, Wai Kin Wong, Pingchuan Ma, and Shuai Wang. 2024 b . Evaluating c/c++ vulnerability detectability of query-based static application security testing tools. IEEE Transactions on Dependable and Secure Computing

    2024

  13. [21]

    Stephan Lipp, Sebastian Banescu, and Alexander Pretschner. 2022. An empirical study on the effectiveness of static c code analyzers for vulnerability detection. In Proceedings of the 31st ACM SIGSOFT international symposium on software testing and analysis, pages 544--555

    2022

  14. [22]

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. 2024 a . Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434

    2024 arXiv

  15. [23]

    Yu Liu, Lang Gao, Mingxin Yang, Yu Xie, Ping Chen, Xiaojin Zhang, and Wei Chen. 2024 b . Vuldetectbench: Evaluating the deep capability of vulnerability detection with large language models. arXiv preprint arXiv:2406.07595

    2024 arXiv

  16. [24]

    Zhijie Liu, Yutian Tang, Xiapu Luo, Yuming Zhou, and Liang Feng Zhang. 2024 c . No need to lift a finger anymore? assessing the quality of code generation by chatgpt. IEEE Transactions on Software Engineering

    2024

  17. [25]

    Panagiotis Louridas. 2006. Static code analysis. Ieee Software, 23(4):58--61

    2006

  18. [26]

    Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568

    2023 arXiv

  19. [27]

    Sanoop Mallissery and Yu-Sung Wu. 2023. Demystify the fuzzing methods: A comprehensive survey. ACM Computing Surveys, 56(3):1--38

    2023

  20. [28]

    Valentin JM Man \`e s, HyungSeok Han, Choongwoo Han, Sang Kil Cha, Manuel Egele, Edward J Schwartz, and Maverick Woo. 2019. The art, science, and engineering of fuzzing: A survey. IEEE Transactions on Software Engineering, 47(11):2312--2331

    2019

  21. [29]

    Stefan Nagy and Matthew Hicks. 2019. Full-speed fuzzing: Reducing fuzzing overhead through coverage-guided tracing. In 2019 IEEE Symposium on Security and Privacy (SP), pages 787--802. IEEE

    2019

  22. [30]

    Ana Nunez, Nafis Tanveer Islam, Sumit Kumar Jha, and Peyman Najafirad. 2024. Autosafecoder: A multi-agent framework for securing llm code generation through static analysis and fuzz testing. arXiv preprint arXiv:2409.10737

    2024 arXiv

  23. [31]

    OpenAI. 2024. https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/ Gpt-4o mini: advancing cost-efficient intelligence

    2024

  24. [32]

    Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2022. Asleep at the keyboard? assessing the security of github copilot’s code contributions. In 2022 IEEE Symposium on Security and Privacy (SP), pages 754--768. IEEE

    2022

  25. [33]

    Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. https://api.semanticscholar.org/CorpusID:257985497 Instruction tuning with gpt-4 . ArXiv, abs/2304.03277

    2023 arXiv

  26. [34]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950

    2023 arXiv

  27. [35]

    Mohammed Latif Siddiq and Joanna CS Santos. 2022. Securityeval dataset: mining vulnerability examples to evaluate machine learning-based code generation techniques. In Proceedings of the 1st International Workshop on Mining Software Repositories Applications for Privacy and Se...

    2022

  28. [36]

    Benjamin Steenhoek, Md Mahbubur Rahman, Monoshi Kumar Roy, Mirza Sanjida Alam, Earl T Barr, and Wei Le. 2024. A comprehensive study of the capabilities of large language models for vulnerability detection. arXiv preprint arXiv:2403.17218

    2024 arXiv

  29. [37]

    Darko Stefanovi \'c , Danilo Nikoli \'c , Du s anka Daki \'c , Ivana Spasojevi \'c , and Sonja Risti \'c . 2020. Static code analysis tools: A systematic literature review. In Ann. DAAAM Proc. Int. DAAAM Symp, volume 31, pages 565--573

    2020

  30. [38]

    Catherine Tony, Markus Mutas, Nicol \'a s E D \' az Ferreyra, and Riccardo Scandariato. 2023. Llmseceval: A dataset of natural language prompts for security evaluations. In 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR), pages 588--592. IEEE

    2023

  31. [39]

    Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M

    Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cant \'o n Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes,...

    2023 arXiv

  32. [40]

    Petar Tsankov, Mohammad Torabi Dashti, and David Basin. 2012. Secfuzz: Fuzz-testing security protocols. In 2012 7th International Workshop on Automation of Software Test (AST), pages 1--7. IEEE

    2012

  33. [41]

    Jiexin Wang, Liuwen Cao, Xitong Luo, Zhiping Zhou, Jiayuan Xie, Adam Jatowt, and Yi Cai. 2023. Enhancing large language models for secure code generation: A dataset-driven study on vulnerability mitigation. arXiv preprint arXiv:2310.16263

    2023 arXiv

  34. [42]

    Michel Wermelinger. 2023. Using github copilot to solve simple programming problems. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1, pages 172--178

    2023

  35. [43]

    Rui Xie, Zhengran Zeng, Zhuohao Yu, Chang Gao, Shikun Zhang, and Wei Ye. 2024. Codeshell technical report. arXiv preprint arXiv:2403.15747

    2024 arXiv

  36. [44]

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244

    2023 arXiv

  37. [45]

    Weixiang Yan, Haitian Liu, Yunkun Wang, Yunzhe Li, Qian Chen, Wen Wang, Tingyu Lin, Weishan Zhao, Li Zhu, Hari Sundaram, and Shuiguang Deng. 2024. https://doi.org/10.18653/v1/2024.acl-long.301 C ode S cope: An execution-based multilingual multitask multidimensional benchmark f...

    2024 doi

  38. [46]

    Zhang, Yuxiao Dong, and Jie Tang

    Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, P. Zhang, Yuxiao Dong, and Jie Tang. 2022. https://api.semanticscholar.org/CorpusID:252715691 Glm...

    2022 arXiv

  39. [47]

    Weihao Zeng, Can Xu, Yingxiu Zhao, Jian-Guang Lou, and Weizhu Chen. 2024. Automatic instruction evolving for large language models. arXiv preprint arXiv:2406.00770

    2024 arXiv

  40. [48]

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. http://arxiv.org/abs/2403.13372 Llamafactory: Unified efficient fine-tuning of 100+ language models . In Proceedings of the 62nd Annual Meeting of the Association for Compu...

    2024 arXiv

  41. [49]

    Xin Zhou, Ting Zhang, and David Lo. 2024. Large language model for vulnerability detection: Emerging results and future directions. In Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results, pages 47--51

    2024

  42. [50]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  43. [51]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.