Pith. sign in

REVIEW 5 major objections 6 minor 6 cited by

SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Code

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read State-of-the-art LLMs leave more than 60% of their generated code flagged as vulnerable under ordinary zero-shot prompting, according to the SafeGenBench benchmark's dual-judge evaluation of 13 models.

desk verdict A genuinely broader security benchmark whose headline numbers contradict each other: §6.4's 61.12% both-secure is mathematically impossible alongside Table 3's 37.44% overall and 42.61% LLM-Judge averages. read the letter →

arxiv 2506.05692 v3 pith:WD2SZFTU submitted 2025-06-06 cs.CR cs.AI

Xinghang Li , Jingzhe Ding , Chao Peng , Bing Zhao , Xiang Gao , Hongwan Gao , Xinchen Gu This is my paper
classification cs.CRcs.AI
keywords LLMcodegenerationsecuresecuritybenchmarkCWEvulnerabilitySASTLLM-as-a-judgedetectionpromptsafety
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces SafeGenBench, a benchmark that asks whether LLM-generated code is secure, not just whether it runs. It contains 558 realistic code-generation prompts spanning 12 languages and 44 Common Weakness Enumeration (CWE) vulnerability types, and it evaluates model output with both a static-analysis scanner and an LLM judge; a sample is counted as secure only if both judges pass it. On this benchmark, 13 state-of-the-art models averaged 37.44% overall accuracy under zero-shot prompting, meaning the majority of generated code was flagged as vulnerable. Adding a safety reminder raised the average to 58.01%, and adding few-shot insecure examples raised it to 61.14%. A sympathetic reader would take this as evidence that secure code generation is still a weak point of current models, and that prompt-level security guidance is a cheap but incomplete mitigation.

What carries the argument

The machinery is the SafeGenBench dataset plus a dual-judge evaluation pipeline. The dataset, built from OWASP Top 10 and CWE Top 25, organizes 44 CWE identifiers into 8 categories and uses human-reviewed, LLM-written prompts that avoid security terminology, so the model is not tipped off. After code extraction, the SAST judge (Semgrep) scans broadly for any vulnerability and gives score 1 only if no ERROR or CRITICAL finding appears, while the LLM judge (DeepSeek-R1) inspects only the task-specified CWE and returns 0 or 1. The final verdict is the logical AND of the two judges: any flag means insecure. This structure lets the benchmark measure both the model's default security behavior and the effect of safety prompting.

What would settle it

Take the 558 prompts, run all 13 models again, and have two independent human security experts label every output for the relevant CWE rather than a 9% sample; if human-model agreement drops below the reported 92%, or if the judge's labels vary across repeated runs for particular models or languages, the absolute secure rate of 37.44% and the per-model rankings would need revision.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is empirical: when 13 leading open- and closed-source LLMs are asked to solve 558 diverse programming tasks without any security reminder, the average fraction of outputs that pass both judges as vulnerability-free is 37.44%. The benchmark's strict definition—secure only if both the SAST judge and the LLM judge score 1—means that roughly 63% of generated samples carry at least one detected vulnerability. The paper also shows that prompting matters: a safety instruction lifts average accuracy above 58%, and few-shot insecure examples add about three more points. Finally, the dual judges catch different weaknesses: the LLM judge flagged 30.05% of samples that SAST missed, while SAST caught 6.24% that the LLM judge missed, which the paper uses to justify its combined evaluation.

Load-bearing premise

The headline number depends on DeepSeek-R1 judging all 44 CWE types in 12 languages accurately and without favoring its own outputs, even though the judge was validated against human experts on only 9% of the benchmark.

Editorial extensions

If this is right

  • Without safety prompting, a typical state-of-the-art model produces code flagged as vulnerable on over 60% of SafeGenBench tasks, so directly integrating generated code into a product without review carries substantial risk.
  • Adding a one-line safety reminder raises average secure accuracy by more than 20 percentage points, meaning prompt design is a cheap and effective lever for security.
  • Few-shot examples of insecure code add about 3 more points on average, a smaller but consistent gain across models.
  • Reasoning models (o3, DeepSeek-R1, o1, QWQ-32B) beat non-reasoning models in all three settings, suggesting chain-of-thought style training helps security alignment.
  • Memory-safety tasks are comparatively easy (76.16% average) while insecure configuration tasks are hardest (12.50% average), so model security competence is uneven across vulnerability categories.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the prompts were translated into English, per-model scores could shift because the current questions are in Chinese and models differ in Chinese proficiency; the paper flags language as a limitation, so 37.44% is best read as a Chinese-prompt result.
  • Because DeepSeek-R1 serves as both an evaluated model and the judge, a judge-ensemble or full human audit would be needed to rule out self-preference; this is an extension, not a claim the paper makes.
  • The benchmark could serve as a regression suite: rerunning these 558 tasks after security fine-tuning would quantify whether a new model version actually improves its vulnerability-free rate.
  • The strict AND-scoring rule likely overstates exploitable risk, since one judge flag can mark a sample insecure even when the flaw is latent or unreachable; pairing the benchmark with exploitability validation would separate true security failures from code hygiene issues.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. This paper introduces SafeGenBench, a benchmark for evaluating whether LLM-generated code contains security vulnerabilities. The dataset consists of 558 test questions spanning 12 programming languages and 44 CWE vulnerability types, organized into 8 categories. Test questions are LLM-generated and then human-annotated. The evaluation framework uses a dual-judge setup: Semgrep as a SAST judge scanning all vulnerability types and DeepSeek-R1 as an LLM judge checking the task-specified CWE, with a sample scored secure only if both judges assign 1. The paper evaluates 13 models under zero-shot, zero-shot-with-safety-instruction, and few-shot prompting, reporting an average zero-shot overall accuracy of 37.44% and substantial improvements with safety instructions and few-shot examples. The authors claim that state-of-the-art LLMs exhibit notable deficiencies in producing vulnerability-free code.

Significance. The benchmark addresses a real gap: existing code-security benchmarks (CodeLMSec, CWEval, SecRepoBench) cover fewer languages and CWEs and rely on a single evaluation method. The dual-judge design is a sensible attempt to combine the breadth of SAST with the semantic depth of LLM judgment, and the human-annotation stage adds quality assurance. If the numbers are correct and the artifacts are released, SafeGenBench would be a useful resource. However, the quantitative conclusions are not currently established: there is an internal contradiction between the framework-consistency section and the main results table, the LLM judge is validated on a small sample and is also used to score its own family of models, and the dataset and evaluation code are not yet public. The paper's contribution is conditional on resolving these issues.

major comments (5)
  1. [§6.4, Table 3] The claim that 61.12% of zero-shot samples were deemed secure by both judges is mathematically incompatible with Table 3. Under the scoring definition in §4.1, a generated sample is secure iff both judges assign a score of 1, so the 'both secure' rate equals the Overall column. The zero-shot Overall average in Table 3 is 37.44%, and the LLM-Judge column averages 42.61% with a maximum of 51.79% (o3). Because 'both secure' is a subset of 'LLM-Judge = 1', the both-secure rate cannot exceed the LLM-Judge rate; the reported 61.12% violates this bound by roughly 9–24 percentage points depending on the aggregation. The four percentages in §6.4 (61.12 + 30.05 + 6.24 + 2.59 = 100) also imply that the SAST-Judge flags 8.83% of samples and the LLM-Judge flags 32.64%, contradicting the SAST-Judge average of 89.28% and the LLM-Judge average of 42.61% in Table 3. The authors must provide per-sample judge outputs and correct whichever table is wrong; until then, the headline 37.44% zero-shot secure rate and the framework-consistency claim in §6.4 cannot both be interpreted.
  2. [§5, §6.4] Using DeepSeek-R1 as the unified judge while also evaluating DeepSeek-R1 creates a self-scoring risk that is not addressed. The validation in §6.4 is based on only a 9% stratified sample (about 50 cases) and reports a single overall accuracy of 92% with a wide binomial confidence interval (81.2%–96.8%). No per-CWE or per-language accuracy is reported, so this validation cannot support reliable binary labels across 44 CWEs and 12 languages. Moreover, the validation does not test whether the judge is systematically lenient or strict toward code generated by DeepSeek-R1 itself. The authors should report judge accuracy by CWE category and by evaluated model, and should re-run the evaluation with at least one additional independent judge (e.g., a different model) to verify that the rankings and absolute secure rates in Table 3 are stable.
  3. [§3.1.1, Table 2 vs. Table 6] The paper says the dataset covers 44 CWEs and Table 2 lists exactly 44 CWE IDs, but Table 6, which is described as providing the full names for those IDs, lists 49 distinct CWE IDs. Extra IDs in Table 6 include CWE-74, CWE-98, CWE-915, CWE-601, and CWE-489; some of these appear in the SAST findings in §6.2 (e.g., CWE-915 and CWE-601). This mismatch affects a central quantitative claim of the benchmark. Either the dataset actually covers 49 CWEs and Table 2 and the abstract should be updated, or the extra CWE rows should be removed from Table 6 and the SAST-detected CWEs should be reconciled with the stated taxonomy.
  4. [Abstract, §5] The dataset, evaluation scripts, and detailed configurations are not released ('will be released soon'), and the exact Semgrep rule set, Semgrep version, and the generation prompts for the zero-shot, safety-instruction, and few-shot settings are not specified. For a benchmark paper, these artifacts are the primary contribution; without them, Table 3 cannot be reproduced or extended. The authors should release the full dataset, raw per-sample binary scores from both judges, the exact Semgrep configuration (including rule files and severity thresholds), and the complete prompt templates for all evaluation settings.
  5. [Table 3, §6.1] All accuracies in Table 3 appear to come from a single run per model/setting, with no confidence intervals or multiple sampling seeds. With 558 test cases, the standard error of a 37% proportion is roughly 2 percentage points, so the differences among top models (e.g., o3 at 74.91% vs. DeepSeek-R1 at 74.19% in the few-shot setting) are within sampling noise. The authors should report binomial confidence intervals or multiple runs, and should not draw fine-grained ordering conclusions without such support.
minor comments (6)
  1. [Throughout] Typographical errors: 'OW ASP' should be 'OWASP' throughout; 'Claude-3.7-Soonet' should be 'Claude-3.7-Sonnet'; 'LLMS' in the Limitations should be 'LLMs'; '65,43' in Table 4 should be '65.43'.
  2. [Appendix F] The sentence at the end of Appendix F says the complete CWE list is 'summarized in Table5'; the reference appears to point to the wrong table and should be corrected.
  3. [Limitations] The Limitations section states that the test questions are written in Chinese, but the examples in Appendices B and C are shown in English. The authors should clarify the actual language of the prompts and any translation process.
  4. [Figures 4 and 5] The y-axis labels of Figures 4 and 5 are inconsistent ('Average Frequency (%)' vs. 'Average LLM-Judge Score (%)'); the figures should state explicitly whether the numbers are averaged per model or over all generated samples.
  5. [§6.2] In §6.2, CWE-1104 and CWE-778 are described as 'emphasized' by the LLM-Judge with accuracies of 8.79% and 11.54%; since lower accuracy means the judge flags these CWEs more often, the wording should clarify this direction.
  6. [Appendix D] The LLM judge prompt in Appendix D uses CWE-401 ('Memory Leak') as an example, but CWE-401 is not among the 44 CWEs in the benchmark taxonomy; replace it with an example drawn from the actual taxonomy.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark and evaluation are externally grounded; the LLM-judge self-reference and a Section 6.4/Table 3 numerical inconsistency are validity or reporting concerns, not definitional reductions.

full rationale

SafeGenBench is an empirical benchmark rather than a derivation, so most circularity patterns do not apply. The dataset questions are LLM-generated but then human-annotated, filtered, and quality-assured (Section 3.1.3), providing external grounding for the benchmark inputs. The evaluation uses Semgrep plus a DeepSeek-R1 judge whose labels were validated against expert-curated ground truth on a stratified 9% sample (Section 6.4, 92% accuracy, 95% binomial CI 81.2-96.8), so the judge is not merely reproducing the paper's own conclusions. The fact that DeepSeek-R1 serves both as judge and as one of the 13 evaluated models is a real measurement-bias risk, especially for its own row, but it is not circular by construction: the judge was not fitted to Table 3, and the SAST judge provides an independent signal. The only overlapping-author citation (Liu et al. 2024b, cited in Section 1 alongside several other productivity references) is not load-bearing. The paper's own Limitations section acknowledges single-judge and single-SAST limitations. Separately, there is a non-circular reporting inconsistency: Section 6.4 reports 61.12% both-secure in zero-shot, while Section 4.1 defines overall accuracy as both judges scoring 1 and Table 3 reports a 37.44% zero-shot average (LLM-Judge average 42.61%). Since the both-secure rate cannot exceed the LLM-Judge rate, the two numbers cannot both describe the same evaluation; this is a correctness issue that should be resolved, not a circular step.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

SafeGenBench is a new benchmark artifact, but it is not an invented natural entity such as a particle or force. The ledger therefore focuses on hand-chosen scoring thresholds and domain assumptions required by the evaluation. The benchmark's empirical conclusions depend on prompt validity, judge reliability, and the representativeness of the CWE taxonomy, none of which are externally verified in this preprint.

free parameters (2)
  • SAST severity threshold = ERROR/CRITICAL -> 0; WARNING/INFO -> 1
    Hand-chosen mapping in Section 4.3; no sensitivity analysis is provided, and it determines whether SAST findings count as insecure.
  • LLM-Judge security score rule = Score 0 only for a clear unprotected vulnerability; partial protections receive 1
    Defined in Section 4.4 and Appendix D; treating partial protections as secure can inflate reported secure rates.
assumptions (4)
  • domain assumption Semgrep rule coverage is sufficient to detect most non-task-specific vulnerabilities across 12 languages.
    Section 4.3 relies on Semgrep without publishing the rule set, version, or configuration; Appendix C shows known misses.
  • domain assumption LLM judge accuracy on a 9% expert-validated sample generalizes to all 558 cases and all 44 CWEs.
    Section 6.4 reports 92% accuracy on a stratified sample; no per-CWE or per-language breakdown, and no external test set.
  • domain assumption Test prompts accurately represent real developer requests without leaking the intended vulnerability type.
    Section 3.1.2 states prompts are generated by LLMs to avoid security terminology, then human-reviewed; the dataset is not released for independent inspection.
  • domain assumption The CWE taxonomy and assignment of each test case to one vulnerability category is correct.
    Section 3.1.1 builds the taxonomy from OWASP and CWE with expert analysis; the case-to-CWE mapping is not published.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Code." pith.science (2026). https://pith.science/paper/WD2SZFTU

@misc{pith2026250605692,
  author       = {Pith},
  title        = {Pith review of: SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Code},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WD2SZFTU}},
  note         = {Machine review of arXiv:2506.05692}
}
read the original abstract

The code generation capabilities of large language models(LLMs) have emerged as a critical dimension in evaluating their overall performance. However, prior research has largely overlooked the security risks inherent in the generated code. In this work, we introduce SafeGenBench, a benchmark specifically designed to assess the security of LLM-generated code. The dataset encompasses a wide range of common software development scenarios and vulnerability types. Building upon this benchmark, we develop an automatic evaluation framework that leverages both static application security testing(SAST) and LLM-based judging to assess the presence of security vulnerabilities in model-generated code. Through the empirical evaluation of state-of-the-art LLMs on SafeGenBench, we reveal notable deficiencies in their ability to produce vulnerability-free code. Our findings highlight pressing challenges and offer actionable insights for future advancements in the secure code generation performance of LLMs. The data and code will be released soon.

Figures

Figures reproduced from arXiv: 2506.05692 by the authors.

Figure 1
Figure 1. Overview of the construction process of SafeGenBench Dataset. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Automatic evaluation system for SafeGen [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Overall accuracy(%) of representative models from major AI providers on SafeGenBench under 3 [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Top-10 most frequent CWE types within all [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Top-10 CWE types with the lowest accuracy [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: LLM judge system prompt 17 [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Illusion of Safety: Multi-Tier Verification of AI vs. Human C++ Code

    cs.SE 2026-06 unverdicted novelty 7.0 of 10

    On 8,918 competitive-programming C++ programs, AI code roughly doubles the odds of sanitizer-confirmed runtime violations versus human code; static analysis alone masks the gap.

  2. The Patchwork Problem in LLM-Generated Code

    cs.SE 2026-07 conditional novelty 6.5 of 10

    Structural failures in LLM-generated code form a distinct class of defects that standard type checking, tests, and SAST miss, and models produce qualitatively different failure profiles.

  3. Poster: Rethinking Security in LLM Code Generation through Real-World Risk Scenarios

    cs.CR 2026-07 conditional novelty 6.0 of 10

    In realistic ambiguous, underspecified, or conflict-laden prompts, all eight tested LLMs produced vulnerable code in over 56% of outputs, and security-aware prompting reduced vulnerability by up to 45%.

  4. Adversarial Stress Testing of Role-Playing Language Agents using Multi-Agent Evaluation

    cs.AI 2026-08 conditional novelty 5.0 of 10

    A multi-agent adversarial evaluation platform with six progressive attack strategies shows that role-playing LLMs degrade under sustained pressure, with automated judging correlating with human ratings.

  5. Extracting Recurring Vulnerabilities from Black-Box LLM-Generated Software

    cs.CR 2026-02 reject novelty 5.0 of 10

    Frontend features of LLM-generated apps can predict hidden backend vulnerabilities that a given model tends to reproduce, enabling black-box attack triage.

  6. The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs

    cs.CL 2025-06 conditional novelty 3.0 of 10

    A structured survey of LLM safety evaluation that proposes a why/what/where/how taxonomy and catalogs metrics, datasets, benchmarks, evaluators, and frameworks.

Reference graph

Works this paper leans on

39 extracted references · 14 canonical work pages · cited by 6 Pith papers

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Anthropic. 2024. https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf The claude 3 model family: Opus, sonnet, haiku

    2024

  3. [3]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and 1 others. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732

    arXiv 2021

  4. [4]

    Manish Bhatt, Sahana Chennabasappa, Yue Li, Cyrus Nikolaidis, Daniel Song, Shengye Wan, Faizan Ahmad, Cornelius Aschermann, Yaohui Chen, Dhaval Kapil, and 1 others. 2024. Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for large language models. arXiv preprint arXiv:2404.13161

    arXiv 2024

  5. [5]

    Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, and 1 others. 2023. Multipl-e: a scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software Engineering, 49(7):3675--3691

    work page 2023

  6. [6]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    arXiv 2021

  7. [7]

    Connor Dilgren, Purva Chiniya, Luke Griffith, Yu Ding, and Yizheng Chen. 2025. Secrepobench: Benchmarking llms for secure code generation in real-world repositories. arXiv preprint arXiv:2504.21205

    arXiv 2025

  8. [8]

    Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. 2023. Crosscodeeval: a diverse and multilingual benchmark for cross-file code completion. In Proceedings of the 37th International Conference on Neural Information Processing Systems, N...

    work page 2023

Show all 39 references
  1. [9]

    Ahmed El-Kishky, Alexander Wei, Andre Saraiva, Borys Minaiev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera, Jakub Pachocki, and 1 others. 2025. Competitive programming with large reasoning models. arXiv preprint arXiv:2502.06807

    2025 arXiv

  2. [10]

    Matteo Esposito, Valentina Falaschi, and Davide Falessi. 2024. An extensive comparison of static application security testing tools. In Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering, pages 69--78

    2024

  3. [11]

    Google Cloud . 2025. https://cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-5-pro Gemini 2.5 Pro Model Card

    2025

  4. [12]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, and 1 others. 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594

    2024 arXiv

  5. [13]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    2025 arXiv

  6. [14]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, and 1 others. 2024. Deepseek-coder: When the large language model meets programming--the rise of code intelligence. arXiv preprint arXiv:2401.14196

    2024 arXiv

  7. [15]

    Hossein Hajipour, Keno Hassler, Thorsten Holz, Lea Schonherr, and Mario Fritz. 2024. https://doi.org/10.1109/SaTML59370.2024.00040 CodeLMSec Benchmark: Systematically Evaluating and Finding Security Vulnerabilities in Black-Box Code Language Models . In 2024 IEEE Conference on...

    2024

  8. [16]

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and 1 others. 2021. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938

    2021 arXiv

  9. [17]

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, and 1 others. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720

    2024 arXiv

  10. [18]

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974

    2024 arXiv

  11. [19]

    Xuefeng Jiang, Lvhua Wu, Sheng Sun, Jia Li, Jingjing Xue, Yuwei Wang, Tingting Wu, and Min Liu. 2024. Investigating large language models for code vulnerability detection: An experimental study. arXiv preprint arXiv:2412.18260

    2024 arXiv

  12. [20]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. https://openreview.net/forum?id=VTF8yNQM66 SWE -bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Re...

    2024

  13. [21]

    Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2023. Ds-1000: a natural and reliable benchmark for data science code generation. In Proceedings of the 40th International Conference on Machine ...

    2023

  14. [22]

    Jinfeng Li. 2020. Vulnerabilities mapping based on owasp-sans: A survey for static application security testing (sast). Annals of Emerging Technologies in Computing (AETiC), 4(3):1--8

    2020

  15. [23]

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, R \'e mi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, and 1 others. 2022. Competition-level code generation with alphacode. Science, 378(6624):1092--1097

    2022

  16. [24]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024 a . Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

    2024 arXiv

  17. [25]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. In Proceedings of the 37th International Conference on Neural Information Processing System...

    2023

  18. [26]

    Yizhou Liu, Pengfei Gao, Xinchen Wang, Jie Liu, Yexuan Shi, Zhao Zhang, and Chao Peng. 2024 b . Marscode agent: Ai-native automated bug fixing. arXiv preprint arXiv:2409.00899

    2024 arXiv

  19. [27]

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, and 1 others. 2024. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173

    2024 arXiv

  20. [28]

    Meta AI . 2025. https://ai.meta.com/blog/llama-4-multimodal-intelligence/ The Llama 4 herd: The beginning of a new era of natively multimodal ai innovation . Meta AI Blog. Online; accessed 2025-05-15

    2025

  21. [29]

    MITRE Corporation . 2024. https://cwe.mitre.org/top25/archive/2024/2024-top25.html Cwe top 25 most dangerous software weaknesses . Accessed: 2025-05-15

    2024

  22. [30]

    OpenAI . 2025. https://openai.com/index/o3-o4-mini-system-card Openai o3 and o4‑mini system card

    2025

  23. [31]

    OWASP Foundation . 2021. https://owasp.org/www-project-top-ten/ Owasp top 10: The ten most critical web application security risks . Accessed: 2025-05-15

    2021

  24. [32]

    Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2025. Asleep at the keyboard? assessing the security of github copilot’s code contributions. Communications of the ACM, 68(2):96--105

    2025

  25. [33]

    Jinjun Peng, Leyi Cui, Kele Huang, Junfeng Yang, and Baishakhi Ray. 2025. Cweval: Outcome-driven evaluation on functionality and security of llm code generation. arXiv preprint arXiv:2501.08200

    2025 arXiv

  26. [34]

    Qwen Team . 2025 a . https://qwenlm.github.io/blog/qwen3/ Qwen3: Think deeper, act faster

    2025

  27. [35]

    Qwen Team . 2025 b . https://qwenlm.github.io/blog/qwq-32b/ QwQ-32B : Embracing the power of reinforcement learning

    2025

  28. [36]

    Anton Ryd\' e n, Erik N\" a slund, Elad Michael Schiller, and Magnus Almgren. 2024. https://doi.org/10.1007/978-3-031-76934-4_7 Llmseccode: Evaluating large language models for secure coding . In Cyber Security, Cryptology, and Machine Learning: 8th International Symposium, CS...

    2024 doi

  29. [37]

    Shaznin Sultana, Sadia Afreen, and Nasir U Eisty. 2024. Code vulnerability detection: A comparative analysis of emerging large language models. arXiv preprint arXiv:2409.10490

    2024 arXiv

  30. [38]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  31. [39]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.