Pith. sign in

REVIEW 4 major objections 6 minor 3 cited by

Human-Written vs. AI-Generated Code: A Large-Scale Study of Defects, Vulnerabilities, and Complexity

T0 review · 4 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AI-generated code is simpler and more repetitive than human code, yet more prone to unused constructs, leftover debug code, and high-severity security vulnerabilities such as command injection and hardcoded credentials.

desk verdict The biggest, best-documented human-vs-AI code quality comparison to date, but the headline claims are confounded by length and context and need normalization before they fully hold up. read the letter →

arxiv 2508.21634 v1 pith:2CIEN7AC submitted 2025-08-29 cs.SE

Domenico Cotroneo , Cristina Improta , Pietro Liguori This is my paper
classification cs.SE
keywords AIcodegenerationOrthogonalDefectClassificationCommonWeaknessEnumerationsecurityvulnerabilitiesstaticanalysiscomplexitylargelanguagemodelsPythonandJavaquality
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether code written by human developers and code produced by large language models differ in ways that matter for software quality, and answers yes. It compares more than 500,000 Python and Java functions—human functions mined from open-source repositories against functions that three LLMs generated from the same docstrings—and classifies every static-analysis finding using two standard taxonomies, one for defects and one for security weaknesses. The paper finds that AI-generated functions are shorter, less complex, and more repetitive, yet they carry more unused constructs, leftover debug code, and high-risk vulnerabilities such as command injection and hardcoded credentials. Human functions are longer and more complex, with a defect profile dominated by maintainability, algorithmic, and exception-handling issues. If the comparison holds, AI-assisted development needs its own quality-assurance and security-review practices rather than a direct transplant of the ones built for human code.

What carries the argument

The paired-generation design carries the argument: each human function's own docstring and signature become the prompt for three LLMs, so every author produces an answer to the same specification. On top of that, two normalization layers make the counts comparable across languages and authors: Orthogonal Defect Classification maps tool-specific rule violations from Python and Java static analyzers into a common set of defect types, and the Common Weakness Enumeration maps security-rule hits into a common catalog of weakness types. Without these layers, raw static-analyzer counts would not be commensurable.

What would settle it

Re-run the same static-analysis pipeline on length-matched and complexity-matched subsets of human and AI functions, for example only functions of 4–8 lines with cyclomatic complexity 1–2, or human functions stripped of their surrounding project context. If the defect-profile and vulnerability-rate gaps shrink to statistical insignificance, the authorship effect is an artifact of comparing short isolated snippets with longer contextual functions; if they persist, the paper's conclusion survives.

Watch

Extended reading notes

Core claim

On the paper's own terms: author identity leaves a systematic fingerprint on code. When defects are normalized through Orthogonal Defect Classification (a taxonomy sorting bugs by type—assignment, algorithm, interface, checking, timing, structure) and security findings through the Common Weakness Enumeration, AI-generated code clusters in the assignment and class-structure categories (unused arguments, unused variables, placeholder classes), while human code clusters in algorithm, interface, and checking categories (complex control flow, protected-member access, weak exception handling). AI code triggers high-severity CWE entries—OS command injection, uncontrolled resource consumption, hardc

Load-bearing premise

The comparison treats a human function extracted from a real repository, with surrounding project context, as the same unit as an AI function generated as an isolated snippet from a docstring, and compares raw violation counts without adjusting for the large length and complexity differences; if context or length drives violation rates, the authorship effect is confounded.

Editorial extensions

If this is right

  • AI-generated code will not 'look like' human code to existing quality gates: the same rules will fire on different categories, so teams cannot assume human-calibrated thresholds transfer.
  • Security review of AI-generated code should prioritize injection, resource-consumption, hardcoded-secret, and information-exposure weaknesses, which are the categories where the gap is largest.
  • Because AI functions are shorter and simpler, complexity-based quality metrics will systematically understate their risk; simple code is not automatically safe code.
  • Model choice matters: the three LLMs differ sharply in syntax failure rate, defect profile, and vulnerability rate, so 'AI-generated code' is not a single category for QA purposes.
  • The quality gap is wider in Java than in Python, suggesting that language strictness amplifies the defects LLMs introduce.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Since AI functions average roughly half the lines of human functions, per-line or per-token vulnerability density could be even higher than the per-sample differences reported; computing rates normalized by length would sharpen the comparison.
  • The massive unused-argument counts across all three models suggest LLMs preserve prompt signatures without binding parameters to the body; varying signatures while holding docstrings fixed could test whether this is a generation-strategy artifact.
  • The human baseline comes from mature open-source projects, so its maintainability-heavy profile partly reflects accumulated project debt; sampling first-commit or context-free human snippets might narrow—or widen—the gap and isolate the authorship effect.
  • Static-analysis rule sets may need reweighting for AI code: rules that dominate human code can swamp counts, while rules that matter most for AI code (unused constructs, debug output) are underweighted in typical quality dashboards.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents a large-scale comparison of human-written and AI-generated code (ChatGPT, DeepSeek-Coder, Qwen-Coder) across Python and Java, using Pylint/PMD for defect detection, Semgrep for security vulnerabilities, and structural complexity metrics (NLOC, CCN, tokens, unique tokens). Defects are mapped to Orthogonal Defect Classification (ODC) and vulnerabilities to CWE. The dataset pairs 507,044 docstrings with human functions from CodeSearchNet/HMCorp and LLM-generated implementations. The central claims are that AI code is simpler and more repetitive, with more unused constructs and hardcoded debugging, while human code is more complex and maintainability-challenged; and that AI code contains more high-risk vulnerabilities. The paper includes a replication package and documents the rule-to-ODC mapping.

Significance. If the results are valid, the paper is significant: it provides a large-scale, cross-language, multi-model characterization of AI- versus human-written code quality and security, with practical implications for QA practices and for the design of static analysis tools. Strengths include the scale (over 500k samples, ~17k repositories), the use of two standard taxonomies (ODC and CWE), the multiple LLMs, and the public replication artifacts. The documented two-stage consensus mapping of tool rules to ODC is a positive methodological feature. However, the central comparison is currently confounded by task format and code length, so the headline conclusions are not yet established.

major comments (4)
  1. [§IV and §V-A, Tables III–V] The central unit of comparison is not equivalent. Human functions are mined from 16,928 real repositories with surrounding imports, classes, call sites, and project conventions, while AI functions are generated from docstrings and signatures as isolated snippets. Table V shows human functions average 12.72–13.38 NLOC versus 4.47–8.42 for AI functions. Raw defect counts are not volume-adjusted, and no stratified analysis or regression control by length/complexity is reported. Since the prompts include the original signature, rules such as unused-argument (up to 213,264 for Qwen in Python) may be triggered by prompt design rather than by an author-specific defect propensity. This confound affects every comparison in Tables III–V and Figures 1–2, including the abstract's claim that AI code is 'more prone to unused constructs'. I recommend reporting defect densities per function and per NLOC
  2. [§V-B, Table IV and Figure 2] The security comparison uses Semgrep's syntactic pattern matching on isolated snippets, with no manual validation of true positives. High-risk CWEs such as CWE-78 (OS Command Injection), CWE-798 (Hardcoded Credentials), and CWE-489 (Active Debug Code) may be over-reported when functions are generated standalone from docstrings that contain examples or configuration-like text. The claim that AI code contains 'more high-risk security vulnerabilities' depends on these raw counts. The authors should validate a random sample of the flagged vulnerabilities (stratified by CWE and author) and report precision; otherwise the observed differences may reflect the generation task rather than a stable security profile.
  3. [§III-A and §V-A, Table III] The handling of 'incorrect' samples needs to be made precise and its effect on the comparisons assessed. For DSC, the paper reports 20,783 incorrect Python samples and 34,382 incorrect Java samples, mostly empty or unparseable outputs, while humans have 3,712 and 245. These samples are excluded from defect counts, so the reported 'defective sample' percentages are not directly comparable across authors. The paper should state whether the percentages in Table III are computed over all samples or only parseable ones, and should analyze sensitivity to the exclusion of incorrect outputs (e.g., by treating empty outputs as a distinct outcome).
  4. [§VI, Threats to Validity] The threats section acknowledges the static-analysis and context limitations, but it does not test whether they change the conclusions. Specifically, the internal-validity paragraph focuses on generation consistency and does not address the human/AI task-format asymmetry. Since the confound is load-bearing, the paper needs a concrete robustness analysis (e.g., comparing only functions of similar length, or re-running the defect analysis on a context-matched subsample) rather than a general acknowledgment.
minor comments (6)
  1. [§V-A, Fig. 1] Figure 1 captions are identical for (a) and (b) and do not state the language; the language is only in the subplot labels. Please make the captions self-contained.
  2. [§V-A, Table III] The table uses blue/red for best/worst scores, but these colors are not accessible in grayscale and are not described in the text. Consider adding symbols or a separate column.
  3. [§IV, Table II] The meaning of 'Avg. Code Len.' is ambiguous (characters? tokens? lines?). Define it in the table caption or text.
  4. [§V-A] The text says DSC Python has 19,612 empty predictions, while Table III reports 20,783 incorrect samples for DSC Python. Clarify the relationship between 'empty' and 'incorrect' and make the numbers consistent.
  5. [§III-A] The ODC mapping table (Table I) would benefit from an explicit statement of inter-rater agreement (e.g., Cohen's kappa) rather than only 'disagreements were resolved through discussion'.
  6. [§V-C, Table V] The unique-token metric is computed per author across the entire corpus, but the text describes it as a per-function metric. Clarify the aggregation unit in the metric definition (§III-C).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: all central results are empirical comparisons against external static-analysis tools and standard taxonomies.

full rationale

The paper's derivation chain is empirical rather than definitional. The dataset pairs human functions from CodeSearchNet/HMCorp with LLM outputs generated from the same docstrings and signatures (Section IV). Defect counts come from Pylint and PMD, security findings from Semgrep, and complexity metrics from lizard and tiktoken (Sections III and V). No parameter is fitted to the outcome and then renamed as a prediction; the reported distributions could, in principle, have favored either author class. The ODC and CWE mappings are interpretive but are applied uniformly and are not constructed so as to force the conclusions. Self-citations in the paper (e.g., prior work on AI-code vulnerabilities and training-data quality, plus the replication package) are contextual or artifact pointers and are not load-bearing premises for the main comparisons. The strongest concern is a construct-validity issue: human functions come from real project contexts and are longer than isolated AI-generated snippets, so raw counts may be confounded with task format and length. This is a validity threat, not circular reasoning, and the authors partially acknowledge it in Section VI. Accordingly, no circular step meets the standard of quoting a specific reduction of a result to its own inputs.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No model is fit; the study is a descriptive comparison. The only researcher-chosen quantities are rule exclusion thresholds (350/226 rules, 12 class-related rules excluded) and the manual ODC mapping, which are categorical choices rather than fitted numeric parameters. No new entities are postulated; ODC and CWE are existing taxonomies.

assumptions (4)
  • domain assumption Static-analysis rule violations detected by Pylint, PMD, and Semgrep are valid and comparable indicators of defects and vulnerabilities across human and AI code.
    Central to RQ1/RQ2; the paper relies on these tools without runtime validation (Section III-A, III-B; also acknowledged as a limitation in Section VI).
  • domain assumption Human functions in CodeSearchNet and LLM-generated functions from their docstrings are comparable units despite differences in integration context, length, and cleaning.
    The whole human-vs-AI comparison in Section V assumes this comparability; it is only partially addressed in Section VI.
  • domain assumption The manual rule-to-ODC mapping is reliable; consensus between two authors is sufficient.
    Section III-A describes the two-stage mapping; no inter-rater agreement metric is reported.
  • ad hoc to paper Wrapping Java functions in dummy classes and excluding 12 class-related PMD rules preserves function-level comparability.
    Section III-A; this setup-specific decision affects all Java defect counts, but it is introduced to satisfy PMD parsing requirements.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Human-Written vs. AI-Generated Code: A Large-Scale Study of Defects, Vulnerabilities, and Complexity." pith.science (2026). https://pith.science/paper/2CIEN7AC

@misc{pith2026250821634,
  author       = {Pith},
  title        = {Pith review of: Human-Written vs. AI-Generated Code: A Large-Scale Study of Defects, Vulnerabilities, and Complexity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2CIEN7AC}},
  note         = {Machine review of arXiv:2508.21634}
}
read the original abstract

As AI code assistants become increasingly integrated into software development workflows, understanding how their code compares to human-written programs is critical for ensuring reliability, maintainability, and security. In this paper, we present a large-scale comparison of code authored by human developers and three state-of-the-art LLMs, i.e., ChatGPT, DeepSeek-Coder, and Qwen-Coder, on multiple dimensions of software quality: code defects, security vulnerabilities, and structural complexity. Our evaluation spans over 500k code samples in two widely used languages, Python and Java, classifying defects via Orthogonal Defect Classification and security vulnerabilities using the Common Weakness Enumeration. We find that AI-generated code is generally simpler and more repetitive, yet more prone to unused constructs and hardcoded debugging, while human-written code exhibits greater structural complexity and a higher concentration of maintainability issues. Notably, AI-generated code also contains more high-risk security vulnerabilities. These findings highlight the distinct defect profiles of AI- and human-authored code and underscore the need for specialized quality assurance practices in AI-assisted programming.

Figures

Figures reproduced from arXiv: 2508.21634 by the authors.

Figure 1
Figure 1. Distribution of ODC defect types by code author. [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Heatmap of top-10 CWEs distribution across code [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Bridging the Gap on AI-Assisted Scientific Software Development Through Transparency and Traceability

    cs.SE 2026-05 conditional novelty 6.0 of 10

    Proposes guidance for responsible AI use in scientific software development under NQA-1 standards, illustrated with TMAP8 V&V cases to ensure accountability and auditability.

  2. Towards an Agent Operating System - Lessons from Classical and Cloud OS

    cs.AI 2026-07 conditional novelty 5.0 of 10

    Agentic systems need an Agent-OS of thirteen OS/Cloud-OS-derived primitives with precise platform-enforceable semantics before portable production adoption is possible.

  3. Specification-Driven Development as the Foundation of AI-Native Enterprise Software Engineering

    cs.SE 2026-07 conditional novelty 5.0 of 10

    Specification-Driven Development, formalized as the SGRM reference model, is argued to be the necessary governance layer for enterprise AI-native software engineering, with the strongest quantitative evidence still un...

Reference graph

Works this paper leans on

42 extracted references · 28 canonical work pages · cited by 3 Pith papers

  1. [1]

    Ai-driven development is here: Should you worry?

    N. A. Ernst and G. Bavota, “Ai-driven development is here: Should you worry?” IEEE Software, vol. 39, no. 2, pp. 106–110, 2022

    work page 2022

  2. [2]

    Open AI ChatGPT,

    Open AI, “Open AI ChatGPT,” https://chatgpt.com

    work page

  3. [3]

    GitHub Copilot,

    GitHub, “GitHub Copilot,” https://copilot.github.com

    work page

  4. [4]

    Deepseek-coder: When the large language model meets programming–the rise of code intelligence,

    D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y . Wu, Y . Li et al., “Deepseek-coder: When the large language model meets programming–the rise of code intelligence,” arXiv preprint arXiv:2401.14196, 2024

    arXiv 2024

  5. [5]

    Automatic programming: Large language models and beyond,

    M. R. Lyu, B. Ray, A. Roychoudhury, S. H. Tan, and P. Thongtanunam, “Automatic programming: Large language models and beyond,” ACM Transactions on Software Engineering and Methodology , 2024

    work page 2024

  6. [6]

    Anthropic CEO Dario Amodei, https://www.inc.com/joe-procopio/ anthropics-ceo-said-all-code-will-be-ai-generated-in-a-year/91163367

    work page arXiv

  7. [7]

    Codereval: A benchmark of pragmatic code generation with generative pre-trained models,

    H. Yu, B. Shen, D. Ran, J. Zhang, Q. Zhang, Y . Ma, G. Liang, Y . Li, Q. Wang, and T. Xie, “Codereval: A benchmark of pragmatic code generation with generative pre-trained models,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering , 2024, pp. 1–12

    work page 2024

  8. [8]

    Test-driven development and llm- based code generation,

    N. S. Mathews and M. Nagappan, “Test-driven development and llm- based code generation,” in Proceedings of the 39th IEEE/ACM Interna- tional Conference on Automated Software Engineering , 2024

    work page 2024

Show all 42 references
  1. [9]

    Evaluating the code quality of ai-assisted code generation tools: An empirical study on github copilot, amazon codewhisperer, and chatgpt,

    B. Yetis ¸tiren, I. ¨Ozsoy, M. Ayerdem, and E. T ¨uz¨un, “Evaluating the code quality of ai-assisted code generation tools: An empirical study on github copilot, amazon codewhisperer, and chatgpt,” arXiv preprint arXiv:2304.10778, 2023

    2023 arXiv

  2. [10]

    Refining chatgpt-generated code: Characterizing and mitigating code quality issues,

    Y . Liu, T. Le-Cong, R. Widyasari, C. Tantithamthavorn, L. Li, X.- B. D. Le, and D. Lo, “Refining chatgpt-generated code: Characterizing and mitigating code quality issues,” ACM Transactions on Software Engineering and Methodology , vol. 33, no. 5, pp. 1–26, 2024

    2024

  3. [11]

    How secure is ai-generated code: a large-scale comparison of large language models,

    N. Tihanyi, T. Bisztray, M. A. Ferrag, R. Jain, and L. C. Cordeiro, “How secure is ai-generated code: a large-scale comparison of large language models,” Empirical Software Engineering, vol. 30, no. 2, pp. 1–42, 2025

    2025

  4. [12]

    Se- curity weaknesses of copilot generated code in github,

    Y . Fu, P. Liang, A. Tahir, Z. Li, M. Shahin, J. Yu, and J. Chen, “Se- curity weaknesses of copilot generated code in github,” arXiv preprint arXiv:2310.02059, 2023

    2023 arXiv

  5. [13]

    Is stack overflow obsolete? an empirical study of the characteristics of chatgpt answers to stack overflow questions,

    S. Kabir, D. N. Udo-Imeh, B. Kou, and T. Zhang, “Is stack overflow obsolete? an empirical study of the characteristics of chatgpt answers to stack overflow questions,” in Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems , 2024, pp. 1–17

    2024

  6. [14]

    TIOBE Index,

    TIOBE, “TIOBE Index,” https://www.tiobe.com/tiobe-index/

  7. [15]

    Qwen2.5-coder technical report,

    B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Dang et al., “Qwen2.5-coder technical report,” arXiv preprint arXiv:2409.12186, 2024

    2024 arXiv

  8. [16]

    PyLint - Python linter,

    Logilab, “PyLint - Python linter,” https://www.pylint.org

  9. [17]

    Source Code Analyzer,

    PMD, “Source Code Analyzer,” https://pmd.github.io

  10. [18]

    Static Code Analyzer,

    Semgrep, “Static Code Analyzer,” https://github.com/semgrep/semgrep

  11. [19]

    Orthogonal defect classification- a concept for in-process measurements,

    R. Chillarege, I. S. Bhandari, J. K. Chaar, M. J. Halliday, D. S. Moebus, B. K. Ray, and M.-Y . Wong, “Orthogonal defect classification- a concept for in-process measurements,” IEEE Transactions on software Engineering, vol. 18, no. 11, pp. 943–956, 1992

    1992

  12. [20]

    Artifact,

    D. Cotroneo, C. Improta, and P. Liguori, “Artifact,” https://doi.org/10. 5281/zenodo.16853222

  13. [21]

    Replication Package,

    ——, “Replication Package,” https://github.com/dessertlab/Human vs AI Code Quality

  14. [22]

    Vulnerabilities in AI Code Generators: Exploring Targeted Data Poisoning Attacks,

    D. Cotroneo, C. Improta, P. Liguori, and R. Natella, “Vulnerabilities in AI Code Generators: Exploring Targeted Data Poisoning Attacks,” in Proceedings of the 32nd IEEE/ACM International Conference on Program Comprehension, 2024, pp. 280–292

    2024

  15. [23]

    A quantitative analysis of quality and consistency in ai-generated code,

    A. Clark, D. Igbokwe, S. Ross, and M. F. Zibran, “A quantitative analysis of quality and consistency in ai-generated code,” in 2024 7th International Conference on Software and System Engineering (ICoSSE). IEEE, 2024, pp. 37–41

    2024

  16. [24]

    Asleep at the keyboard? assessing the security of github copilot’s code contri- butions,

    H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri, “Asleep at the keyboard? assessing the security of github copilot’s code contri- butions,” Communications of the ACM , vol. 68, no. 2, 2025

    2025

  17. [25]

    Security and quality in llm-generated code: A multi-language, multi-model analysis,

    M. Kharma, S. Choi, M. AlKhanafseh, and D. Mohaisen, “Security and quality in llm-generated code: A multi-language, multi-model analysis,” arXiv preprint arXiv:2502.01853 , 2025

    2025

  18. [26]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374 , 2021

    2021 arXiv

  19. [27]

    Quality assessment of chatgpt generated code and their use by developers,

    M. L. Siddiq, L. Roney, J. Zhang, and J. C. D. S. Santos, “Quality assessment of chatgpt generated code and their use by developers,” in Proceedings of the 21st International Conference on Mining Software Repositories, 2024, pp. 152–156

    2024

  20. [28]

    Quality in, quality out: Investigating training data’s role in ai code generation,

    C. Improta, R. Tufano, P. Liguori, D. Cotroneo, and G. Bavota, “Quality in, quality out: Investigating training data’s role in ai code generation,” in 2025 IEEE/ACM 33rd International Conference on Program Com- prehension (ICPC). IEEE Computer Society, 2025, pp. 454–465

    2025

  21. [29]

    Understanding defects in generated codes by language models,

    A. M. Esfahani, N. Kahani, and S. A. Ajila, “Understanding defects in generated codes by language models,” in 2024 34th International Conference on Collaborative Advances in Software and COmputiNg (CASCON). IEEE, 2024, pp. 1–10

    2024

  22. [30]

    A taxonomy for python vulnerabilities,

    F. C. G. Bogaerts, N. Ivaki, and J. Fonseca, “A taxonomy for python vulnerabilities,” IEEE Open Journal of the Computer Society , 2024

    2024

  23. [31]

    A comparative analysis between ai generated code and human written code: A preliminary study,

    A. Patel, K. Z. Sultana, and B. K. Samanthula, “A comparative analysis between ai generated code and human written code: A preliminary study,” in 2024 IEEE International Conference on Big Data (BigData) . IEEE, 2024, pp. 7521–7529

    2024

  24. [32]

    Common Weakness Enumeration,

    The MITRE Corporation (MITRE), “Common Weakness Enumeration,” https://cwe.mitre.org/

  25. [33]

    GitHub, “CodeQL,” https://github.com/github/codeql

  26. [34]

    PyCQA, “Bandit,” https://github.com/PyCQA/bandit

  27. [35]

    Top 25 Most Dangerous Software Weaknesses,

    MITRE, “Top 25 Most Dangerous Software Weaknesses,” https://cwe. mitre.org/top25/

  28. [36]

    2021 OW ASP Top 10,

    OW ASP, “2021 OW ASP Top 10,” https://owasp.org/Top10/

    2021

  29. [37]

    Yin, “Lizard,” https://github.com/terryyin/lizard

    T. Yin, “Lizard,” https://github.com/terryyin/lizard

  30. [38]

    Tiktoken, fast BPE tokenizer,

    Hauntsaninja, “Tiktoken, fast BPE tokenizer,” https://pypi.org/project/ tiktoken/

  31. [39]

    Distinguishing llm-generated from human-written code by contrastive learning,

    X. Xu, C. Ni, X. Guo, S. Liu, X. Wang, K. Liu, and X. Yang, “Distinguishing llm-generated from human-written code by contrastive learning,” ACM Transactions on Software Engineering and Methodol- ogy, vol. 34, no. 4, pp. 1–31, 2025

    2025

  32. [40]

    CodeSearchNet challenge: Evaluating the state of semantic code search,

    H. Husain, H.-H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt, “CodeSearchNet challenge: Evaluating the state of semantic code search,” arXiv preprint arXiv:1909.09436 , 2019

    1909 arXiv

  33. [41]

    A survey of large language models for code: Evolution, benchmarking, and future trends,

    Z. Zheng, K. Ning, Y . Wang, J. Zhang, D. Zheng, M. Ye, and J. Chen, “A survey of large language models for code: Evolution, benchmarking, and future trends,” arXiv preprint arXiv:2311.10372 , 2023

    2023 arXiv

  34. [42]

    Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,

    Y . Wang, W. Wang, S. Joty, and S. C. Hoi, “Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” arXiv preprint arXiv:2109.00859 , 2021. 12

    2021 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.