Pith. sign in

REVIEW 7 cited by

Can We Trust Large Language Models Generated Code? A Framework for In-Context Learning, Security Patterns, and Code Evaluations Across Diverse LLMs

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.12513 v1 pith:PK4BG3TB submitted 2024-06-18 cs.CR

Ahmad Mohsin , Helge Janicke , Adrian Wood , Iqbal H. Sarker , Leandros Maglaras , Naeem Janjua This is my paper
classification cs.CR
keywords codellmssecuritygeneratedgenerationlearningresearchsoftware
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large Language Models (LLMs) such as ChatGPT and GitHub Copilot have revolutionized automated code generation in software engineering. However, as these models are increasingly utilized for software development, concerns have arisen regarding the security and quality of the generated code. These concerns stem from LLMs being primarily trained on publicly available code repositories and internet-based textual data, which may contain insecure code. This presents a significant risk of perpetuating vulnerabilities in the generated code, creating potential attack vectors for exploitation by malicious actors. Our research aims to tackle these issues by introducing a framework for secure behavioral learning of LLMs through In-Content Learning (ICL) patterns during the code generation process, followed by rigorous security evaluations. To achieve this, we have selected four diverse LLMs for experimentation. We have evaluated these coding LLMs across three programming languages and identified security vulnerabilities and code smells. The code is generated through ICL with curated problem sets and undergoes rigorous security testing to evaluate the overall quality and trustworthiness of the generated code. Our research indicates that ICL-driven one-shot and few-shot learning patterns can enhance code security, reducing vulnerabilities in various programming scenarios. Developers and researchers should know that LLMs have a limited understanding of security principles. This may lead to security breaches when the generated code is deployed in production systems. Our research highlights LLMs are a potential source of new vulnerabilities to the software supply chain. It is important to consider this when using LLMs for code generation. This research article offers insights into improving LLM security and encourages proactive use of LLMs for code generation to ensure software system safety.

Discussion (0). Sign in to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Insecure Coding Preferences in Long-Term Memory: Security Risks for LLM-based Code Generation

    cs.CR 2026-07 conditional novelty 7.0 of 10

    Injected insecure coding preferences in LLM long-term memory raise vulnerability rates by 2.7-50.3 pp and suppress warnings; memory-level filtering restores safe behavior in the tested set.

  2. HiLDe: Intentional Code Generation via Human-in-the-Loop Decoding

    cs.HC 2025-05 conditional novelty 7.0 of 10

    HiLDe, a code completion UI that exposes and lets users override the LLM's token-level choices, reduced security vulnerabilities in generated code compared to a baseline assistant in a within-subjects study of 18 programmers.

  3. The Illusion of Secure LLM Code: Closing the Security Gap via Iterative Reprompting

    cs.CR 2026-07 conditional novelty 5.5 of 10

    Across five coding assistants, authentication code is insecure under basic or generic-secure prompts; single-shot NIST help improves it, but only iterative reprompting approaches defense-in-depth.

  4. A Causal Perspective on Measuring, Explaining and Mitigating Smells in LLM-Generated Code

    cs.SE 2025-11 conditional novelty 5.0 of 10

    A probabilistic score of code-smell propensity in LLM output is validated, used in a causal analysis, and shown to drop when prompts explicitly discourage known smells.

  5. Curiosity by Design: An LLM-based Coding Assistant Asking Clarification Questions

    cs.AI 2025-07 conditional novelty 4.0 of 10

    A fine-tuned classifier and question generator let a small coding assistant detect under-specified prompts and ask for clarification, which users rated better than a baseline in a small study.

  6. Software Engineering for Large Language Models: Research Status, Challenges and the Road Ahead

    cs.SE 2025-06 conditional novelty 4.0 of 10

    A literature review organizes LLM development into a six-phase software engineering lifecycle and identifies challenges and research directions for each phase.

  7. Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI

    cs.SE 2025-05 conditional novelty 3.0 of 10

    A qualitative taxonomy positions vibe coding and agentic coding as complementary paradigms rather than rivals in AI-assisted software development.

Pith tools