Pith. sign in

REVIEW 7 cited by

A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systems

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.18649 v1 pith:634I2C4Y submitted 2024-02-28 cs.CR cs.AI

Fangzhou Wu , Ning Zhang , Somesh Jha , Patrick McDaniel , Chaowei Xiao This is my paper
classification cs.CRcs.AI
keywords securitysystemsconstraintsanalysisattackgpt4individualobjects
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large Language Model (LLM) systems are inherently compositional, with individual LLM serving as the core foundation with additional layers of objects such as plugins, sandbox, and so on. Along with the great potential, there are also increasing concerns over the security of such probabilistic intelligent systems. However, existing studies on LLM security often focus on individual LLM, but without examining the ecosystem through the lens of LLM systems with other objects (e.g., Frontend, Webtool, Sandbox, and so on). In this paper, we systematically analyze the security of LLM systems, instead of focusing on the individual LLMs. To do so, we build on top of the information flow and formulate the security of LLM systems as constraints on the alignment of the information flow within LLM and between LLM and other objects. Based on this construction and the unique probabilistic nature of LLM, the attack surface of the LLM system can be decomposed into three key components: (1) multi-layer security analysis, (2) analysis of the existence of constraints, and (3) analysis of the robustness of these constraints. To ground this new attack surface, we propose a multi-layer and multi-step approach and apply it to the state-of-art LLM system, OpenAI GPT4. Our investigation exposes several security issues, not just within the LLM model itself but also in its integration with other components. We found that although the OpenAI GPT4 has designed numerous safety constraints to improve its safety features, these safety constraints are still vulnerable to attackers. To further demonstrate the real-world threats of our discovered vulnerabilities, we construct an end-to-end attack where an adversary can illicitly acquire the user's chat history, all without the need to manipulate the user's input or gain direct access to OpenAI GPT4. Our demo is in the link: https://fzwark.github.io/LLM-System-Attack-Demo/

Discussion (0). Sign in to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Understanding the Supply Chain and Risks of Large Language Model Applications

    cs.SE 2025-07 conditional novelty 7.0 of 10

    A new benchmark dataset traces dependencies across 3,859 LLM applications, 109,211 models, 2,474 datasets, and 8,862 libraries, and finds widespread known vulnerabilities in application dependencies.

  2. How Do You Choose Your AI Component? An Interview Study of Secure AI Integration in Practice

    cs.SE 2026-07 conditional novelty 6.0 of 10

    In interviews, 22 practitioners chose AI models primarily on functionality, cost, and trust in vendors, with security rarely a formal criterion, suggesting the industry is repeating early software supply chain mistakes.

  3. Direct Behavior Optimization: Unlocking the Potential of Lightweight LLMs

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A gradient-free Monte Carlo tree search over JSON key-step plans produces few-shot demonstrations that let LLaMA3-8B and LLaMA3.2-3B outperform GPT-3.5 on most of seven BIG-Bench Hard tasks.

  4. Turning the Spell Around: Lightweight Alignment Amplification via Rank-One Safety Injection

    cs.CL 2025-08 conditional novelty 5.0 of 10

    ROSI bakes the refusal direction into a model's weight matrices via a rank-one update, raising refusal and jailbreak robustness with minimal measured utility cost.

  5. SAFEFLOW: A Principled Protocol for Trustworthy and Transactional Autonomous Agent Systems

    cs.AI 2025-06 reject novelty 5.0 of 10

    SAFEFLOW wraps LLM/VLM agents in fine-grained information-flow control, verifier-gated trust adjustment, and transactional concurrency, and its authors report near-perfect safety on their own benchmark plus AgentHarm,...

  6. Prompt-in-Content Attacks: Exploiting Uploaded Inputs to Hijack LLM Behavior

    cs.CR 2025-08 conditional novelty 4.0 of 10

    Embedding a short 'system instruction' in a .docx file causes several commercial LLMs to refuse, substitute, redirect, or bias their output during summarization tasks.

  7. A Survey on Model Extraction Attacks and Defenses for Large Language Models

    cs.CR 2025-06 conditional novelty 4.0 of 10

    A taxonomy of model extraction attacks and defenses for large language models, with proposed evaluation metrics and future research directions.

Pith tools