REVIEW 20 cited by
Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The integration of large language models with external content has enabled applications such as Microsoft Copilot but also introduced vulnerabilities to indirect prompt injection attacks. In these attacks, malicious instructions embedded within external content can manipulate LLM outputs, causing deviations from user expectations. To address this critical yet under-explored issue, we introduce the first benchmark for indirect prompt injection attacks, named BIPIA, to assess the risk of such vulnerabilities. Using BIPIA, we evaluate existing LLMs and find them universally vulnerable. Our analysis identifies two key factors contributing to their success: LLMs' inability to distinguish between informational context and actionable instructions, and their lack of awareness in avoiding the execution of instructions within external content. Based on these findings, we propose two novel defense mechanisms-boundary awareness and explicit reminder-to address these vulnerabilities in both black-box and white-box settings. Extensive experiments demonstrate that our black-box defense provides substantial mitigation, while our white-box defense reduces the attack success rate to near-zero levels, all while preserving the output quality of LLMs. We hope this work inspires further research into securing LLM applications and fostering their safe and reliable use.
Forward citations
Cited by 20 Pith papers
-
DualView: Preventing Indirect Prompt Injection in Personal AI Agents
DualView extends Dual-LLM symbol isolation into the shared user environment via dual Agent/Human views, blocking both immediate and stored IPI at 0% ASR while preserving near-baseline utility.
-
Information Discernment in Large Language Models
LLMs update their stated numeric beliefs almost regardless of source reliability or whether a claim moves them closer to the truth, performing near chance on both dimensions.
-
MNC: Scope-Bound Semantic Declassification for Private LLM-Agent Communication
MNC binds LLM-agent disclosures to recipient, purpose, lifetime, logging, memory, and forwarding scopes, and its monitor enforces them, cutting excess inference leakage from internal agent channels.
-
Beyond Aggregate Risk: Role-Stratified Conformal Risk Control for LLM Tool Calls
Role-stratified per-field conformal risk control gives each semantic argument role its own threshold and risk budget, so rare high-risk fields in LLM tool calls are certified individually rather than averaged away.
-
ContainmentBench: Trace-Based Evaluation of Post-Injection Containment in Tool-Using LLM Agents
Terminal policy labels are insufficient: two containment policies with identical zero-harm endpoints still differ in 73.5% of trajectories and in authorized-work completion.
-
PARSE: Provenance-Aware Retrieval Sanitization for Professional Domain LLM Agents
PARSE reduces prompt injection success rate from 25.4% to 15.6% on a 122-task real-document benchmark across five professional domains while retaining 86.9% utility, unlike paraphrasing which shows no benefit.
-
Chain-of-Models: Cross-Model Auditing for Bias-Robust LLM Judges
Per-bias selection of a cross-family LLM auditor lifts biased-judgment accuracy from 0.805/0.824 baselines to 0.884.
-
WebGuard: Building a Generalizable Guardrail for Web Agents
WebGuard introduces an action-level risk dataset for web agents and shows that a fine-tuned 7B model improves risk-prediction accuracy from about 38% to 80% and high-risk recall from 20% to 76%, still below deployment...
-
Context manipulation attacks : Web agents are susceptible to corrupted memory
Web agents with protected prompts can still be hijacked by injecting malicious steps into their stored task plans, reaching up to 63% success on privacy leaks.
-
A Critical Evaluation of Defenses against Prompt Injection Attacks
StruQ, SecAlign, Instruction Hierarchy, PromptGuard, and Attention Tracker are substantially less effective and utility-preserving than claimed when evaluated with diverse prompts and adaptive attacks.
-
Invisible Prompts, Visible Threats: Malicious Font Injection in External Resources for Large Language Models
Maliciously crafted fonts can hide adversarial prompts from human eyes while LLMs read and obey them, enabling content manipulation and data exfiltration via MCP tools.
-
When Benchmarks Lie: Evaluating Malicious Prompt Classifiers Under True Distribution Shift
Standard train/test evaluation overstates prompt-attack classifier generalization by 8+ AUC points; leave-one-dataset-out reveals dataset shortcuts that CV hides.
-
Lexical Hints of Accuracy in LLM Reasoning Chains
Hesitation words in reasoning chains are claimed to flag incorrect LLM answers, but the manuscript body is a different paper and contains no such study.
-
LLM Agents Should Employ Security Principles
A position paper proposing AgentSandbox, a framework that applies Saltzer-Schroeder security principles to LLM agents and reports large attack-success-rate reductions on AgentDojo.
-
Ranking Free RAG: Replacing Re-ranking with Selection in RAG for Sensitive Domains
METEORA uses DPO-tuned rationales to select and verify evidence chunks in RAG, and claims better recall, precision, evidence efficiency, and poisoning defense, though key evaluation details are missing.
-
EVA: Evolving Semantic Adversaries for Red-Teaming GUI Agents Against Environmental Injection Attacks
EVA evolves environmental injection payloads through a keyword-utility loop, achieving up to 80% attack success in the body's pop-up tests, while the abstract's stronger claims are not supported by the reported experiments.
-
Prompt-in-Content Attacks: Exploiting Uploaded Inputs to Hijack LLM Behavior
Embedding a short 'system instruction' in a .docx file causes several commercial LLMs to refuse, substitute, redirect, or bias their output during summarization tasks.
-
BLOCKS: Blockchain-supported Cross-Silo Knowledge Sharing for Efficient LLM Services
BLOCKS combines a Cosmos-based blockchain, a reputation mechanism, and a priority cache to let LLMs retrieve prompts from untrusted knowledge silos.
-
JavelinGuard: Low-Cost Transformer Architectures for LLM Security
A study of five small transformer classifier architectures for LLM jailbreak and prompt injection detection claims low-latency accuracy comparable to large models, led by the multi-task Raudra design.
-
Prompt Injection 2.0: Hybrid AI Threats
A structured taxonomy of hybrid prompt injection attacks shows how XSS, CSRF, and SQL injection vectors converge with LLM manipulation to bypass traditional controls.
Discussion (0). Sign in to comment.