REVIEW 6 cited by
Attention Tracker: Detecting Prompt Injection Attacks in LLMs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large Language Models (LLMs) have revolutionized various domains but remain vulnerable to prompt injection attacks, where malicious inputs manipulate the model into ignoring original instructions and executing designated action. In this paper, we investigate the underlying mechanisms of these attacks by analyzing the attention patterns within LLMs. We introduce the concept of the distraction effect, where specific attention heads, termed important heads, shift focus from the original instruction to the injected instruction. Building on this discovery, we propose Attention Tracker, a training-free detection method that tracks attention patterns on instruction to detect prompt injection attacks without the need for additional LLM inference. Our method generalizes effectively across diverse models, datasets, and attack types, showing an AUROC improvement of up to 10.0% over existing methods, and performs well even on small LLMs. We demonstrate the robustness of our approach through extensive evaluations and provide insights into safeguarding LLM-integrated systems from prompt injection vulnerabilities.
Forward citations
Cited by 6 Pith papers
-
Robust Context-Aware Detection of Malicious Instructions in Text
A query- and context-aware sentence detector with adversarial training provides a tunable utility-versus-security defense against indirect prompt injection in LLM agents.
-
CrackedPDFs: A Controlled Benchmark for Hidden Prompt Injection in PDFs
A document-aware hybrid detector that inspects PDF structure before text flattening outperforms text-only guardrails and structural-only models on a new 29,322-file controlled hidden-prompt-injection benchmark.
-
LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systems
A systematic review that categorizes LLM threats, severity scores, and mitigations across development and operation life cycles and multiple deployment scenarios.
-
A Critical Evaluation of Defenses against Prompt Injection Attacks
StruQ, SecAlign, Instruction Hierarchy, PromptGuard, and Attention Tracker are substantially less effective and utility-preserving than claimed when evaluated with diverse prompts and adaptive attacks.
-
MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agents
MELON detects indirect prompt injection in LLM agents by re-executing each step with the user prompt replaced by a neutral task prompt and flagging attacks when the original and masked runs produce similar tool calls.
-
PromptArmor: Simple yet Effective Prompt Injection Defenses
PromptArmor prompts a separate LLM to detect and remove injected prompts from agent inputs, achieving below 1% false positive and false negative rates on AgentDojo.
Discussion (0). Continue with ORCID to comment.