REVIEW 8 cited by
Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
While Large Language Models (LLMs) are increasingly being used in real-world applications, they remain vulnerable to prompt injection attacks: malicious third party prompts that subvert the intent of the system designer. To help researchers study this problem, we present a dataset of over 126,000 prompt injection attacks and 46,000 prompt-based "defenses" against prompt injection, all created by players of an online game called Tensor Trust. To the best of our knowledge, this is currently the largest dataset of human-generated adversarial examples for instruction-following LLMs. The attacks in our dataset have a lot of easily interpretable stucture, and shed light on the weaknesses of LLMs. We also use the dataset to create a benchmark for resistance to two types of prompt injection, which we refer to as prompt extraction and prompt hijacking. Our benchmark results show that many models are vulnerable to the attack strategies in the Tensor Trust dataset. Furthermore, we show that some attack strategies from the dataset generalize to deployed LLM-based applications, even though they have a very different set of constraints to the game. We release all data and source code at https://tensortrust.ai/paper
Forward citations
Cited by 8 Pith papers
-
Transferable Direct Prompt Injection via Activation-Guided MCMC Sampling
An activation-guided energy model plus MCMC sampling creates transferable direct prompt injection attacks in a black-box setting, reaching 49.6% average attack success across five LLMs.
-
Steering Instruction Hierarchies at Inference Time
V-Steer boosts system-prompt value spans and suppresses conflicting user spans, raising controlled instruction-hierarchy adherence from under 18% to as high as 92%.
-
Reasoning Up the Instruction Ladder for Controllable Language Models
RLVR on ~7K verifiable system/user conflict examples teaches LLMs to prioritize higher-priority instructions, improving instruction-hierarchy and safety benchmarks.
-
MetaBreak: Jailbreaking Online LLM Services via Special Token Manipulation
Special tokens that structure LLM conversations can be injected and swapped for lookalike words to bypass both built-in safety and external content filters.
-
Efficient LLM Adversarial Training via Low-Rank Defense and Circuit-Guided Surrogates
Combining suffix-window representation finetuning with an ActGrad-pruned surrogate cuts latent-adversarial-training FLOPs per step by 48.1% with only 0.0118% trainable parameters, while accepting higher attack success rates.
-
Simple Prompt Injection Attacks Can Leak Personal Data Observed by LLM Agents During Task Execution
Prompt injection can make LLM agents leak personal data they observed while executing tasks, with measured attack success rates around 15-20 percent and password leakage much rarer.
-
JavelinGuard: Low-Cost Transformer Architectures for LLM Security
A study of five small transformer classifier architectures for LLM jailbreak and prompt injection detection claims low-latency accuracy comparable to large models, led by the multi-task Raudra design.
-
Provably Secure Retrieval-Augmented Generation
SAG encrypts RAG knowledge bases and claims formal security, but its proofs are flawed and its benchmarks guarantee zero attack success by design.
Discussion (0). Sign in to comment.