Pith. sign in

REVIEW 8 cited by

Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systems

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.17840 v3 pith:EQ4OBNMD submitted 2024-02-27 cs.CL cs.AIcs.CRcs.LG

Zhenting Qi , Hanlin Zhang , Eric Xing , Sham Kakade , Himabindu Lakkaraju This is my paper
classification cs.CLcs.AIcs.CRcs.LG
keywords datadatastoregptsmodelscustomizedextractgenerationleakage
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Retrieval-Augmented Generation (RAG) improves pre-trained models by incorporating external knowledge at test time to enable customized adaptation. We study the risk of datastore leakage in Retrieval-In-Context RAG Language Models (LMs). We show that an adversary can exploit LMs' instruction-following capabilities to easily extract text data verbatim from the datastore of RAG systems built with instruction-tuned LMs via prompt injection. The vulnerability exists for a wide range of modern LMs that span Llama2, Mistral/Mixtral, Vicuna, SOLAR, WizardLM, Qwen1.5, and Platypus2, and the exploitability exacerbates as the model size scales up. We also study multiple effects of RAG setup on the extractability of data, indicating that following unexpected instructions to regurgitate data can be an outcome of failure in effectively utilizing contexts for modern LMs, and further show that such vulnerability can be greatly mitigated by position bias elimination strategies. Extending our study to production RAG models GPTs, we design an attack that can cause datastore leakage with a 100% success rate on 25 randomly selected customized GPTs with at most 2 queries, and we extract text data verbatim at a rate of 41% from a book of 77,000 words and 3% from a corpus of 1,569,000 words by prompting the GPTs with only 100 queries generated by themselves.

Discussion (0). Sign in to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Isolated but Exposed: Persistence-Based Memory Extraction Attack on LLM Agents

    cs.CR 2026-07 conditional novelty 7.5 of 10

    Malicious tools can systematically extract isolated LLM-agent long-term memory via persistence, pure-anchor retrieval steering, and reactivation payloads, reaching 80% extraction with unlimited triggers and 47% with 20.

  2. Not All Entities are Created Equal: A Dynamic Anonymization Framework for Privacy-Preserving RAG

    cs.CR 2026-03 unverdicted novelty 7.0 of 10

    TRIP-RAG dynamically anonymizes only high-risk entities in RAG knowledge bases via three context-aware metrics, achieving privacy comparable to full anonymization with under 35% recall drop and up to 56% better genera...

  3. Adaptive Backtracking for Privacy Protection in Large Language Models

    cs.CR 2025-08 conditional novelty 6.0 of 10

    A training-free backtracking defense that rewrites RAG output at the first sign of privacy leakage improves privacy utility scores by up to 15% over sanitization and prompting baselines on a new healthcare and finance...

  4. Benchmarking Knowledge-Extraction Attack and Defense on Retrieval-Augmented Generation

    cs.CR 2026-02 conditional novelty 5.0 of 10

    A unified benchmark comparing RAG knowledge-extraction attacks and defenses, showing query diversity boosts extraction, embedding attacks fail to transfer, and graph indexing raises per-token leakage.

  5. Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generation

    cs.CL 2025-08 reject novelty 5.0 of 10

    Sparse autoencoder activation perturbation (SFPF) applied on top of existing jailbreak prompts raises attack success rate on Qwen3-32B, but with no defense evaluation and weak reproducibility.

  6. Diverse And Private Synthetic Datasets Generation for RAG evaluation: A multi-agent framework

    cs.CL 2025-08 conditional novelty 4.0 of 10

    A multi-agent LLM framework generates synthetic QA datasets for RAG evaluation by combining clustering-based sampling, PII pseudonymization, and QA curation, with reported diversity gains and 0.75-0.90 masking accuracy.

  7. RAG Security and Privacy: Formalizing the Threat Model and Attack Surface

    cs.CR 2025-09 conditional novelty 3.0 of 10

    A formal RAG threat model is defined with four adversary classes and game-based notions of membership inference, leakage, and poisoning, but the definitions largely restate known concepts and the main DP-based protect...

  8. Provably Secure Retrieval-Augmented Generation

    cs.CR 2025-08 reject novelty 2.0 of 10

    SAG encrypts RAG knowledge bases and claims formal security, but its proofs are flawed and its benchmarks guarantee zero attack success by design.

Pith tools