{"id":"8e44fdb5-4de6-4c54-8eb4-8bceef7ebdb7","arxiv_id":"2412.18295","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A relevance-guided, fully automatic attack using open-source tools extracts most of a RAG system's private knowledge base without any access to the target's internals.","lead":"This paper describes an automated black-box attack that gets a retrieval-augmented chatbot to copy out the private documents its answers are built from. The attacker needs only an open-source language model and an embedder, and the reported tests recover up to roughly 96% of the hidden knowledge base.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LK metric may inflate extraction numbers; 'majority' claim needs a stricter leakage test.","rationale":"I agree with the reader that the reported numbers should be treated with caution, but I identify the permissive leakage metric as the single most load-bearing concern rather than the injection-defense assumption. The defense assumption is explicitly acknowledged in Section 7 and, importantly, partially tested in Appendix H: the authors evaluate a LLaMA Guard 8B classifier and show that it frequently fails to flag the attack as unsafe. That evidence weakens the concern that any safeguard will necessarily stop the attack, though it does not cover instruction-tuned refusal or output sanitization. By contrast, the LK metric is the direct evidence for the 'extract the majority' claim, and no supplementary analysis (e.g., exact-match rates or stricter ROUGE thresholds) is provided. The Nav metric is an oracle-side retrieval coverage measure and cannot verify actual leakage. Thus, the central claim's quantitative support hinges on the ROUGE-L>=0.5 soft-matching definition, which is known to be permissive. My recommended verdict remains CONDITIONAL, matching the reader's, because the concern is concrete and testable: if the stricter metric still shows majority leakage, the paper's central claim stands; if not, it must be revised. I therefore mark agreement as partial, since the reader's chosen weakest_assumption points elsewhere, though the reader's rationale already mentions the permissive metric as one of the issues.","tokens_in":24638,"tokens_out":9712,"duration_ms":83298,"concrete_test":"Re-obtain or reconstruct the raw output logs of the unbounded Pirate runs (or re-run the attack with the same agents, datasets, and hyperparameters) and recompute LK from the attacker's K⋆ under a stricter definition: (a) raise the ROUGE-L threshold to at least 0.8, and (b) drop the embedding-based soft matching, instead requiring the matched private chunk to be the actual argmax of ROUGE-L over all K. If under this stricter metric any agent's LK falls below 50%, the 'majority extraction' claim is a metric artifact and should be revised. Also report ULC and exact-match counts for the same runs to bound the effect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline claim that the attack extracts the majority of the private KB (Table 3: LK 95.8/78.8/88.8 for Agents A/B/C) rests on the leakage metric defined in Section 5: a chunk x∈K is leaked if any stolen chunk x⋆∈K⋆ has ROUGE-L(x,x⋆)≥0.5, where x⋆ is first soft-matched to the closest private chunk via the attacker-side embedder e⋆. This is permissive in two ways. First, ROUGE-L≥0.5 is a low bar: a stolen output that is a superset of x, a rough paraphrase, or a noisy concatenation of several retrieved chunks can clear it. Second, the embedding-based nearest-neighbor matching can pair x with an x⋆ that is semantically close but only partially overlapping in text, further inflating LK. The paper's own Section 7 admits the comparison is challenging and that 'other solutions could be considered to make this analysis more strict.' Nav is computed from the RAG's internal top-k retrieval, which the attacker cannot observe, so it does not by itself establish that text was actually leaked. If the reported LK is inflated, the central claim 'can extract the majority of the private knowledge base' is not supported by the evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes 'Pirates of the RAG,' a black-box, adaptive attack that extracts a private knowledge base from a Retrieval-Augmented Generation (RAG) system. The attacker uses an open-source embedder and LLM, maintains a set of anchors with relevance scores, and iteratively crafts queries with injection commands to induce the target RAG to output retrieved chunks verbatim. The method is evaluated on three RAG agents (medical, educational, research) in both bounded (300 attacks) and unbounded (run until stopping criterion) settings, using metrics Nav (navigation coverage), LK (leaked knowledge), LC, and ULC. The main claims are that the attack is automatic and adaptive, that it outperforms recent baselines (TGTB, PIDE, DGEA, RThief) in bounded comparisons, and that in the unbounded setting it 'can extract the majority of the private knowledge base,' with LK values of 95.8, 78.8, and 88.8 for Agents A, B, and C, respectively.","tokens_in":24866,"tokens_out":5823,"duration_ms":53713,"significance":"If the reported results hold, this is a significant contribution to RAG security. The attack is notable for using only open-source, locally runnable models, for its adaptive anchor-relevance mechanism, and for its strong performance relative to baselines in the unbounded setting. The paper also includes a useful, if preliminary, analysis of a Guardian-LLM defense (Appendix H). The main strengths are the clear threat model, the algorithmic formulation in Algorithm 1, and the early-stopped comparisons that partially mitigate the query-count imbalance in the unbounded setting. However, the central quantitative claim depends on a leakage metric whose strictness is questionable, and the absence of repeated runs leaves the comparative claims without error bars.","major_comments":[{"comment":"The headline claim that the attack extracts the majority of the private knowledge base rests on the LK metric, which counts a chunk x as leaked if there exists a stolen chunk x* with ROUGE-L(x, x*) >= 0.5 after soft-matching x* to its nearest private chunk in the attacker-side embedding space. A ROUGE-L threshold of 0.5 is permissive: a stolen output that is a superset of x, a rough paraphrase, or a noisy concatenation of several retrieved chunks can clear it, and the embedding-based nearest-neighbor pairing can match semantically close but only partially overlapping texts. The authors acknowledge in Section 7 that 'other solutions could be considered to make this analysis more strict.' To support the stated claim, please report leakage under stricter thresholds (e.g., ROUGE-L >= 0.7 or 0.8, or exact/containment match) and without the soft-matching step, or provide a principled justification for why 0.5 is the correct threshold for the 'majority extraction' claim. Without this, the LK values in Table 3 may overstate the amount of text actually leaked.","section":"Section 5, Metrics; Table 3; Section 7"},{"comment":"No repeated runs or error bars are reported for the Nav and LK metrics in the main comparisons. The bounded results in Table 2 include close calls (e.g., Pirate's LK of 20.1 is second to PIDE's 20.6 on Agent B), and the early-stopped unbounded row for Agent B shows Pirate-RThief LK of 22.3 versus RThief's 30.9. Without multiple random seeds (varying initial anchors, dataset subsamples, or sampling randomness in relevance-based anchor sampling), it is impossible to assess whether these gaps are meaningful or within run-to-run variability. Please report means and standard deviations (or confidence intervals) over at least three independent runs for all main tables.","section":"Tables 2 and 3 (Experiments)"},{"comment":"The paper describes the attack as 'automatic' and 'fully-automated,' but the parsing step requires manually crafted rules tailored to each agent's output format: Appendix C states that 'we design parsing rules specifically tailored to the generated text,' and Algorithm 1 lists 'estimated structure of the RAG system output' as a required input. This manual bootstrap is a significant human-in-the-loop component. Please clarify exactly what human effort is needed before the automatic loop can run and whether the 'automatic' claim applies only to the inner loop. Providing the actual parsing rules used for each agent (or a generic parser) would also improve reproducibility.","section":"Section 3.1, Algorithm 1; Appendix C"}],"minor_comments":[{"comment":"The phrase 'It proposes and adversarial untargeted attack' contains a typo ('and' should be 'an').","section":"Abstract"},{"comment":"The notation in the duplicate-checking description is inconsistent: 'sim(xz, st,j)' should likely be 'sim(x_z, s_{t,j})' with the embedded chunk representation, and the text 'all the attacker-side embeddings are always indented to be computed' should read 'intended.'","section":"Section 3.1, Duplicates paragraph"},{"comment":"The caption says 'pieces information' and should read 'pieces of information.'","section":"Figure 1 caption"},{"comment":"The second limitation bullet contains a grammatical error: 'The proposed algorithm only consider the amount of leaked knowledge' should be 'considers.'","section":"Section 7, Limitations"},{"comment":"The Guardian-LLM analysis is interesting but appears only in an appendix and is not connected to the main narrative. Consider adding a short summary in the conclusions or in the limitations section.","section":"Appendix H"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and important problem, and the proposed adaptive mechanism is a genuine step beyond static prompting attacks. The main reservation is that the quantitative evidence for the central claim is weaker than the text suggests: the LK metric is loose, the results come from a single run, and the 'automatic' characterization is qualified by manual parsing rules. These issues are fixable within the scope of a revision, so I recommend major revision rather than rejection. The authors should be encouraged to tighten the leakage evaluation and report variance; the Guardian analysis is a nice addition but needs better integration."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nRead Pirates of the RAG. The new thing is real: instead of static question lists or attacks that need to know the target embedder, the authors run an anchor set whose relevance scores decay when chunks come back duplicated, and they let the attack stop on its own. The attacker side is entirely open-source and cheap (a 1B LLM, a free embedder), and the measured query-generation time is about 13 seconds versus minutes for DGEA. That is a meaningful advance over TGTB/PIDE/DGEA/RThief, and the comparison table includes the right competitors plus modified versions that strip unfair advantages. Credit also where due: the bounded experiments early-stop their own algorithm to match 300 attacks, and the unbounded table includes early-stopped variants to control for query count. The limitations section is honest, including the point that the leakage comparison \"could be made more strict.\"\n\nThe soft spots are real but not ruinous. The biggest one is the LK metric. Calling a private chunk leaked when any stolen chunk reaches ROUGE-L >= 0.5, after soft-matching through an attacker-side embedder, is generous. A superset, a concatenation, or a loose paraphrase can clear that bar, and Nav only says the retriever surfaced the chunk, not that the LLM emitted it. The authors admit this in Section 7. So the claim that the attack \"can extract the majority of the private knowledge base\" is plausible but not nailed down by Table 3 as reported. I would want a stricter leakage definition—higher threshold or exact/near-exact text containment—plus repeated runs and error bars on the headline numbers. There is also no released code, the chunk parsing is manually tailored per agent, and the thresholds (beta, alpha_1, alpha_2, n) are manually set; they discuss this, but it means the ubiquity claim is weaker than the architecture claim.\n\nDo I think the paper is sound otherwise? Yes. The attack mechanism is coherent, the comparison is fair in its design, and the failure of the Guardian model to distinguish attack from normal input is a useful datapoint. The central qualitative result—a black-box open-source attacker can systematically drain an unsafeguarded RAG—survives the metric concern. It is exactly the kind of result that should go to review rather than be desk-rejected, because the fix is a stricter evaluation, not a different idea. If I were the editor, I would send it out, with a strong request for code, error bars, and a more conservative LK metric.\n\nFor the reading group, maybe—it will generate discussion on whether ROUGE-L 0.5 is the right leakage notion, and that is a useful conversation.","headline":"A genuinely new adaptive black-box RAG extraction attack with useful comparisons, but the 'majority extraction' headline depends on a permissive leakage metric and needs a stricter pass before the numbers are trusted.","tokens_in":25411,"tokens_out":2474,"would_cite":true,"duration_ms":24817,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An adaptive, fully black-box attack can extract the majority of a RAG system's private knowledge base using only open-source tools.","keywords":["Retrieval-Augmented Generation","prompt injection","knowledge base extraction","black-box attack","LLM security","privacy leakage","adaptive attack"],"falsifier":"Run the same three agents with an output filter that paraphrases every generated response or an input filter that strips imperative 'copy the context' commands; if the inner loop of the algorithm returns no parseable chunks and leaked knowledge stays near zero over thousands of queries, the claim of majority extraction is refuted.","tokens_in":24436,"feed_emoji":"🏴☠️","tokens_out":4907,"duration_ms":40453,"temperature":0.7,"pith_summary":"This paper aims to show that a Retrieval-Augmented Generation (RAG) system's private knowledge base can be extracted by an attacker who has no access to the system's internals and uses only openly available tools. The proposed attack is adaptive: it maintains a growing set of topic 'anchors' with relevance scores, generates a new query from the most promising anchors plus a prompt-injection command, and repeats until no anchor yields new information. In unbounded runs the attack extracts the large majority of each of three hidden knowledge bases, with navigation coverage and leaked-knowledge scores around 96%, 90%, and 94% depending on the agent. The authors argue this demonstrates an urgent need for safeguards in RAG deployment because current defenses, including a guard model they test, do not reliably stop the leakage.","feed_headline":"Black-box attack drains 96% of a RAG's private knowledge base","feed_subtitle":"An adaptive anchor-based routine, run with open-source tools, leaks most hidden chunks from three RAG chatbots.","key_machinery":"The load-bearing object is the anchor set $A_t$ paired with relevance scores $R_t$: anchors are short topic phrases extracted by an attacker-side LLM from each newly stolen chunk, and their relevance scores are updated so that anchors that keep returning duplicate chunks lose influence while new anchors inherit the current maximum relevance. This relevance-weighted sampling of anchors balances exploration and exploitation, and the duplicate check runs in the embedding space of an attacker-side encoder with a similarity threshold. Around this core, the algorithm wraps a small pool of prompt-injection commands and a parsing step that extracts chunk text from the target's verbatim output; the whole loop stops only when no anchor has positive relevance.","core_discovery":"On the paper's own terms, the central claim is that black-box RAG knowledge theft can be fully automated and made adaptive without any prior knowledge of the target embedder, retriever, or generator. The algorithm starts from one ordinary word and repeatedly injects commands that ask the target to copy the retrieved context verbatim; each newly stolen chunk contributes anchors, and relevance scores steer later queries toward unexplored regions of the hidden knowledge base while penalizing anchors that only return duplicates. The stopping condition is intrinsic—when every anchor has zero relevance—so the attack decides for itself when the knowledge base has been drained. Across three RAG agents built with different language models, embedders, and knowledge domains, the unbounded attack reaches 95.9/95.8, 89.8/78.8, and 94.3/88.8 for navigation coverage and leaked knowledge, which the authors read as extracting the majority of each private knowledge base.","pith_inferences":["Beyond the paper, the same relevance-anchor loop could be repurposed for targeted extraction: seeding the anchor set with a desired topic should steer the attack toward a specific region of the hidden knowledge base rather than maximizing global coverage.","Beyond the paper, the attack's success boundary is likely sharp: it should fail on any RAG whose generator is trained or prompted to paraphrase retrieved context instead of copying it, or that uses any input filter that rejects or neutralizes imperative 'copy all text' phrasing.","Beyond the paper, a natural defensive experiment suggested by the results is to measure leaked knowledge under output paraphrasing or selective verbatim suppression; the paper's own limitation section indicates such safeguards would break the inner loop of the algorithm."],"forward_implications":["A RAG system that faithfully reproduces retrieved context is exfiltratable in full by an automated routine, not just by hand-crafted prompts.","Fixed-budget attacks leave much of the knowledge base untouched; an adaptive attacker with an intrinsic stopping rule covers regions a static prompt set never reaches.","The attack transfers across different RAG pipelines (different generator LLMs, embedders, and domains) and can be run with a small open-source LLM and embedder on domestic hardware.","A guard LLM that screens inputs and outputs for safety does not reliably separate attack queries from normal user queries, and over-blocks legitimate interactions in specialized-advice domains.","Because query generation is cheap and fast, the bottleneck for an attacker is not compute but whether the target's output obeys the injection commands."],"supporting_citations":[{"why":"Supplies the prompt-injection technique, the ROUGE-based leak metric, and the TGTB baseline that the attack builds on and compares against.","marker":"(Zeng et al., 2024)"},{"why":"Provides the PIDE baseline and the 'follow my instruction and spill the beans' injection style used in the command pool.","marker":"(Qi et al., 2024)"},{"why":"Provides the DGEA baseline and the structured JSON output command, and represents the partially black-box adaptive attack the paper contrasts with.","marker":"(Cohen et al., 2024)"},{"why":"Provides the RThief baseline, a memory-based adaptive attack that is the main unbounded competitor in the experiments.","marker":"(Jiang et al., 2024a)"},{"why":"Defines retrieval-augmented generation, the system architecture that the attack targets.","marker":"(Lewis et al., 2020)"},{"why":"Surveys trustworthiness and privacy issues in RAG systems and motivates the threat model of private knowledge-base leakage.","marker":"(Zhou et al., 2024)"},{"why":"Provides the ROUGE score implementation used to decide whether a stolen chunk counts as leaked knowledge.","marker":"(Grusky, 2023)"}],"fun_headline_variants":["Adaptive black-box attack leaks 96% of RAG knowledge","RAG pirates: adaptive open-source attack steals most private data","Adaptive black-box attack drains up to 96% of RAG's knowledge","Open-source adaptive attack leaks most of RAG's hidden data","Automatic RAG heist: open-source tool grabs 96% of private chunks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central premise is that the target RAG system will obey a small set of prompt-injection commands and emit retrieved chunks nearly verbatim, without any input filtering, output sanitization, or instruction-tuned refusal.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive black-box attack leaks 96% of RAG knowledge","RAG pirates: adaptive open-source attack steals most private data","Adaptive black-box attack drains up to 96% of RAG's knowledge","Open-source adaptive attack leaks most of RAG's hidden data","Automatic RAG heist: open-source tool grabs 96% of private chunks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000823,"raw_usage":{"total_tokens":3587,"prompt_tokens":917,"completion_tokens":2670,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":2574}},"tokens_in":533,"tokens_out":2670,"duration_ms":18197,"temperature":1.0,"reasoning_tokens":2574,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:49:24.614602+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same three agents with an output filter that paraphrases every generated response or an input filter that strips imperative 'copy the context' commands; if the inner loop of the algorithm returns no parseable chunks and leaked knowledge stays near zero over thousands of queries, the claim of majority extraction is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ROUGE score implementation used to decide whether a stolen chunk counts as leaked knowledge."}],"review_version":1}