{"id":"49775d6b-4492-41dc-904d-35a17ed3b64a","arxiv_id":"2605.05632","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"RAG architecture choice creates up to 58 percentage point gaps in robustness to poisoning attacks, with recursive language models faring best at 24.4% success versus 81.9% for vanilla RAG.","lead":"This paper tests four RAG architectures against knowledge base poisoning attacks using a new adversarial method on 921 QA questions. It finds large differences in attack success rates despite similar clean performance.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.3","headline":"LLM judge precision of ~48.5% is the load-bearing measurement risk for the reported attack success spread","rationale":"The reader's weakest assumption correctly isolates the evaluator and reimplementation issues as the least secure link in the measurement pipeline that supports the central quantitative claim. No deeper internal inconsistency (e.g., in the attack construction or retrieval decomposition) appears once the judge reliability is granted; the large sample size (921) and public code mitigate other risks.","tokens_in":1861,"tokens_out":411,"duration_ms":47614,"concrete_test":"Randomly sample 100 model outputs per architecture (400 total) from the CorruptRAG-AK condition; have two independent human annotators apply the paper's seven-category taxonomy and binary attack-success label; compute Cohen's kappa and per-architecture precision/recall against the LLM judge; if kappa < 0.6 or if relative ordering of attack success rates changes, the quantitative claim is unreliable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline claim quantifies architecture as a high-impact variable via attack success rates (81.9% vanilla to 24.4% RLM) under CorruptRAG-AK, with the gap localized to the content-reasoning stage after retrieval. These rates, plus the seven-category taxonomy and MADAM-RAG contradiction detection figures, are produced by the same LLM judge. The authors explicitly state ~48.5% precision (over-identification) and that MADAM-RAG rates are upper bounds; they also flag possible implementation divergences in their MADAM-RAG reimplementation that produce a 41.4% non-answer rate even on clean inputs. If judge errors are not uniform across architectures (e.g., more false positives on hedging or contradiction-handling systems), the 58-point spread and the decomposition into retrieval vs. reasoning could be artifacts of the evaluator rather than true architectural differences. Clean accuracy comparability (~92%) is also judge-dependent for some categories.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper evaluates four RAG architectures (vanilla RAG, agentic RAG, MADAM-RAG, and Recursive Language Models) on 921 Natural Questions QA pairs under clean, naive injection, and CorruptRAG-AK poisoning conditions. It claims that architecture is a high-impact variable for robustness to knowledge-base poisoning, reporting attack success rates ranging from 81.9% (vanilla) to 24.4% (RLM) under CorruptRAG-AK while clean accuracy remains comparable (~92%). The performance gap is decomposed to show that adversarial framing (rather than retrieval) drives most of the advantage after the poisoned document is retrieved, localizing vulnerability at the content-reasoning stage. A seven-category behavioral taxonomy is introduced, and the authors note that their LLM judge has ~48.5% precision (over-identifying contradictions) and that their MADAM-RAG reimplementation yields a 41.4% non-answer rate even on clean inputs.","tokens_in":2053,"tokens_out":853,"duration_ms":25313,"significance":"If the empirical findings hold after addressing evaluator concerns, the work provides concrete evidence that architectural design choices can produce large differences in adversarial robustness for RAG systems (nearly 58 percentage points in attack success) without sacrificing clean performance. The public release of code, data, and analysis notebooks is a clear strength that supports reproducibility and follow-on work. The seven-category taxonomy offers a useful lens beyond binary accuracy for studying contradiction handling and hedging behaviors.","major_comments":[{"comment":"Abstract and evaluation methodology: The headline 58-percentage-point spread in attack success rates (81.9% vanilla to 24.4% RLM) and the decomposition attributing most of CorruptRAG-AK's advantage to adversarial framing at the content-reasoning stage are both produced by the same LLM judge whose precision is reported as only ~48.5%. If judge errors are not uniform across architectures (e.g., higher false-positive rates on hedging or multi-agent systems), both the magnitude of the architecture effect and the localization to reasoning could be artifacts of the evaluator rather than true differences.","section":"Abstract"},{"comment":"Abstract: The MADAM-RAG reimplementation is reported to achieve the highest apparent contradiction-detection rate, yet it produces a 41.4% non-answer rate on clean inputs; the authors flag possible implementation divergences from the original paper. Without a direct side-by-side validation (e.g., matching non-answer rates or detection behavior on the original authors' test set), it is difficult to interpret whether the reported detection advantage is real or an artifact of the reimplementation.","section":"Abstract"},{"comment":"Evaluation section (decomposition analysis): The claim that 'once the poisoned document is retrieved, adversarial framing—not retrieval optimization—drives the majority of CorruptRAG-AK's advantage for three of four architectures' requires explicit experimental controls and metrics showing how retrieval success is isolated from downstream reasoning. It is unclear whether the decomposition uses oracle retrieval, controlled injection rates, or post-hoc analysis of retrieved documents, which is load-bearing for the localization of vulnerability to the reasoning stage.","section":"Evaluation"}],"minor_comments":[{"comment":"The abstract states 'comparable clean accuracy (~92%)' but does not report per-architecture clean accuracies or confidence intervals. Adding a table with exact clean and attacked metrics for all four architectures would improve transparency.","section":"Abstract"},{"comment":"The seven-category behavioral taxonomy is introduced but its inter-annotator agreement or mapping to the LLM judge prompts is not detailed. Clarifying the taxonomy definitions and judge prompt templates would aid reproducibility.","section":null}],"recommendation":"major_revision","confidential_remarks":"The central measurement risk (LLM judge precision) is common in current RAG/adversarial papers but remains a methodological weakness; the authors are transparent about it, which is positive. The work fits the scope of a security-focused venue given the poisoning threat model."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their detailed and insightful comments on our manuscript. We address each of the major comments below and have updated the paper accordingly to strengthen the presentation of our results and limitations.","responses":[{"response":"We acknowledge this valid concern regarding the reliability of the LLM judge. The manuscript already reports the judge's precision at ~48.5% and qualifies all contradiction detection rates as upper bounds. We did not conduct a dedicated analysis of error uniformity in the original submission. In the revised version, we will expand the limitations section to discuss the possibility of non-uniform errors across architectures and their implications for the reported gaps. Additionally, we will release the complete set of model outputs and judgments to allow for external auditing.","revision_made":"yes","referee_comment":"[Abstract] Abstract and evaluation methodology: The headline 58-percentage-point spread in attack success rates (81.9% vanilla to 24.4% RLM) and the decomposition attributing most of CorruptRAG-AK's advantage to adversarial framing at the content-reasoning stage are both produced by the same LLM judge whose precision is reported as only ~48.5%. If judge errors are not uniform across architectures (e.g., higher false-positive rates on hedging or multi-agent systems), both the magnitude of the architecture effect and the localization to reasoning could be artifacts of the evaluator rather than true differences."},{"response":"We agree that the reimplementation of MADAM-RAG introduces interpretative challenges, as noted in the manuscript where we flag possible divergences from the original work. Since the original authors' code and specific test set are not publicly available, a direct side-by-side validation on their exact setup is not feasible. Our reimplementation adheres to the published description, and the 41.4% non-answer rate on clean inputs is presented as an observed behavior. In the revision, we will include more detailed pseudocode and hyperparameter settings for the reimplementation to facilitate reproducibility and community scrutiny.","revision_made":"partial","referee_comment":"[Abstract] Abstract: The MADAM-RAG reimplementation is reported to achieve the highest apparent contradiction-detection rate, yet it produces a 41.4% non-answer rate on clean inputs; the authors flag possible implementation divergences from the original paper. Without a direct side-by-side validation (e.g., matching non-answer rates or detection behavior on the original authors' test set), it is difficult to interpret whether the reported detection advantage is real or an artifact of the reimplementation."},{"response":"The decomposition relies on post-hoc analysis of retrieval logs from our experiments. For each query, we record whether the poisoned document was retrieved within the top-k results and compute attack success rates conditional on successful retrieval of the poison. This approach uses controlled single-document injection into the knowledge base rather than oracle retrieval. We have revised the evaluation section to explicitly describe this methodology, including the addition of retrieval success metrics per architecture and conditional performance tables to clarify how the reasoning-stage vulnerability is isolated.","revision_made":"yes","referee_comment":"[Evaluation] Evaluation section (decomposition analysis): The claim that 'once the poisoned document is retrieved, adversarial framing—not retrieval optimization—drives the majority of CorruptRAG-AK's advantage for three of four architectures' requires explicit experimental controls and metrics showing how retrieval success is isolated from downstream reasoning. It is unclear whether the decomposition uses oracle retrieval, controlled injection rates, or post-hoc analysis of retrieved documents, which is load-bearing for the localization of vulnerability to the reasoning stage."}],"tokens_in":1787,"tokens_out":759,"duration_ms":63470,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"Architecture choices matter a lot for how RAG systems hold up against poisoned knowledge bases. Under the new CorruptRAG-AK attack the paper reports attack success rates dropping from 81.9% in vanilla RAG to 24.4% in recursive language models, while clean accuracy stays comparable around 92% across the four setups tested on Natural Questions data.","headline":"RAG architecture cuts poisoning attack success from 82% to 24% but the LLM judge's 48% precision makes the exact spread and decomposition hard to trust.","tokens_in":2525,"tokens_out":154,"would_cite":true,"duration_ms":26330,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"RAG architecture choice produces attack success rates from 24.4 percent to 81.9 percent under the same knowledge base poisoning attack.","keywords":["RAG","knowledge base poisoning","adversarial robustness","contradiction detection","recursive language models","agentic retrieval","retrieval-augmented generation","security evaluation"],"falsifier":"Re-scoring the 921 Natural Questions responses with human evaluators instead of the LLM judge and checking whether the 58-percentage-point spread in attack success rates across the four architectures remains intact.","tokens_in":2746,"feed_emoji":"🛡️","tokens_out":819,"duration_ms":40651,"temperature":0.7,"pith_summary":"The paper compares four retrieval-augmented generation designs on their ability to withstand an adversarial attack that inserts a single contradictory document into the knowledge base. All four designs answer clean questions at roughly the same rate of 92 percent, yet their rates of adopting the poisoned information range from 24.4 percent for recursive language models to 81.9 percent for standard retrieve-then-generate pipelines. The gap arises mainly after the poisoned document has already been retrieved, because adversarial wording affects how each architecture weighs conflicting evidence. Readers who deploy RAG systems for factual tasks would care because the results indicate that architecture selection can cut the impact of poisoning attacks by more than half without any loss in normal performance.","feed_headline":"RAG architecture cuts poisoning attack success from 82% to 24%","feed_subtitle":"Tests on 921 questions show recursive language models resist adversarial framing far better than vanilla retrieve-then-generate pipelines.","key_machinery":"Controlled comparison of vanilla RAG, agentic RAG, MADAM-RAG, and recursive language models under single-document poisoning by the CorruptRAG-AK attack, which uses meta-epistemic framing to target credibility assessment in the retrieved content.","core_discovery":"Architecture is a high-impact variable in adversarial robustness: under CorruptRAG-AK, attack success rates range from 81.9% (vanilla) to 24.4% (RLM) - a spread of nearly 58 percentage points across architectures with comparable clean accuracy (~92%). Decomposing this gap, once the poisoned document is retrieved, adversarial framing - not retrieval optimization - drives the majority of CorruptRAG-AK's advantage for three of four architectures, localizing the cross-architecture vulnerability at the content-reasoning stage. The study also introduces a seven-category behavioral taxonomy capturing contradiction detection, hedging, and failure modes beyond binary accuracy.","pith_inferences":["High-stakes RAG deployments should consider recursive language models when poisoning is a plausible threat.","Future work could test whether the same architecture ranking holds when attackers inject multiple conflicting documents rather than one.","Better mechanisms for resolving detected contradictions, not just spotting them, would likely narrow the robustness differences across architectures.","Any production use of reimplemented agent-based systems requires direct validation against original reported behaviors to avoid divergence in non-answer rates."],"forward_implications":["Recursive language models reduce the rate at which poisoned information is adopted by more than 57 percentage points compared with vanilla RAG.","For three of the four architectures, the attack's advantage comes from how the poisoned document is framed rather than from any change in retrieval ranking.","MADAM-RAG produces the highest rates of apparent contradiction detection but still yields a 41.4 percent non-answer rate on clean inputs.","A seven-category taxonomy of behaviors reveals failure modes such as hedging and unresolved contradictions that binary accuracy metrics miss."],"fun_headline_variants":["RAG architectures vary 58 points in poisoning attack success","Vanilla RAG has 82 percent poisoning success versus 24 percent for RLM","58 point RAG architecture gap in resistance to knowledge base poisoning","RAG attack success ranges from 82 percent vanilla to 24 percent recursive"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The LLM-based judge correctly identifies when a system detects and responds to contradictions, and the reimplemented MADAM-RAG behaves the same as the version described in its original paper.","fun_headline_variants_meta":{"raw":{"variants":["RAG architectures vary 58 points in poisoning attack success","Vanilla RAG has 82 percent poisoning success versus 24 percent for RLM","58 point RAG architecture gap in resistance to knowledge base poisoning","RAG attack success ranges from 82 percent vanilla to 24 percent recursive"]},"model":"grok-4.3","cost_usd":0.010098,"raw_usage":{"total_tokens":4487,"prompt_tokens":842,"num_sources_used":0,"completion_tokens":76,"cost_in_usd_ticks":100978000,"prompt_tokens_details":{"text_tokens":842,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3569,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":842,"tokens_out":76,"duration_ms":22442,"temperature":1.0,"reasoning_tokens":3569,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-08T09:38:35.644581+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Re-scoring the 921 Natural Questions responses with human evaluators instead of the LLM judge and checking whether the 58-percentage-point spread in attack success rates across the four architectures remains intact.","supporting_citations":[],"review_version":1}