{"id":"78ba78b2-8b15-4867-a188-6861eb206727","arxiv_id":"2608.04756","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A black-box poisoning attack that frames false information as a fact-compatible update defeats conflict-resolution safeguards in RAG on most tested settings.","lead":"This paper introduces a new way to poison the knowledge sources that retrieval-augmented AI assistants rely on, by crafting fake updates that do not contradict facts the system already trusts. The attack beats earlier poisoning methods on most tested configurations, showing that conflict-checking safeguards in RAG can be sidestepped by consistent-looking injections.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central mechanism hinges on Fq faithfully standing in for the resolver's reference, but Eq. 3 is only checked against Fq itself; no fidelity measurement ties proxy facts to the resolver's actual parametric and retrieved evidence.","rationale":"I read PURPOSE as an empirical attack paper whose central claim has two parts: (i) a black-box attacker can construct non-contradicting updates that survive conflict resolution, and (ii) this yields superior ASR. The paper deserves credit for strong supporting evidence elsewhere: the component ablation (V1 to V2 in Table 5) shows that Fq grounding matters, the retrieval-conditioned ASR (Table 9) shows the advantage is not just retrieval, and the cross-attacker generalization (Section 5.3) shows the mechanism is not tied to one probing model. However, the mechanism is defined relative to Fq, and Fq is never validated against any actual resolver reference. The paper explicitly concedes that exact correspondence is not guaranteed. The conflict-score evaluation in Section 5.2.2 measures consistency with Fq, not with the resolver's parametric knowledge or co-retrieved evidence. Thus the distinctive 'non-contradicting' property is asserted rather than demonstrated. This is not an internal inconsistency; it is an unmeasured external validity condition. If it fails, the method may reduce to a well-crafted assertive attack, and the paper's headline implication—that contradiction checking alone is insufficient—would be supported by the wrong mechanism. My proposed test would settle the question by correlating measured fidelity with attack success. Other issues flagged in the reader's rationale, such as the LLM judge sharing a model family with the probing LLM and target answers differing across attacks, are real but secondary: they affect the magnitude of the empirical advantage, whereas the fidelity issue affects whether the paper's central theoretical construction is actually doing what it claims. The verdict therefore remains conditional in the same way the reader concluded.","tokens_in":22336,"tokens_out":7918,"duration_ms":99452,"concrete_test":"Take a random 100-query subset of NQ. For each query: (1) elicit Fq using DeepSeek-V3.2 as in the paper; (2) extract the resolver's actual reference: AstuteRAG's parametric dump before consolidation, FaithfulRAG's self-fact set, and the clean top-5 retrieved passages; (3) have an independent judge (not DeepSeek, e.g., GPT-5.2 or human annotators) score each fact in Fq as entailed or consistent with the resolver's reference; (4) form a per-query fidelity score. Then split the queries at the median fidelity and compare PURPOSE's ASR and ASRstrict on high- versus low-fidelity queries. If low-fidelity ASR is not lower, the proxy assumption is not doing the work; if it drops substantially, the Eq. 3 mechanism is only as sound as an unmeasured fidelity assumption.","verdict_should_be":"UNCHANGED","load_bearing_attack":"PURPOSE's defining move is the proxy-fact-grounded pivot: Eq. 3 requires Fq ∪ {e} to be consistent, Fq ∪ {e} to entail y~, and e to entail ¬y*, where Fq is meant to approximate what the resolver treats as settled (Section 3.2). The paper's only justification for this approximation is that 'modern LLMs share substantial well-attested factual knowledge' (Section 3.2), plus the observation that cross-attacker ASR varies with the probing model (Section 5.3). Neither establishes that Fq matches the reference actually used by AstuteRAG, FaithfulRAG, or MADAM-RAG: the parametric knowledge consulted at inference and the clean co-retrieved passages. The conflict scores in Section 5.2.2 are computed between the poisoned document and Fq by a DeepSeek judge, so they validate internal construction, not compatibility with the resolver. If Fq diverges materially, e may contradict evidence the resolver trusts; then the claimed non-contradicting update mode is not achieved, and the +9.7 mean ASR advantage could stem from authority scaffolding, query alignment, or persuasive wording rather than from the proxy-grounded mechanism. The component ablation (Table 5) shows that grounding in Fq helps, but it cannot distinguish 'grounded in the correct reference' from 'grounded in a plausible proxy that happens to be persuasive.' This is the load-bearing premise: without fidelity, Eq. 3 is a condition on an unobserved set, and the central theoretical claim that contradiction checking is insufficient is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes PURPOSE, a black-box poisoning attack against RAG systems that include post-retrieval conflict resolution. The attack elicits a set of proxy facts Fq from a public LLM (Section 3.2), then constructs a pivot event e satisfying Eq. (3) so that e is consistent with Fq, Fq ∪ {e} entails the target answer, and e entails the negation of the ground truth. The poisoned document is additionally given authority-style references and query-aligned phrasing. The evaluation covers three QA benchmarks, five target generators, three conflict-resolution methods and vanilla RAG, and three prior baselines, with retrieval-conditioned, component-ablation, retriever-sensitivity, cross-attacker, and question-type analyses. The headline result is the highest ASR in 35 of 45 conflict-resolution settings and a +9.7 mean ASR gain over the strongest baseline, leading to the claim that contradiction checking alone is insufficient for secure RAG.","tokens_in":22632,"tokens_out":7723,"duration_ms":88137,"significance":"If the central mechanism holds, the paper identifies a practically important and understudied attack mode: non-contradicting, update-style poisoning that can bypass explicit conflict-resolution safeguards. Strengths include the unusually broad evaluation grid, the conditional retrieval analysis, the transparent discussion of why substring matching fails under conflict resolution, and an informative case study. The main qualifications are that the proxy-fact fidelity premise is not directly measured, the headline metric counts hedged answers that still contain the ground truth as attack success, and the same model family is used as prober, judge, and one target generator. The claims are therefore plausible but not yet demonstrated at the level the central conclusion requires.","major_comments":[{"comment":"The pivot conditions in Eq. (3) are checked only against the proxy fact set Fq, but the design claim is that the poisoned document is non-contradicting with respect to what the resolver actually treats as settled, namely its parametric knowledge plus co-retrieved clean passages. Section 3.2 explicitly concedes that exact correspondence is not guaranteed, yet no measurement of proxy fidelity is reported. The conflict scores in §5.2.2 are computed by a DeepSeek judge between the poisoned document and Fq, so they validate internal construction, not compatibility with the resolver. Consequently, the +9.7 mean ASR advantage over prior attacks could, in principle, be driven by authority scaffolding, query alignment, or persuasive wording rather than by the proxy-grounded mechanism. Please add a direct fidelity test, for example on a subset where the resolver's reference can be inspected (e.g., AstuteRAG's parametric dump), measure agreement between Fq and that reference; also compare ASR when Fq is replaced by random facts, by deliberately wrong facts, and by the resolver's actual extracted facts. This is the load-bearing premise of the paper.","section":"§3.2, Eq. (3)"},{"comment":"The headline metric ASR = Pr[INCORRECT ∪ BOTH] counts hedged answers that preserve the ground truth as attack successes, while the same BOTH outputs also count toward ACC. The paper's own analysis notes that on MADAM-RAG many gains take the form of hedging and that several cells satisfy ACC + ASR > 100. Because the abstract and Section 5.1 emphasize '35 of 45' and the +9.7 mean ASR margin, which are based on this inclusive metric, the reported attack success overstates the extent to which the final answer is steered away from y⋆. Please report the corresponding per-cell win counts and aggregate margins for ASRstrict, and separately report the BOTH rate per condition; a clean metric that excludes BOTH from both ACC and ASR would sharpen the claim.","section":"§4, Evaluation Metrics"},{"comment":"The default probing LLM, the conflict/support judge, and one of the target generators are all DeepSeek-V3.2. This same-family overlap creates a concrete optimism channel: the fact set used to construct the attack is elicited from the same model family that later labels outputs as INCORRECT or BOTH, and in the DeepSeek-generator condition the judging model is the attacked model itself. The cross-attacker experiments show model sensitivity but do not remove the overlap in the default configuration. Please validate on a random subset with an independent judge (e.g., a different model family and a human sample) and report ASR and ASRstrict under that judge; this directly tests whether the headline numbers are affected by self-grounded labeling.","section":"§4 and §5.2.2"},{"comment":"The paper's motivating claim that 'existing black-box poisoning methods all assert the target answer in frontal contradiction with what the resolver treats as settled' is contradicted by its own related work and appendix: CorruptRAG-AK (Zhang et al., 2026a) rewrites an outdated-answer claim into fluent adversarial knowledge and is described in B.2 as a closely related update-style attack. Since CorruptRAG-AK is excluded from Table 1, the '35 of 45' and '+9.7' comparisons and the novelty claim are overstated as stated. Please include CorruptRAG-AK in the main comparison (with the Gemini exclusion clearly disclosed) or revise the claim about prior attack modes to 'direct-assertion attacks' rather than 'all black-box poisoning methods.'","section":"Abstract, §1, Related Work, B.2"}],"minor_comments":[{"comment":"The +9.7 mean ASR margin is not directly derivable from the stated per-method margins (+14.7, +6.5, +7.9, +4.9); please specify how the aggregate is computed.","section":"Abstract / §5.1"},{"comment":"The panel header states that attack columns report ACC↓/ASRstrict↑/ASR↑ while Clean reports ACC only, but the Clean column is also used for perplexity in Panel B; please clarify the notation.","section":"Table 1"},{"comment":"The number of proxy facts l is a free parameter of the pipeline, but no distribution or sensitivity analysis is reported; please provide its range and a check of whether Eq. (3) satisfaction varies with l.","section":"§3.3 / Algorithm 1"},{"comment":"Withholding complete prompts and the poisoned-document collection limits reproducibility; consider releasing a sanitized or redacted artifact set, or detailed pseudocode of the prompts, to allow verification without enabling direct misuse.","section":"Ethics Statement"},{"comment":"The CorruptRAG-AK reproduction relies on a third-party implementation because the official one is unavailable; please report the overlap or manual verification between your reproduction and the published description.","section":"Appendix B.2"}],"recommendation":"major_revision","confidential_remarks":"The main reasons for a major revision rather than an accept are the unmeasured proxy-fidelity premise, the inclusive ASR metric, and the same-family judge/prober overlap. None of these is necessarily fatal, but together they mean the central quantitative claims are not yet established. The relation to CorruptRAG-AK should also be clarified, since the current framing reads as broader than the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere is my read of arXiv:2608.04756. The genuinely new thing is the proxy-fact-grounded pivot: instead of asserting the target answer in frontal contradiction with what the resolver treats as settled, the attacker elicits a set of facts Fq from a public LLM and constructs an event e that is consistent with Fq, entails the target answer, and individually entails not the gold answer. That is a real new mechanism, and the evaluation is broad enough to show it works better than the direct-assertion baselines across most settings. The retrieval-conditioned numbers and the component ablation (Table 5) are the strongest evidence: the advantage persists after successful retrieval, and the gain comes from grounding the pivot in Fq rather than from authority cues or query alignment alone.\n\nThe main soft spot is exactly what the stress-test flags. Eq. (3) is a condition on Fq, but Fq is only a proxy for what the resolver actually consults. The paper never measures proxy fidelity against the resolver's parametric knowledge or clean co-retrieved passages. Without that, the conflict scores in Section 5.2.2 validate internal construction, not compatibility with the target. So the theoretical claim that contradiction checking is insufficient is plausible but not fully established. I don't think this is fatal. The empirical comparison with CorruptRAG-AK and the cross-attacker results show a consistent advantage, and the case study traces concrete failure paths in each resolver. If Fq were wildly wrong, you would expect more failures than the ablations show. Still, the headline ASR counting hedged BOTH outputs as success is a real concern; ASRstrict is reported and generally leads too, but the +9.7 mean ASR headline leans on the inclusive metric.\n\nOther soft spots: the judge is DeepSeek-V3.2, the same family as the default prober and one of the target generators, so there is possible circularity; the cells are point estimates on 100 queries with no error bars; and prompts, code, and data are withheld. The withholding is defensible as dual-use policy, but it makes independent reproduction hard.\n\nBottom line: this is a solid empirical attack paper with a genuinely new injection mode. The central mechanism deserves scrutiny, and the proxy-fidelity question should be addressed with a direct measurement or at least a sensitivity analysis. I would send it to peer review. A serious reviewer will ask for proxy-fidelity evidence and confidence intervals, but the work is coherent, honest about limitations, and worth engaging.","headline":"A genuinely new non-contradicting RAG poisoning mechanism, well evaluated, but the load-bearing proxy-fidelity assumption is never directly tested and the headline metric counts hedged answers as success.","tokens_in":23202,"tokens_out":1895,"would_cite":true,"duration_ms":23217,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A black-box poisoning attack defeats conflict resolution in RAG by framing injected documents as consistent updates rather than contradictions.","keywords":["knowledge poisoning","retrieval-augmented generation","conflict resolution","black-box attack","proxy facts","pivot event","non-contradicting injection","LLM-as-judge evaluation"],"falsifier":"Measure the attack success rate of PURPOSE while intentionally corrupting the proxy facts—for example, by using a probing model with outdated knowledge for a set of queries, or by directly editing $F_q$ before composition—and compare against the default setup; if ASR stays high under corrupted $F_q$, then minimizing conflict with the proxy is not the active mechanism, and if it drops sharply, proxy fidelity is the lever the paper claims.","tokens_in":22117,"feed_emoji":"🎯","tokens_out":5353,"duration_ms":54849,"temperature":0.7,"pith_summary":"This paper argues that post-retrieval conflict resolution in retrieval-augmented generation (RAG) is vulnerable to a new class of knowledge-poisoning attack. Instead of asserting the attacker's target answer in open contradiction to what the resolver treats as settled, the proposed PURPOSE attack first elicits \"proxy facts\" from a public LLM as an estimate of the resolver's reference, then fabricates a recent event that is consistent with those facts but whose logical consequence is the target answer. Across three QA benchmarks, five generators, and three conflict-resolution methods, the attack reports the highest success rate in 35 of 45 settings, exceeding the strongest prior attack by +9.7 mean ASR points. If the central claim holds, contradiction detection alone is not a sufficient safeguard, and defenses must move toward provenance and temporal verification of claimed updates.","feed_headline":"RAG conflict filters fall to update-style poisoning in 35 of 45 tests","feed_subtitle":"Fabricated 'recent events' grounded in facts the model already accepts beat prior attacks by +9.7 ASR points.","key_machinery":"The load-bearing mechanism is the proxy-fact-grounded pivot update. Proxy facts $F_q$ are elicited by prompting a public LLM with the query, serving as the attacker's estimate of what the resolver will treat as settled, covering both parametric knowledge and co-retrieved passages. The pivot event $e$ is a fabricated recent development (e.g., a leadership transition or a database revision) that preserves every proxy fact while redirecting the answer; the three logical conditions in Eq. (3) define what the construction must satisfy. A five-stage prompt-based pipeline (Elicit, Perturb, IdentAuth, Compose, Align) realizes the construction with single black-box calls, and the final alignment stage repeats the query and places the target answer in the opening and closing sentences to exploit similarity-based retrieval and primacy/recency biases.","core_discovery":"The central claim is that a strict black-box attacker can defeat conflict resolution by reframing the injection as a minimal update rather than a counter-claim. For a query $q$ with correct answer $y^*$ and target answer $\\tilde{y}$, the attacker constructs a poisoned document through five prompt-only stages that elicit a set $F_q$ of query-related proxy facts from a publicly accessible LLM, generate a plausible $\\tilde{y}$, shortlist authoritative-sounding sources, compose a pivot event $e$ and narrative, and align the document to the query. The pivot event is engineered to satisfy $F_q \\cup \\{e\\} \\not\\models \\bot$, $F_q \\cup \\{e\\} \\models \\tilde{y}$, and $e \\models \\neg y^*$: it introduces no contradiction with what the resolver may treat as settled, it yields the target answer together with those facts, and it alone already overturns the correct answer. The paper reports that this design attains the highest attack success rate in 35 of 45 conflict-resolution settings and a +9.7 mean ASR gain over the strongest prior attack, with retrieval statistics comparable to baselines, showing the gain is post-retrieval rather than a retrieval artifact.","pith_inferences":["If proxy facts are a good model of the resolver's reference, then defenses that only check consistency will stay vulnerable; a natural next test is whether resolvers that verify event provenance or timestamps break the attack.","The paper's decision to count hedged outputs (BOTH) as attack success implies that even without flipping the final answer, an attacker can succeed by forcing resolvers into indecision; defenders may need to treat confidence erosion as a first-class harm.","A testable extension: force the probing LLM to elicit deliberately outdated or wrong facts for a subset of queries and measure how often the pivot still succeeds; this would isolate how much of the attack hinges on proxy fidelity."],"forward_implications":["Contradiction-based conflict resolution is not a sufficient defense against knowledge poisoning in RAG; update-style injections that avoid frontal contradiction can still steer outputs.","The attack's advantage persists after successful retrieval, implying it operates at the arbitration stage rather than by merely improving retrievability.","The non-contradicting injection mode works across open- and closed-source generators and across retrievers with different similarity functions, so the vulnerability is general rather than tied to one pipeline.","Because the gain narrows on vanilla RAG, where any retrievable assertion already works, the real impact is on exactly the systems that added safety checks."],"supporting_citations":[{"why":"Defines the PoisonedRAG baseline and supplies the 100-QA-pair evaluation subsets and the foundational direct-assertion attack that PURPOSE must beat.","marker":"(Zou et al., 2025)"},{"why":"Establishes the one-injection setting and the authority-based AuthChain baseline, against which PURPOSE compares and on which it builds the alignment and credibility scaffolding.","marker":"(Chang et al., 2025)"},{"why":"Defines the PARADOX attack, the strongest prior black-box baseline, and the paper shows PURPOSE exceeds it while operating under a strictly more restrictive threat model.","marker":"(Choi et al., 2025a)"},{"why":"Provides the AstuteRAG conflict-resolution method, one of the three target defenses, and supports the design principle that conflicts with parametric knowledge trigger closer scrutiny.","marker":"(Wang et al., 2025a)"},{"why":"Provides the FaithfulRAG fact-level conflict-resolution method, a target defense whose semantic (not logical) conflict check is what the pivot event is designed to pass.","marker":"(Zhang et al., 2025a)"},{"why":"Provides the MADAM-RAG multi-agent debate defense, the third target resolver, and the paper shows its aggregation turns minority dissent into hedged outputs.","marker":"(Wang et al., 2025b)"},{"why":"Supplies the premise that modern LLMs share substantial well-attested factual knowledge, which justifies using a probing LLM's elicited facts as a proxy for the target resolver's reference.","marker":"(Mallen et al., 2023)"}],"fun_headline_variants":["Update-style poisoning beats RAG conflict resolution in 35 of 45","Poisoning RAG conflict resolvers: 35 of 45 attacks succeed","RAG conflict filters duped by fact-grounded injection attacks","Proxy-fact updates outdo prior RAG poisoning by 9.7 ASR","Non-contradicting attacks beat existing RAG poisoning by 9.7"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The attack stands on the assumption that facts elicited from a public LLM accurately approximate what the target conflict resolver treats as settled, including both parametric knowledge and co-retrieved passages; if that proxy is wrong, the fabricated event may contradict trusted evidence and the attack loses its documented advantage.","fun_headline_variants_meta":{"raw":{"variants":["Update-style poisoning beats RAG conflict resolution in 35 of 45","Poisoning RAG conflict resolvers: 35 of 45 attacks succeed","RAG conflict filters duped by fact-grounded injection attacks","Proxy-fact updates outdo prior RAG poisoning by 9.7 ASR","Non-contradicting attacks beat existing RAG poisoning by 9.7"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000831,"raw_usage":{"total_tokens":3658,"prompt_tokens":1002,"completion_tokens":2656,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":618,"completion_tokens_details":{"reasoning_tokens":2555}},"tokens_in":618,"tokens_out":2656,"duration_ms":19980,"temperature":1.0,"reasoning_tokens":2555,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:18:53.701657+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the attack success rate of PURPOSE while intentionally corrupting the proxy facts—for example, by using a probing model with outdated knowledge for a set of queries, or by directly editing $F_q$ before composition—and compare against the default setup; if ASR stays high under corrupted $F_q$, then minimizing conflict with the proxy is not the active mechanism, and if it drops sharply, proxy fidelity is the lever the paper claims.","supporting_citations":[],"review_version":1}