{"id":"d5647856-b1a2-47da-831a-81c150a603d3","arxiv_id":"2508.04276","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Modifying less than 0.05% of source text can collapse GraphRAG question-answering accuracy from 95% to 50% by poisoning the knowledge graph during extraction.","lead":"This paper shows that changing just a few words in the source text fed to a graph-based retrieval system can silently corrupt the knowledge graph and flip or collapse the answers the system later gives. Because this attack works while keeping the text fluent and evades current defenses, it exposes a real vulnerability in how AI systems get their facts from structured knowledge graphs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No raw-RAG baseline: without showing the same word edits damage non-graph RAG less, the claim that the graph-construction stage is the vulnerable surface is unsupported.","rationale":"The reader's verdict is UNVERDICTED because the abstract lacks protocol details and variance statistics; I agree that such details are essential. However, the reader's weakest_assumption focused on generalizability across datasets and the threat model of altering source text. I identify a more specific and more load-bearing omitted control: a non-graph RAG baseline. The central claim, as phrased in the reader's strongest_claim, is that the vulnerability lives in the upstream LLM extraction step of GraphRAG. That attribution requires showing that the attack is not simply an ordinary corpus-poisoning attack on any RAG system. The abstract says the graph changes, but changing the text necessarily changes both the graph and any text-based retrieval index. Without a raw-RAG comparison, the 95% to 50% QA collapse and 93.1% targeted success might be explained by the retriever selecting the poisoned passages, independent of graph construction. My concern is not that the authors are dishonest, but that the available evidence does not yet distinguish between 'GraphRAG-specific poisoning via graph extraction' and 'generic RAG poisoning via text changes.' The proposed concrete test would settle that distinction. I keep the verdict UNCHANGED because the full text may already contain such a comparison; the concern is an evidentiary gap in the abstract, not a demonstrated flaw in the argument.","tokens_in":1050,"tokens_out":6447,"duration_ms":70323,"concrete_test":"Re-run UKPA/TKPA against a standard non-graph RAG pipeline using the same source text, the same retriever embeddings, the same top-k setting, and the same QA LLM. Measure QA accuracy after applying the same word-level edits to both the GraphRAG index and the raw-text index. Also compare graph-structural change metrics against retrieval-score changes. If the non-graph RAG accuracy drops by the same or larger margin, the attack is not specific to GraphRAG's graph-construction stage; if it drops substantially less, the GraphRAG-specific claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract claims that 'modifying only a few words in the source text can significantly change the constructed graph, poison the GraphRAG, and severely mislead downstream reasoning.' But the attack operates on the source text before any graph construction or retrieval. If the same word-level edits were applied to a standard dense-retrieval RAG pipeline that indexes the same text, the QA accuracy might drop just as much, because the retriever would surface the poisoned passages before any graph is used. In that case, the reported 'poisoning' is generic data-poisoning of the text corpus, not evidence that GraphRAG's graph-construction stage is the operative attack surface. The abstract provides no comparison against a non-graph RAG baseline, and the effect sizes are bare point estimates without dataset names, extraction LLM, QA benchmark, or variance. The most load-bearing missing control is thus the raw-RAG comparison: it is necessary to attribute the vulnerability to the graph-construction stage rather than to the underlying text.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two knowledge poisoning attacks on GraphRAG systems, both exploiting the LLM-based graph-extraction stage. Targeted KPA (TKPA) uses graph-theoretic node selection and LLM rewriting to steer specific QA outcomes, reporting 93.1% success; Universal KPA (UKPA) modifies fewer than 0.05% of the source text using linguistic cues and is claimed to collapse QA accuracy from 95% to 50%. The abstract also states that state-of-the-art defense methods fail to detect the attacks.","tokens_in":1153,"tokens_out":2524,"duration_ms":30717,"significance":"If the reported results hold, the work identifies a practical and previously under-explored attack surface in GraphRAG pipelines: the graph-construction step is shown to be manipulable by small, fluent text edits. The two attacks are methodologically distinct—one graph-theoretic and one purely linguistic—and the UKPA edit budget is remarkably low. These are potentially significant contributions to LLM security and retrieval-augmented generation. However, the abstract alone provides no reproducibility details, no baseline comparisons, and no statistical characterization, so the significance is conditional on the full experimental evidence.","major_comments":[{"comment":"The central claim that the vulnerability resides in the graph-construction stage is not supported without a control condition: the same word-level edits applied to a non-graph dense-retrieval RAG pipeline should be reported. If QA accuracy drops comparably in that baseline, the attack is generic text poisoning, not a GraphRAG-specific surface. This control is load-bearing for the paper's main thesis.","section":"Abstract (Universal KPA)"},{"comment":"No dataset names, extraction LLM, QA benchmark, defense configurations, or number of trials are given. The reported success rates (93.1%, 95%→50%) are bare point estimates with no error bars or confidence intervals, which is particularly problematic for LLM-based evaluations that are known to be highly variable.","section":"Abstract (experimental reporting)"},{"comment":"The attacker is assumed to be able to alter source text before graph construction, with no provenance checks, filtering, or human review. This assumption is not stated explicitly in the abstract; it should be specified and justified, including whether the edits remain plausible in realistic ingestion pipelines and whether any manual inspection would flag them.","section":"Abstract (threat model)"},{"comment":"The statement that 'state-of-the-art defense methods fail to detect these attacks' is currently an unsupported assertion. A list of the defenses tested and the detection metrics used is necessary to evaluate the claim. Without this, the abstract overstates the robustness of the attacks.","section":"Abstract (defense evaluation)"}],"minor_comments":[{"comment":"The '0.05% of full text' figure is a relative edit budget; absolute numbers (e.g., words edited per document) would make the attack's practicality clearer.","section":"Abstract (UKPA)"},{"comment":"The phrase 'severely mislead downstream reasoning' is vague; specifying the QA task types (e.g., multi-hop, single-hop, open-book) would help the reader understand the scope.","section":"Abstract (general)"},{"comment":"It is unclear how the 'graph-theoretic analysis' identifies vulnerable nodes; a one-sentence summary of the node selection criterion would improve the abstract's self-containedness.","section":"Abstract (TKPA)"}],"recommendation":"uncertain","confidential_remarks":"The abstract reports high-impact results but lacks the methodological and statistical detail necessary for a soundness judgment. The editor may wish to secure the full manuscript and, in particular, verify that a raw-RAG (non-graph) baseline is included. If the full paper does not contain this baseline, the central claim would likely be a major revision. The fit with a security-focused venue might be better than a general NLP venue, given the emphasis on adversarial attacks."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know one thing up front: the claim that altering fewer than 0.05% of tokens collapses GraphRAG QA from 95% to 50% is the kind of result that gets people's attention, but this abstract alone doesn't support the stronger conclusion that the graph-construction stage is the vulnerable part. There's no comparison against a standard dense-retrieval RAG pipeline with the same word edits. If the same edits hurt that baseline just as much, then the attack is generic text poisoning, not a GraphRAG-specific vulnerability.\n\nWhat's genuinely new: the paper identifies LLM-based graph extraction as an attack surface and proposes two distinct attacks, TKPA (graph-theoretic node targeting) and UKPA (linguistic-cue based word swapping). That's a real gap in the literature as far as I know. The mechanism is plausible: if an LLM extracts triples from text, small adversarial edits can propagate into the graph structure and mislead downstream reasoning.\n\nWhat's soft, in proportion: this is an abstract-only review, so there's no dataset, extraction model, QA benchmark, defense configuration, or variance information. The numbers are bare point estimates. The state-of-the-art defenses claim is overbroad without naming the defenses. The threat model assumes the attacker can modify source text before ingestion with no provenance check or human review, which is a standard assumption but should be stated explicitly. None of these are detected flaws; they're missing information.\n\nThe stress-test concern about a raw-RAG baseline is the most load-bearing. If the full paper includes that control and the graph-based system degrades significantly more, the attack surface claim holds. If not, the paper needs that experiment before publication.\n\nI'd send this to peer review rather than desk reject, because the topic is important, the claims are falsifiable, and the attack mechanisms are novel enough to warrant referee time. But I'd tell the reviewers to demand the raw-RAG baseline, full protocol details, and statistical precision. The serious thinker question is hard to answer from an abstract alone, but there's no internal contradiction or obvious circularity, so I lean yes until proven otherwise.","headline":"The attack surface is real and worth exploring, but the abstract's headline numbers are unverified and the missing raw-RAG baseline undermines the attribution to graph construction.","tokens_in":1755,"tokens_out":1725,"would_cite":false,"duration_ms":20137,"reading_group":"maybe","serious_thinker":"unclear","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that changing a few words in source text can poison GraphRAG's knowledge graph and mislead downstream LLM answers.","keywords":["knowledge poisoning","GraphRAG","retrieval-augmented generation","knowledge graph construction","LLM security","adversarial attacks","question answering"],"falsifier":"Run the Universal KPA on a public QA dataset with a different extraction LLM than the one used in the paper, keeping the 0.05% word-modification budget, and check whether QA accuracy still collapses from 95% to 50%. If the drop is much smaller, the claim of universality fails. Alternatively, test whether a simple provenance filter that flags any edit to source text before ingestion detects both attacks.","tokens_in":812,"feed_emoji":"🎯","tokens_out":3361,"duration_ms":33969,"temperature":0.7,"pith_summary":"This paper claims that the graph-construction stage of GraphRAG is a practical target for knowledge poisoning. Because GraphRAG relies on an LLM to convert raw text into a knowledge graph, an attacker who subtly alters the source text can change the constructed graph and steer downstream question answering. The proposed Targeted KPA controls specific QA outcomes with a reported success rate of 93.1%, while the Universal KPA alters fewer than 0.05% of words and drops QA accuracy from 95% to 50%. The paper further reports that state-of-the-art defenses fail to detect these attacks, suggesting that securing the extraction step remains an open problem. A sympathetic reader would care because GraphRAG is promoted for accuracy and explainability, and this work identifies a cheap way to break both.","feed_headline":"Poisoning GraphRAG takes only a few altered words","feed_subtitle":"Targeted edits steer QA outcomes 93.1% of the time; universal edits drop accuracy from 95% to 50%.","key_machinery":"The machinery is the LLM-based knowledge extraction step that builds the graph from raw text, paired with two attack strategies. TKPA performs graph-theoretic analysis of the constructed graph to identify vulnerable nodes and then uses LLMs to rewrite the surrounding narratives. UKPA exploits linguistic cues such as pronouns and dependency relations to choose globally influential words whose alteration disrupts structural integrity. The extracted knowledge graph is the central object: the attacks work by changing what the graph contains before retrieval and generation ever run.","core_discovery":"The central discovery is that the LLM-based knowledge extraction step, not the retrieval or generation step, is the vulnerable surface in GraphRAG pipelines. Removing or rewriting a handful of words in the source narratives changes which entities and relations land in the graph. TKPA uses graph-theoretic measures to locate vulnerable nodes and rewrites the corresponding text to force a desired answer, while UKPA targets pronouns and dependency relations to globally disrupt graph structure. The paper's reported numbers—93.1% targeted steering success, a 45-point accuracy collapse under a 0.05% text modification, and failed defenses—are the evidence offered that this attack surface is real and","pith_inferences":["The same poisoning mechanism likely applies to any system that builds structured knowledge (graphs, tables, triples) from untrusted text via an LLM, not just GraphRAG.","The reported success rates may be sensitive to the unstated test configuration; a natural testable extension is to rerun both attacks across different extraction LLMs and datasets to map where the attack surface shrinks.","A concrete mitigation worth testing is to verify source-text provenance or to use a separate model to check whether extracted entities and relations are consistent with the original wording before the graph is finalized."],"forward_implications":["GraphRAG pipelines that ingest untrusted source text are exposed to poisoning before any retrieval or generation happens.","The attacks can be carried out while preserving fluency and naturalness, making them hard to spot by human readers.","Defenses that inspect only the final answer or the retrieved subgraph are unlikely to catch the poisoning, since the manipulation is at the extraction stage.","If the reported effect sizes hold generally, QA systems built on GraphRAG face a severe robustness gap that is not covered by current defense suites."],"supporting_citations":[],"fun_headline_variants":["Few-word edits poison GraphRAG, derail Q&A","GraphRAG poisoned by tweaking 0.05% of text","93% success: steering answers by rewording source","Universal attack collapses GraphRAG accuracy 95%→50%","Defenses fail against GraphRAG knowledge poisoning"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The central claim depends on the attack's reported effect sizes generalizing beyond the specific, unnamed test configuration, and on the attacker being able to alter source text before graph construction without any provenance check or human review.","fun_headline_variants_meta":{"raw":{"variants":["Few-word edits poison GraphRAG, derail Q&A","GraphRAG poisoned by tweaking 0.05% of text","93% success: steering answers by rewording source","Universal attack collapses GraphRAG accuracy 95%→50%","Defenses fail against GraphRAG knowledge poisoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000489,"raw_usage":{"total_tokens":2267,"prompt_tokens":792,"completion_tokens":1475,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":1391}},"tokens_in":536,"tokens_out":1475,"duration_ms":9919,"temperature":1.0,"reasoning_tokens":1391,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T00:44:26.871691+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the Universal KPA on a public QA dataset with a different extraction LLM than the one used in the paper, keeping the 0.05% word-modification budget, and check whether QA accuracy still collapses from 95% to 50%. If the drop is much smaller, the claim of universality fails. Alternatively, test whether a simple provenance filter that flags any edit to source text before ingestion detects both attacks.","supporting_citations":[],"review_version":1}