{"id":"d3a129f2-7ef1-4d3c-8163-7fa52bf8c8b9","arxiv_id":"2608.08944","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Adding missing support repairs one third of joint-eligible RAG failures and deleting verified nonsupport repairs one seventh, but the exact counterfactual response is not recoverable from the original failure and varies across readers.","lead":"A new audit called Pair-ID measures whether a failed RAG answer would be fixed by adding missing evidence or deleting irrelevant evidence, and finds both fixes work in a meaningful share of cases. It also shows the original failure only weakly predicts which fix works, and that the answer depends on the reading model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The prevalence estimates stand or fall with the oracle definition of missing support and verified nonsupport; incomplete gold annotations or NLI mislabeling would change what the 197/600 and 162/1,190 rates actually measure.","rationale":"The reader's weakest assumption is exactly the oracle-edit validity, and I agree. The paper's internal scoping ('benchmark support semantics', 'oracle-like') prevents this from being an internal inconsistency, but it remains the point where the central claim is least secure: all four headline quantities (addition, deletion, sham contrasts, and the non-residual response classes) inherit the definition of 'missing support' and 'verified nonsupport'. A wrong definition changes the object being measured, not just the confidence interval. The paper's own Section 6 flag is a strength but not a test. I considered the non-probability hash sample as an alternative; it is disclosed and the claims are explicitly sample-scoped, so it is less load-bearing than the meaning of the interventions. The proposed audit of 200 cases would settle whether the oracle assumption holds at a rate that matters; until then CONDITIONAL remains the right verdict.","tokens_in":12146,"tokens_out":17455,"duration_ms":179856,"concrete_test":"Take a random or hash-disjoint subsample of 100 JOINT and 100 deletion-eligible failures from the 1,190 valid cases. For each, an independent annotator (or a strong open-book LLM with access to the source corpora) should determine (i) whether the correct answer can be derived from an unannotated support chain already present in S_q or in public text, and (ii) whether any paragraph placed in N_q contains a bridge entity or reasoning step needed for the multi-hop derivation. If misclassification is below, say, 5%, the oracle assumption is secure; if it is high enough to move 0.328 or 0.136 outside their reported CIs, the prevalence claims must be re-expressed as responses to benchmark-defined edits, not to genuine evidence repair.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central object is a counterfactual response to two evidence repairs, so the repairs must correspond to genuine missing support and genuine nonsupport. The gold annotations define G_q; the DeBERTa-v3 NLI verifier defines N_q. Section 6 concedes \"Unannotated alternative support chains may make a paragraph look more dispensable than it is\" and calls the edits oracle-like. This is not a minor boundary: if an alternative support chain already exists for a JOINT failure, then adding the gold paragraph is not repairing the actual missing-evidence defect, and the 0.328 addition rate counts responses to a repair that was not the one called for. The NLI verifier only tests whether a paragraph entails the final answer alias; a paragraph containing a needed bridge entity for a multi-hop question is labeled nonsupport even if deleting it damages reasoning, so the 0.136 deletion rate measures something other than removing irrelevant material. The direction and magnitude of the resulting bias are unknown because the paper does not validate G_q completeness or verifier labels against an independent standard. Since the rates are the empirical core of the paper, this assumption is load-bearing even though it is explicitly disclosed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces Pair-ID, an offline audit that measures, for an already-failed retrieval-augmented generation run, the counterfactual response to two crossed evidence interventions: adding missing support and deleting verified nonsupport. From a complete funnel of 19,981 HotpotQA and 2WikiMultiHopQA queries, 11,105 native-eligible Qwen failures are identified; a prospectively fixed SHA-256 ordering selects 1,200, of which 1,190 regenerate validly. The main results are that support addition repairs 197/600 JOINT failures (0.328, 95% CI [0.292, 0.367]), verified-nonsupport deletion repairs 162/1,190 pooled failures (0.136, [0.117, 0.155]), length- and position-matched sham contrasts remain positive (0.223 and 0.101), an observation-only full-context predictor improves per-cell Brier but exact-vector accuracy (0.637) does not beat the majority-vector baseline (0.646), and four readers reproduce marginal factor sensitivities while pairwise exact-vector agreement is only 0.675–0.765. The paper concludes that evidence sensitivity is a frame-scoped, reader-conditional empirical object rather than an impossibility result or a deployable repair policy.","tokens_in":12361,"tokens_out":8713,"duration_ms":84484,"significance":"If the results hold, the paper makes a useful methodological contribution: it defines a clean same-failure factorial estimand, separates diagnostic validity from action regret, uses prospectively fixed selection, includes matched semantic shams, and reports nested out-of-fold predictions with disclosed limitations (worst-case bounds for nonreproductions, interaction interval including zero, reader-specific kappa). The distinction between marginal factor sensitivity and exact-vector recovery is an important caution for RAG diagnosis. The paper's transparency—including the explicit statement that the hash sample is not a probability sample and that edits are oracle-like—is a strength. The main value is as a reference audit method and a set of bounded empirical findings, not as a universal claim about RAG failures.","major_comments":[{"comment":"The two headline repair rates (197/600 addition, 162/1,190 deletion) are computed under an oracle-like definition of missing support (benchmark gold annotations) and verified nonsupport (DeBERTa-v3 NLI argmax on the final answer alias). Section 6 concedes that 'Unannotated alternative support chains may make a paragraph look more dispensable than it is,' and the NLI check does not test whether a paragraph supplies a bridge entity needed for multi-hop reasoning. Because neither G_q nor N_q is validated against an independent standard, the measured rates may not correspond to genuine evidence repair, and the direction of bias is unknown. This is a load-bearing construct-validity issue for the prevalence claims; the revision should either add a validation sample (e.g., human annotation or a second NLI) or, at minimum, relabel the rates as 'response to adding annotated gold support' and 'response to deleting NLI-verified nonsupport' throughout, including the abstract.","section":"§3.1, §6, Table 2"},{"comment":"The 95% confidence intervals are computed with stratified bootstrap on a deterministic SHA-256-ordered sample without an independently randomized seed. The paper honestly calls this 'a prospectively fixed hash sample rather than a simple probability sample,' but the abstract and results sections still present the intervals as if they support inference to the eligible-failure frame. Because no exchangeability or pseudo-randomization assumption is stated or defended, the intervals should be either justified (for example, by arguing SHA-256 order is effectively random with respect to the estimand) or replaced by the reported worst-case ranges when claims are about the frame. This affects the interpretation of the prevalence estimates.","section":"§3.1, Table 2, §4.1"}],"minor_comments":[{"comment":"The sentence 'The loss-aligned repair is substantially stronger: it reaches 0.100 ... against 0.101 for constant always-joint' is misleading; the difference is 0.001, and the attack selects joint on 1,171 of 1,190 queries. Suggest rewording to 'reaches 0.100, essentially equal to the 0.101 constant policy' and separating the improvement over the 0.223 complete-context attack.","section":"§4.3"},{"comment":"The term 'nested OOF estimates' in Table 4 is not defined in the caption; the text in §3.3 defines it, but readers of the table alone will not know that OOF means nested out-of-fold and that the metrics are evaluated on held-out outer folds.","section":"§3.3, Table 4"},{"comment":"The sentence 'any pair exceeding 512 tokens is excluded without an argmax' should state how many candidate pairs are excluded this way and whether the exclusion could systematically change N_q; the terminal-state count of 5 is for queries, not for pairs.","section":"§3.1"},{"comment":"The reported scientific GPU time of 1.177 RTX-5090 hours appears inconsistent with the described workload (19,981 screening queries, 4,800 primary regenerations, 9,600 reader-regeneration cells, 3,600 interaction-audit outputs, and matched controls). Please verify the units/scope of this figure.","section":"§7"},{"comment":"The statement 'matched prompt lengths differ by less than 0.2 tokens on average' would be more informative with a maximum or distribution, since an average difference may hide large per-query deviations.","section":"§3.2"},{"comment":"The heatmap panel would benefit from a note that the matrix is symmetric and that JOINT-only agreement ranges are given in text, not only in the caption.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The paper's transparency is a genuine strength, but the construct-validity concern about oracle edits is central enough that the revision should be seen by the original referees. I would not recommend rejection; a validation study or a consistently operational re-labeling would make the claims defensible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Plain take: this is a careful, unusually honest empirical audit of counterfactual response to two evidence edits in RAG failures. The crossed two-factor same-failure response vector is genuinely new as a measured object, and the hash-selected frame-sample design is a real improvement over quota samples. The paper also does something rare: it gives worst-case bounds for the lost regenerations, reports that the hash sample is not a probability sample, shows the interaction interval includes zero, and gives reader-specific agreement numbers. It explicitly declines to claim an impossibility result or a runtime policy. That discipline is the paper's main virtue.\n\nWhat's new: prior work does addition, deletion, matching, and action prediction; Pair-ID crosses support addition with nonsupport deletion on the same failed query and treats the resulting three-bit vector as an estimable object. The prevalence estimates (197/600 and 162/1,190) and the four-reader agreement ranges are the substantive output. The observation-only attack is nested out-of-fold, so labels don't leak.\n\nSoft spots, in order of importance. First, the oracle edits are load-bearing. 'Missing support' means 'gold support absent from the retrieved context' per benchmark annotations; if an alternative support chain already exists, adding the gold paragraph isn't repairing the actual defect. Similarly, the DeBERTa-v3 verifier only tests whether a paragraph entails the final answer plus support sentences, so a needed bridge paragraph gets labeled nonsupport. The paper discloses both in Section 6, but this isn't a minor boundary: it determines whether 0.328 and 0.136 are rates of repairing genuine evidence failures or rates of response to specific operational edits. I'd want an independent check of G_q completeness and verifier labels against, say, human annotation on a subsample. Second, the matched shams are doing a lot of work: sham support addition repairs 22.3% of failures, so the uniquely semantic component of the headline addition rate is about 10 points. The paper reports this, but the abstract underplays it. Third, there's no artifact URL or commit hash, so the reproducibility claims can't be checked; that's a mechanical but real problem.\n\nThese are real limitations but not fatal ones. The central argument—within the defined frame and definitions, evidence sensitivity occurs at meaningful rates and is only partially recoverable from the observed failure—holds up. The paper's scope claims are appropriately modest.\n\nThis is for RAG evaluators and people building failure-diagnosis tools. It deserves a serious referee. If the artifact appears, I'd cite it; even without, the method and honest numbers are worth putting in front of a review committee.","headline":"A careful, honestly bounded audit of counterfactual response to evidence edits in RAG; the headline rates are real but only under the paper's own oracle definitions, and it deserves a serious referee.","tokens_in":12885,"tokens_out":7261,"would_cite":true,"duration_ms":71411,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A failed RAG answer is not one diagnosable state: crossing two evidence edits on the same query reveals a reader-conditional response vector.","keywords":["retrieval-augmented generation","counterfactual response","failure diagnosis","evidence intervention","multihop question answering","response vector","reader dependence","offline audit"],"falsifier":"Re-annotate a subset of the same 1,190 failures with human-expanded gold support sets and rerun the addition intervention; if the repair rate drops below the reported 0.292 lower confidence bound, the prevalence estimate is an artifact of incomplete gold annotations.","tokens_in":11910,"feed_emoji":"🧩","tokens_out":10855,"duration_ms":101007,"temperature":0.7,"pith_summary":"Retrieval-augmented generation (RAG) systems can fail, but the observed failure does not reveal which unseen repair would have fixed it. Pair-ID is an offline audit that holds one query, its retrieval state, and its reader fixed, then independently toggles two evidence operations—adding missing support and deleting verified nonsupport—to map each failure to a counterfactual response vector. In a prospectively hash-selected sample of 1,190 eligible failures, adding missing support repairs 197 of 600 joint-eligible cases (0.328) and deleting verified nonsupport repairs 162 of the 1,190 (0.136); matched shams confirm both effects are semantic rather than prompt-length artifacts. The paper establishes that these response rates are real but frame-scoped: the exact three-bit response is only partly recoverable from the original view, and four readers reproduce the marginal sensitivities while disagreeing on the exact vector.","feed_headline":"Adding missing support repairs 1 in 3 studied RAG failures","feed_subtitle":"Deleting bad evidence fixes 1 in 7; exact response depends on the reader","key_machinery":"The load-bearing machinery is Pair-ID's factorial response audit. Its object is the same-failure counterfactual response vector: the four outcomes $Y(a,d)$ obtained by toggling missing-support addition $a$ and verified-nonsupport deletion $d$ on an already-failed query with everything else held constant. The design separates the JOINT stratum, where all four cells are generated, from the COMPLETE stratum, where the addition dimension is structurally aliased, and uses a prospectively fixed SHA-256 selection to define the eligible-failure frame before any response is observed. Matched semantic shams replace each edit with length- and position-matched non-semantic material to isolate content effects from prompt-size effects, and the same design is re-run on four readers to test reader dependence. This machinery turns 'what would fix this failure?' into a measurable, alias-aware exact-match response on a defined sample.","core_discovery":"The central discovery is a same-failure counterfactual response object. For a query that already failed, the answer under each combination of support addition and verified-nonsupport deletion—$Y(a,d)$ with $(a,d)\\in\\{0,1\\}^2$—is generated while keeping the query, retrieval order, rendering, reader, and decoding fixed; the JOINT stratum has all four cells, the COMPLETE stratum has only two because there is no missing support. In the hash-selected eligible-failure frame, support addition repairs 0.328 of JOINT failures and deletion repairs 0.136 of all deletion-eligible failures, and the vector distribution shows 64.6% of failures remain wrong under every edit, 12.8% are addition-sensitive, 10.8% deletion-sensitive, 7.5% complementary, 2.1% substitutable, and 2.3% antagonistic. Exact agreement across four readers on the three-bit vector is 0.675–0.765 pooled and 0.538–0.691 JOINT-only, so the response is reader-conditional. The observed failure is not information-free—per-cell AUROC of 0.678 and Brier 0.152 improve on a marginal baseline—but exact-vector accuracy of 0.637 does not exceed the 0.646 majority-vector baseline.","pith_inferences":["This suggests that RAG repair evaluations should report per-reader counterfactual labels rather than a single query-level cause; cross-reader transfer of a learned diagnosis is unsafe.","A testable extension is to run the same paired-intervention design on long-form generation and on retrieval states produced by a live retriever, since the oracle-like edits here may overstate how often real repair succeeds.","The hash-selection protocol itself is reusable: any group can pre-register a fixed seed and selection order, then publish comparable frame-scoped rates, turning single-lab audits into a cumulative evidence base.","The near-majority-baseline vector recovery is a challenge result, not a proof of impossibility; a stronger observational model that uses retrieval scores, citation overlap, or reader-internal states could close the gap."],"forward_implications":["If a failure is addition-sensitive, adding the missing gold support flips the answer to correct; in the defined frame this occurs for about 32.8% of JOINT-eligible failures and for all four readers, so retrieval-side fixes have real repair value.","A near-constant 'apply both edits' action achieves low mean regret (0.101) without explaining the failure; 421 of 1,190 failures lie outside the residual class, so action utility and diagnostic validity are different quantities.","Response labels are not query-intrinsic: four readers agree on the marginal factors but disagree on the exact three-bit vector, so a diagnosis learned for one reader does not transfer safely to another.","The original failed state contains partial signal: full-context observation models improve per-cell ranking and calibration, but the exact response vector is not recovered beyond the majority baseline, bounding what observation-only diagnosis can promise.","The reported rates condition on the eligible-failure frame and on regenerated failures, so extending them to other benchmarks, retrievers, or readers requires a new frame-scoped audit."],"supporting_citations":[{"why":"supplies queries and gold support annotations that define missing support on one benchmark","marker":"[22]"},{"why":"supplies the second benchmark's queries and annotated support chains for the eligible-failure frame","marker":"[6]"},{"why":"the screening reader whose failures define the eligible-failure frame and the primary reader for the hash sample","marker":"[15]"},{"why":"the pinned NLI verifier that determines which paragraphs count as verified nonsupport","marker":"[5]"},{"why":"evidence that passage utility varies by generator, supporting the reader-conditional reading of response vectors","marker":"[24]"},{"why":"the encoder that maps original fields into embeddings for the observation-only recovery attack","marker":"[21]"},{"why":"one of the four readers whose responses define the reader replication panel","marker":"[2]"},{"why":"one of the four readers in the replication panel","marker":"[3]"},{"why":"one of the four readers in the replication panel","marker":"[8]"}],"fun_headline_variants":["Adding support fixes 1/3 RAG failures, deleting bad evidence fixes 1/7","Counterfactual audit: 33% of RAG errors fixed by adding support, 14% by deletion","RAG failure repair rates: 32.8% add, 13.6% delete, but reader-dependent","Same-failure audit: adding support repairs 1/3, deletion 1/7, reader matters","RAG fix: 1/3 addition, 1/7 deletion, but reader changes outcome"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The interventions are oracle-like: missing support is whatever the benchmark gold annotations say, and nonsupport is whatever the pinned NLI verifier rejects, so if those labels mischaracterize a query's evidence, the measured repair rates do not reflect genuine evidence repair.","fun_headline_variants_meta":{"raw":{"variants":["Adding support fixes 1/3 RAG failures, deleting bad evidence fixes 1/7","Counterfactual audit: 33% of RAG errors fixed by adding support, 14% by deletion","RAG failure repair rates: 32.8% add, 13.6% delete, but reader-dependent","Same-failure audit: adding support repairs 1/3, deletion 1/7, reader matters","RAG fix: 1/3 addition, 1/7 deletion, but reader changes outcome"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000786,"raw_usage":{"total_tokens":3583,"prompt_tokens":1176,"completion_tokens":2407,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":792,"completion_tokens_details":{"reasoning_tokens":2276}},"tokens_in":792,"tokens_out":2407,"duration_ms":19905,"temperature":1.0,"reasoning_tokens":2276,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:19:07.511718+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a subset of the same 1,190 failures with human-expanded gold support sets and rerun the addition intervention; if the repair rate drops below the reported 0.292 lower confidence bound, the prevalence estimate is an artifact of incomplete gold annotations.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"evidence that passage utility varies by generator, supporting the reader-conditional reading of response vectors"},{"cited_title":"Skill-RAG: Failure-State-Aware Retrieval Augmentation via Hidden-State Probing and Skill Routing","cited_arxiv_id":"2604.15771","evidence_quote":"the encoder that maps original fields into embeddings for the observation-only recovery attack"}],"review_version":1}