{"id":"fbb4802d-83ec-4bba-883b-26fce61c73a0","arxiv_id":"2509.09713","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"HANRAG, a single-model router-decomposer-filter for RAG, reports higher accuracy and fewer retrieval steps than Adaptive-RAG across single-hop, multi-hop complex, and a new compound QA benchmark.","lead":"This paper introduces HANRAG, a retrieval-augmented generation framework that sorts questions into four types and uses a single trained Revelator model to route, decompose, and filter them. It reports accuracy gains over prior adaptive RAG methods on single-hop and multi-hop QA benchmarks, though the main comparison uses a stronger base LLM and the new compound benchmark is self-created.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1 reports EM values exceeding F1 for HANRAG on SQuAD and Natural Questions, which is mathematically impossible; the headline single-hop results are therefore not credible as printed.","rationale":"The reader's verdict of CONDITIONAL was based on plausible but unverified performance, with the relevance discriminator's generalization identified as the weakest assumption. I agree that the discriminator is important, and the ablation in Appendix E shows it contributes meaningfully. However, the paper also provides empirical evidence across multiple benchmarks and an oracle analysis suggesting the routing and discriminator generalize reasonably. The more decisive problem is internal: Table 1 contains EM/F1 pairs that violate a mathematical invariant. This is not a matter of interpretation or distribution shift; it is a direct signal that the reported evaluation numbers are flawed. Since the abstract and Section 6 rest on these numbers to claim superiority on single-hop tasks, the central claim is unsupported until the discrepancy is resolved. The concrete test of recomputing metrics from raw outputs will settle whether this is a typo or a systematic error. If corrected numbers still show gains, the paper's contribution may survive, but as printed the primary evidence is unreliable. Therefore, the verdict should be moved to UNVERDICTED: the central claim is not yet assessable from the published evidence. This is not an attack on the authors; it is a request for the minimal data verification needed before confidence can be assigned.","tokens_in":21357,"tokens_out":5202,"duration_ms":46763,"concrete_test":"Obtain the raw prediction strings and ground-truth answers used for Table 1 and recompute EM, F1, and Acc with the standard SQuAD evaluation script (or the exact script described in the paper). If the recomputed HANRAG row differs from the printed values, the reported results are invalid and must be corrected before the central claim can be assessed.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"Table 1 lists HANRAG's SQuAD EM as 39.80 with F1 39.76, and Natural Questions EM as 56.40 with F1 49.12. For any individual prediction, an exact match forces F1 = 1; otherwise F1 is non-negative and can exceed a zero EM, never be lower. Averaging preserves the inequality, so mean EM cannot exceed mean F1. These printed values violate this invariant by margins of 0.04 and 7.28 points respectively. Section 6 explicitly states that HANRAG 'achieves optimal results across all benchmark evaluation metrics' on single-hop datasets, using Table 1 as the evidence. Because the central claim depends on these single-hop numbers, and the same evaluation pipeline produces the multi-hop and compound results, the inconsistency casts doubt on the entire reported evaluation. The error could be a typo or a flawed metric script, but it is load-bearing: without corrected numbers, the state-of-the-art claim is unsupported as printed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HANRAG, a retrieval-augmented generation framework built around a fine-tuned 'Revelator' model that routes queries into four categories (straightforward, single-step, compound, and complex), decomposes compound queries into independent sub-queries, refines seed questions for complex queries, and filters retrieved documents by relevance before generation. The authors contribute a synthetic compound-question benchmark, use Qwen2-72B-instruct to generate training labels for routing, decomposition, refinement, relevance, and ending decisions, and evaluate on SQuAD, Natural Questions, TriviaQA, MuSiQue, HotpotQA, 2WikiMultihopQA, and their own compound benchmark. They report consistent gains over Adaptive-RAG, IRCoT, Self-RAG, and other baselines, with fewer retrieval steps, and include ablations and oracle-routing experiments.","tokens_in":21615,"tokens_out":5677,"duration_ms":45030,"significance":"If the reported results are correct, the paper would make a useful contribution by distinguishing compound from complex multi-hop queries, showing that asynchronous parallel retrieval for compound queries reduces retrieval steps, and demonstrating that relevance filtering improves robustness to noisy retrieved documents. The ablation study and oracle-routing analysis are informative, and the comparison spans several standard benchmarks. However, the central empirical claim is currently undermined by impossible metric pairs in Table 1, inconsistent step counts between Tables 2 and 6, and a main comparison that uses a stronger base LLM for HANRAG than for the key baseline; the self-built compound benchmark is also used for both training and evaluation. These issues must be resolved before the state-of-the-art claim is supported.","major_comments":[{"comment":"Table 1 reports HANRAG SQuAD EM=39.80 with F1=39.76 and Natural Questions EM=56.40 with F1=49.12. For token-level exact-match scoring, EM cannot exceed F1 for any individual prediction, and the inequality is preserved by averaging; the printed values therefore cannot be correct as computed. Because Section 6 uses Table 1 to assert that HANRAG 'achieves optimal results across all benchmark evaluation metrics' for single-hop datasets, this inconsistency is load-bearing. The authors should re-run the metric computation, report corrected values, and check the same evaluation pipeline for the multi-hop and compound tables.","section":"Table 1, Section 6"},{"comment":"The headline comparison is asymmetric in base model: HANRAG uses Llama-3.1-8B-instruct for both the Revelator and the generator, while Adaptive-RAG uses FLAN-T5-XL (3B). The HANRAG-Fair rows in Tables 1 and 2, which use FLAN-T5-XL for HANRAG, show substantially smaller gains (e.g., SQuAD EM drops from 39.80 to 32.60 versus Adaptive-RAG's 26.80; TriviaQA EM is 52.40 versus Adaptive-RAG's 52.20). The main text should base its state-of-the-art claims on the fair variant or justify why the larger-model comparison is the appropriate one. As printed, the 'optimal across all metrics' claim conflates gains from HANRAG with gains from a stronger base LLM.","section":"Tables 1 and 2; Section 5 (Implementation Details)"},{"comment":"The compound multi-hop benchmark is constructed by the authors using a Qwen2-72B-instruct pipeline, and the same benchmark's train split is used to train the router and decomposer while its test split is used for evaluation. A disjoint train/test split within a self-generated distribution does not rule out overfitting to the specific generation pipeline or topic distribution, so the reported 71.76 compound accuracy and 1.24 steps (Table 2) may not transfer to real compound queries. The paper should validate compound handling on an external or independently constructed compound-question set, or at minimum report a human evaluation and an analysis of distributional overlap.","section":"Section 5, Appendix B.1, Section 4.2"},{"comment":"The ablation table reports HANRAG's MuSiQue steps as 3.01, while Table 2 reports HANRAG's MuSiQue steps as 2.45 for what appears to be the same configuration. Since step reduction is a central efficiency claim, this discrepancy must be reconciled; if the ablation uses a different early-stopping or retriever setting, that should be stated explicitly. The same table also shows that removing the relevance discriminator lowers accuracy from 43.2 to 37.8 but changes EM by only 4.6 points, which deserves interpretation in the text.","section":"Table 6 vs Table 2; Section 6"}],"minor_comments":[{"comment":"The section heading is spelled 'Methdology' and should read 'Methodology'.","section":"Section 4 heading"},{"comment":"Algorithm 2 calls 'Revelator(q_i, D)' to produce the sub-answer, but elsewhere the retrieval-and-generation step is described as ANRAG; this is likely a typo but should be corrected for reproducibility.","section":"Algorithm 2, line 16"},{"comment":"In Appendix C, 'teble 5' should be 'Table 5'.","section":"Appendix C"},{"comment":"The case-study tables label Adaptive-RAG's handling of complex queries as 'Asynchronous retrieval', which conflicts with the paper's definition of complex queries as synchronous iterative retrieval; the terminology should be made consistent.","section":"Tables 8 and 9"},{"comment":"Natural Questions and TriviaQA are cited as '(?)' in the datasets paragraph; proper references should be supplied.","section":"Section 5 (Datasets)"},{"comment":"Section 4.2 states 9,741 straightforward-query training samples for the router, while Appendix B.2 states 12,247; the discrepancy should be resolved.","section":"Section 4.2 vs Appendix B.2"},{"comment":"Algorithm 1's output line says 'Top 3 relevant passages for Q', but the algorithm returns LLM(D_rel), not the three passages; the output specification should be corrected.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The impossible EM/F1 pairs in Table 1 are a red flag for the evaluation pipeline. If the authors cannot produce corrected numbers that satisfy EM <= F1 on every dataset, I would be inclined to move to reject. The fair-variant comparison should become the primary comparison, and the compound benchmark needs external validation. The framework idea is reasonable, but the current presentation is not acceptable as is."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the thing you need to know: this is a coherent systems paper with a genuinely useful efficiency story, but the headline numbers in Table 1 are not credible as printed, and the authors need to fix that before you trust the single-hop claims.\n\nThe new part is the Revelator, a single fine-tuned model that routes queries, decomposes compound questions, refines seed questions for complex ones, filters retrieved documents by relevance, and decides when to stop iterating. That combination is new, and the distinction between compound and complex multi-hop queries is worth taking seriously. The asynchronous retrieval for compound queries gives real step reductions—about half the retrieval steps on their compound benchmark. And importantly, the fair comparison (HANRAG-Fair, same base model as Adaptive-RAG) still shows gains on most benchmarks, which is the evidence the paper should lead with.\n\nThe soft spots are real, though. First, Table 1 reports SQuAD EM 39.80 with F1 39.76, and Natural Questions EM 56.40 with F1 49.12. EM can never exceed F1, instance-wise or on averages. These two cells are impossible, and the paper's claim that HANRAG 'achieves optimal results across all benchmark evaluation metrics' on single-hop rests on exactly that table. This is a load-bearing data error, not a typo to shrug off.\n\nSecond, the main comparison is not apples-to-apples: HANRAG uses Llama-3.1-8B while Adaptive-RAG uses FLAN-T5-XL. The HANRAG-Fair row narrows but does not close the gap, so the method probably does help, but the abstract's 'superior performance' overstates what the printed comparisons support.\n\nThird, the compound benchmark is built by the authors, used for both training and testing, and not released. They claim no train/test overlap, but an external reader can't check the pipeline. The compound results should be treated as suggestive, not definitive.\n\nMinor: no error bars or seeds, and a few references are placeholders (Natural Questions, TriviaQA). That's sloppy but fixable.\n\nBottom line: anyone building practical RAG systems for multi-hop QA should read this for the efficiency idea and the compound/complex distinction. It deserves a serious referee, but the referee should send it back for corrected Table 1, a headline fair comparison, released benchmark/code, and variance reporting. With those fixes, it could be a solid contribution.","headline":"Worth engaging with: a real efficiency idea and a sensible compound/complex distinction, but the single-hop results are not credible as printed and the fair comparison should lead.","tokens_in":22101,"tokens_out":2949,"would_cite":false,"duration_ms":24457,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HANRAG claims that a single trained 'Revelator' agent can route, decompose, and noise-filter queries well enough to beat Adaptive-RAG on single-hop and multi-hop question answering while using fewer retrieval steps.","keywords":["retrieval-augmented generation","multi-hop question answering","query routing","query decomposition","relevance filtering","noise resistance","compound queries","complex queries"],"falsifier":"Take the MuSiQue test set and inject, for each query, several retrieved passages that share named entities and topical words with the query but do not contain the answer, then compare HANRAG with and without the relevance filter. If accuracy falls to the no-filter ablation level (37.8 Acc) rather than holding near the full-system level (43.2 Acc), the claimed noise resistance does not generalize beyond the training distribution of the relevance discriminator.","tokens_in":21175,"feed_emoji":"🔍","tokens_out":11388,"duration_ms":85802,"temperature":0.7,"pith_summary":"The paper tries to establish that a single trained agent, the Revelator, can make retrieval-augmented question answering both more accurate and cheaper by deciding what kind of question it faces before retrieving. The key move is to split multi-hop questions into compound ones, whose sub-questions are independent and answerable in parallel, and complex ones, which require a dependent chain of reasoning answered step by step. A relevance filter then removes retrieved documents judged irrelevant, preventing noise from accumulating across iterations. The reported result is that HANRAG beats Adaptive-RAG and other baselines on EM, F1, and accuracy across single-hop, multi-hop, and a newly built compound benchmark, while using fewer retrieval steps. If the claim is right, a relatively small fine-tuned model can replace hand-tuned retrieval policies.","feed_headline":"HANRAG: one 'revelator' agent beats Adaptive-RAG on QA benchmarks","feed_subtitle":"A single trained agent routes, decomposes, and filters queries, claiming top accuracy on single- and multi-hop QA with fewer steps.","key_machinery":"The load-bearing object is the Revelator, a single fine-tuned 8B instruction-following language model that performs five judgments in one network: routing the query class, decomposing compound questions into sub-questions, refining the next seed question for complex questions, deciding whether a retrieved document is relevant, and deciding whether the iterative loop can end. The paper also reports a fair-comparison variant with a smaller 3B generator to show that the gains are not simply a bigger generator. The Revelator lets the framework replace a fixed iterative retrieval policy with a per-query pipeline, and its relevance judgment keeps noisy documents away from the generator.","core_discovery":"On the paper's own terms, the central claim is that the dominant source of error in multi-hop RAG is not the retriever but the absence of query-type awareness and post-retrieval noise control. HANRAG's Revelator first classifies each query into one of four routes, then acts as decomposer for compound questions, as refiner of seed questions for complex questions, and as a relevance discriminator after every retrieval. The compound-versus-complex distinction is presented as the conceptual fix for wasted iterative retrieval: compound sub-questions are retrieved asynchronously in one round, while complex seed questions are refined and retrieved synchronously until an ending discriminator halts the loop. Across three single-hop, three complex multi-hop, and one new compound benchmark, HANRAG is reported to outperform Adaptive-RAG on EM, F1, and accuracy, with the largest accuracy gain on the compound benchmark (71.76 vs 52.13) and with fewer average retrieval steps. The ablation on MuSiQue shows that the relevance discriminator and the refiner each carry a large part of the accuracy, while the ending discriminator is what keeps the step count low.","pith_inferences":["The compound-versus-complex split is a cheap textual signal that other pipelines could adopt without the full Revelator, since a classifier alone might reproduce much of the step reduction.","Because the relevance labels and the compound benchmark are both generated by a single large language model, the framework's generalization to user-issued queries and to documents outside Wikipedia's style remains the main untested boundary.","A natural extension is to plug the same Revelator into a dense retriever or a hybrid retriever; if the gains persist, the framework is retriever-agnostic rather than tuned to BM25's noise profile.","The reported routing accuracy of 83.93% suggests that a targeted study of routing errors, especially straightforward questions sent to retrieval, could yield further gains without new retrieval logic."],"forward_implications":["Compound multi-hop questions can be answered in close to one retrieval step instead of several, because sub-questions are retrieved in parallel and merged.","Post-retrieval relevance filtering improves single-hop accuracy even when the retriever is fixed: HANRAG's EM on SQuAD rises from Adaptive-RAG's 26.8 to 39.8.","Complex multi-hop questions gain accuracy from explicit seed-question refinement, and an ending discriminator prevents unnecessary retrieval-generation cycles.","On the newly constructed compound benchmark, the reported accuracy gain over Adaptive-RAG is 19.63 points with about 1.5 fewer retrieval steps on average."],"supporting_citations":[{"why":"Supplies the Adaptive-RAG baseline that HANRAG is compared against and the dataset setup for single-hop and multi-hop benchmarks.","marker":"Jeong et al., 2024"},{"why":"IRCoT is the iterative-retrieval baseline whose per-hop retrieve-generate loop HANRAG claims to outperform in steps and accuracy.","marker":"Trivedi et al., 2023"},{"why":"MuSiQue provides the complex multi-hop training data for the refiner and ending discriminator and one of the three complex evaluation benchmarks.","marker":"Trivedi et al., 2022"},{"why":"Qwen2-72B-instruct is used to annotate relevance labels and to generate the compound benchmark questions, shaping the Revelator's training signal.","marker":"Yang et al., 2024"},{"why":"The 8B instruction-tuned model family is the base for the Revelator and the untrained generator.","marker":"Grattafiori et al., 2024"},{"why":"BM25 is the shared retriever; HANRAG's gains are claimed on top of this fixed retrieval step.","marker":"Robertson et al., 1994"},{"why":"Self-RAG is included as an adaptive retrieval baseline with self-reflection, defining the comparison class HANRAG targets.","marker":"Asai et al., 2023"}],"fun_headline_variants":["One agent routes, decomposes, and filters for noise-free multi-hop QA","HANRAG's revelator: fewer steps, better answers on multi-hop QA","Route, decompose, denoise: HANRAG's single-agent fix for RAG noise","Compound vs complex: HANRAG's key insight for RAG retrieval","One agent beats Adaptive-RAG with smarter query routing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Revelator can recognize whether a retrieved document is relevant well enough to trust it on new queries, not just on the (query, document) pairs used to train it; the ablation shows that when that filter is removed, MuSiQue accuracy drops from 43.2 to 37.8, so the whole gain rests on this judgment generalizing.","fun_headline_variants_meta":{"raw":{"variants":["One agent routes, decomposes, and filters for noise-free multi-hop QA","HANRAG's revelator: fewer steps, better answers on multi-hop QA","Route, decompose, denoise: HANRAG's single-agent fix for RAG noise","Compound vs complex: HANRAG's key insight for RAG retrieval","One agent beats Adaptive-RAG with smarter query routing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000507,"raw_usage":{"total_tokens":2509,"prompt_tokens":1023,"completion_tokens":1486,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":639,"completion_tokens_details":{"reasoning_tokens":1385}},"tokens_in":639,"tokens_out":1486,"duration_ms":10403,"temperature":1.0,"reasoning_tokens":1385,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:16:28.911256+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the MuSiQue test set and inject, for each query, several retrieved passages that share named entities and topical words with the query but do not contain the answer, then compare HANRAG with and without the relevance filter. If accuracy falls to the no-filter ablation level (37.8 Acc) rather than holding near the full-system level (43.2 Acc), the claimed noise resistance does not generalize beyond the training distribution of the relevance discriminator.","supporting_citations":[],"review_version":2}