Dissecting Agentic RAG: A Component Ablation for Multi-Hop QA with a Local 7B Model

Sheroz Shaikh

REVIEW 1 major objections 2 minor 1 cited by

Dissecting Agentic RAG: A Component Ablation for Multi-Hop QA with a Local 7B Model

T0 review · 1 major / 2 minor · reviewed 2026-06-26 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fixed hybrid retrieval via reciprocal rank fusion outperforms rule-based adaptive routing in agentic RAG for multi-hop QA with a local 7B model.

desk verdict The ablation shows fixed hybrid retrieval beats this one rule-based adaptive router by 1.8 EM on HotpotQA with a 7B model, and two iterations capture most gains, but the adaptive result is tied to a weak heuristic. read the letter →

arxiv 2606.21553 v1 pith:LDVPAMZV submitted 2026-06-19 cs.CL cs.IR

Sheroz Shaikh This is my paper · ORCID

classification cs.CLcs.IR

keywords agenticRAGmulti-hopQAablationstudylocallanguagemodelsretrievalroutingquerydecompositionHotpot

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper runs a controlled ablation of an agentic RAG pipeline on 5,000 HotpotQA questions using only a local 7B model. It compares eight conditions that vary retrieval routing, iteration depth, query decomposition, and reranking. The results show fixed hybrid retrieval beats the adaptive router, two iterations capture nearly all the benefit of five, and decomposition plus reranking add smaller gains. This indicates that on a fixed local-model budget simpler fixed components often match or exceed more elaborate adaptive designs.

What carries the argument

Controlled ablation across eight conditions of the agentic RAG pipeline, isolating effects of fixed versus adaptive retrieval routing, retrieval loop depth, query decomposition, and reranking.

What would settle it

Replace the rule-based router with a learned router and re-run the comparison to check whether adaptive routing then outperforms fixed hybrid retrieval.

Watch

Extended reading notes

Core claim

The full pipeline reaches 53.2% EM and 61.6% F1. Fixed hybrid retrieval via reciprocal rank fusion consistently outperforms rule-based adaptive routing by 1.8 EM and 1.9 F1 points because the routing heuristic over-routes to BM25 on named entities present in nearly all sub-questions. Two retrieval iterations over decomposed sub-questions capture 95% of the gains from five iterations, while query decomposition and cross-encoder reranking each contribute statistically significant but smaller gains.

Load-bearing premise

The particular rule-based adaptive routing heuristic tested is representative of adaptive routing methods in general.

Editorial extensions

If this is right

Most of the performance gain comes from running a short retrieval loop rather than adaptive routing or many iterations.
On a fixed local-model budget, simpler and fixed choices turn out to be competitive with or better than their adaptive versions.
Query decomposition and cross-encoder reranking each add statistically significant but smaller gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A stronger learned router might reverse the ranking between fixed and adaptive approaches.
The same pattern of short loops sufficing may appear on other multi-hop QA datasets.
Reducing iteration depth could lower latency and compute with little accuracy loss.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

read the letter

The key things to know are that a fixed reciprocal rank fusion hybrid beats the tested rule-based adaptive router by 1.8 EM and 1.9 F1, and that two retrieval iterations on decomposed sub-questions get 95% of the lift from five. The full pipeline hits 53.2 EM / 61.6 F1 against a 43.1 / 54.0 single-pass baseline on 5000 HotpotQA distractor questions with Qwen2.5-7B-Instruct.

The work does a clean job of running controlled ablations across eight conditions and reporting concrete deltas plus p-values. Query decomposition and cross-encoder reranking each add statistically detectable but smaller gains. The mechanistic note that the router over-fires on named entities (common in nearly every sub-question) explains why adaptive underperformed here. All of this is done with only local models and no proprietary APIs, which matches the resource-constrained setting they target.

The soft spot is exactly the one the stress-test flags: the adaptive comparison rests on a single rule-based heuristic. That shows this particular router is not helpful, but it leaves open whether a learned router could reverse the ranking. The paper's broader takeaway that simpler fixed choices are competitive or better therefore depends on an untested assumption about the space of adaptive methods. The evaluation details (exact prompts, data splits, implementation of the router) are not visible in the abstract, so replication would require the code.

This is useful for people running multi-hop RAG on local 7B-class models who need guidance on which knobs actually matter. It is not a theoretical advance and the citation pattern is light, but the empirical measurements are internally consistent and falsifiable. It deserves peer review because the ablation design is straightforward and the reported numbers give practitioners something concrete to test against.

Referee Report

1 major / 2 minor

Summary. The paper reports a controlled ablation study of an agentic RAG pipeline for multi-hop QA on 5,000 HotpotQA distractor dev questions using a local Qwen2.5-7B-Instruct model. The full pipeline reaches EM=53.2% and F1=61.6% versus a single-pass dense baseline of 43.1%/54.0%. Across eight conditions it finds that fixed hybrid retrieval (RRF) outperforms a rule-based adaptive router by +1.8 EM / +1.9 F1 because the router over-routes to BM25 on named entities; two retrieval iterations capture 95% of the gain from five; and query decomposition plus cross-encoder reranking each add smaller but statistically significant improvements (p<0.01, p<0.001). The authors conclude that, on a fixed local-model budget, simpler fixed choices are competitive with or better than adaptive versions.

Significance. If the empirical comparisons hold, the work supplies concrete, reproducible evidence on component contributions in resource-constrained agentic RAG, showing that most gains derive from a short retrieval loop rather than adaptive routing or deeper iteration. The large sample, reported p-values, and exclusive use of local models without proprietary APIs are clear strengths that support reproducibility and practical relevance.

major comments (1)

[Abstract] Abstract and final paragraph: the claim that 'simpler and fixed choices turn out to be competitive with or better than their adaptive versions' is load-bearing for the central interpretation, yet rests on a single rule-based routing heuristic whose failure mode (over-routing to BM25 on named entities common to nearly all sub-questions) is explicitly diagnosed. The result demonstrates only that this particular heuristic underperforms fixed RRF; it does not test whether a stronger learned router would reverse the ranking. This assumption is stated as the weakest in the evaluation and directly limits the broader conclusion.

minor comments (2)

[Abstract] The abstract states p<0.01 and p<0.001 for decomposition and reranking but does not name the statistical test or correction method; adding this detail would clarify the significance claims.
[Abstract] The 5,000-question subset is drawn from the HotpotQA distractor dev set; the paper should explicitly state the sampling procedure and confirm that the same questions are used across all ablation conditions to ensure comparability.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for highlighting the scope of our claims on adaptive routing. We address the major comment below and will revise the manuscript accordingly.

read point-by-point responses

Referee: [Abstract] Abstract and final paragraph: the claim that 'simpler and fixed choices turn out to be competitive with or better than their adaptive versions' is load-bearing for the central interpretation, yet rests on a single rule-based routing heuristic whose failure mode (over-routing to BM25 on named entities common to nearly all sub-questions) is explicitly diagnosed. The result demonstrates only that this particular heuristic underperforms fixed RRF; it does not test whether a stronger learned router would reverse the ranking. This assumption is stated as the weakest in the evaluation and directly limits the broader conclusion.

Authors: We agree that the load-bearing claim generalizes from the specific rule-based heuristic tested. The manuscript already diagnoses the over-routing failure mode on named entities. To align the interpretation with the evidence, we will revise the abstract and final paragraph to state that the tested rule-based adaptive routing underperforms fixed RRF, rather than referring to adaptive versions more broadly. This revision will be incorporated in the next version. Evaluating whether a learned router could reverse the result would require additional training data and compute outside the current ablation of off-the-shelf local-model components. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: purely empirical ablation measurements

full rationale

The paper reports results from controlled ablation experiments on a fixed agentic RAG pipeline evaluated on HotpotQA using a local 7B model. All reported quantities (EM, F1 scores, statistical significance) are direct empirical measurements from the runs. There are no derivations, equations, fitted parameters renamed as predictions, self-citations used as load-bearing uniqueness theorems, or ansatzes smuggled in. The central comparisons (fixed hybrid vs. rule-based adaptive routing, iteration depth, etc.) are data-driven observations, not reductions to inputs by construction. This is a standard empirical study with no circular steps.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

Empirical ablation study; claims rest on standard benchmark usage and model choice without additional fitted parameters or postulated entities.

assumptions (2)

domain assumption HotpotQA distractor development set is a valid proxy for multi-hop QA performance
Used as the sole evaluation benchmark for all conditions.
domain assumption Qwen2.5-7B-Instruct is representative of current local 7B-class models
Chosen as the fixed model for the entire ablation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Dissecting Agentic RAG: A Component Ablation for Multi-Hop QA with a Local 7B Model." pith.science (2026). https://pith.science/paper/LDVPAMZV

@misc{pith2026260621553,
  author       = {Pith},
  title        = {Pith review of: Dissecting Agentic RAG: A Component Ablation for Multi-Hop QA with a Local 7B Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LDVPAMZV}},
  note         = {Machine review of arXiv:2606.21553}
}

read the original abstract

Agentic retrieval-augmented generation (RAG) systems combine iterative reasoning loops, query decomposition, and adaptive retrieval to tackle multi-hop question answering. However, the contribution of each component remains poorly understood, particularly under resource-constrained settings using only local language models. Many agentic designs add adaptive retrieval routing and deeper retrieval loops on the assumption that the added complexity helps. To test whether it does, we run a controlled ablation study of a full agentic RAG pipeline evaluated on 5,000 questions from the HotpotQA distractor development set using a local 7B parameter model (Qwen2.5-7B-Instruct). Our full pipeline achieves EM=53.2% and F1=61.6%, compared to a single-pass dense-retrieval baseline of EM=43.1% and F1=54.0%. Across eight ablation conditions, we find that: (1) fixed hybrid retrieval via reciprocal rank fusion consistently outperforms rule-based adaptive routing (+1.8 EM, +1.9 F1), as the routing heuristic over-routes to BM25 by firing on named entities present in nearly all multi-hop sub-questions; (2) two retrieval iterations over the decomposed sub-questions capture 95% of the gains of five, with no meaningful benefit from deeper loops; and (3) query decomposition and cross-encoder reranking each contribute statistically significant but smaller gains (p<0.01 and p<0.001 respectively). Taken together, on a fixed local-model budget, the simpler and fixed choices turn out to be competitive with or better than their adaptive versions: most of the gain comes from running a short retrieval loop, not from adaptive routing or from many iterations. We use no proprietary APIs or large-scale compute.

Figures

Figures reproduced from arXiv: 2606.21553 by the authors.

**Figure 2.** EM and F1 for all ablation variants. Hybrid [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** EM and F1 as a function of retrieval loop depth [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Distribution of the strategy that produced [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Locating Failure in Multi-Page Visually Rich Document Understanding: An Empirical Attribution
cs.AI 2026-08 conditional novelty 6.0 of 10

A controlled attribution study of multi-page document understanding finds that missing evidence and cross-page integration failures dominate over distractor noise and text extraction quality.

Reference graph

Works this paper leans on

12 extracted references · 1 canonical work pages · cited by 1 Pith paper

[1]

and Salakhutdinov, Ruslan and Manning, Christopher D

Yang, Zhilin and Qi, Peng and Zhang, Saizheng and Bengio, Yoshua and Cohen, William W. and Salakhutdinov, Ruslan and Manning, Christopher D. , booktitle=. 2018 , url=

2018
[2]

Retrieval-Augmented Generation for Knowledge-Intensive

Lewis, Patrick and Perez, Ethan and Piktus, Aleksandra and Petroni, Fabio and Karpukhin, Vladimir and Goyal, Naman and K. Retrieval-Augmented Generation for Knowledge-Intensive. Advances in Neural Information Processing Systems , volume=. 2020 , url=

2020
[3]

2023 , url=

Yao, Shunyu and Zhao, Jeffrey and Yu, Dian and Du, Nan and Shafran, Izhak and Narasimhan, Karthik and Cao, Yuan , booktitle=. 2023 , url=

2023
[4]

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions , author=. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=. 2023 , url=

2023
[5]

Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=

Measuring and Narrowing the Compositionality Gap in Language Models , author=. Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=. 2023 , url=

2023
[6]

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

Active Retrieval Augmented Generation , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=. 2023 , url=

2023
[7]

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , pages=

Dense Passage Retrieval for Open-Domain Question Answering , author=. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , pages=. 2020 , url=

2020
[8]

2020 , doi=

Khattab, Omar and Zaharia, Matei , booktitle=. 2020 , doi=

2020

Show all 12 references

[9]

Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval , pages=

Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods , author=. Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval , pages=. 2009 , doi=

2009
[10]

The Probabilistic Relevance Framework:

Robertson, Stephen and Zaragoza, Hugo , journal=. The Probabilistic Relevance Framework:. 2009 , doi=

2009
[11]

2309.07597 , archivePrefix=

Xiao, Shitao and Liu, Zheng and Zhang, Peitian and Muennighoff, Niklas and Lian, Defu and Nie, Jian-Yun , year=. 2309.07597 , archivePrefix=

arXiv
[12]

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

Hierarchical Graph Network for Multi-hop Question Answering , author=. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=. 2020 , url=

2020

Pith tools

Reviewed June 26, 2026 · model on record in the stance chip above.

[1] [1]

and Salakhutdinov, Ruslan and Manning, Christopher D

Yang, Zhilin and Qi, Peng and Zhang, Saizheng and Bengio, Yoshua and Cohen, William W. and Salakhutdinov, Ruslan and Manning, Christopher D. , booktitle=. 2018 , url=

2018

[2] [2]

Retrieval-Augmented Generation for Knowledge-Intensive

Lewis, Patrick and Perez, Ethan and Piktus, Aleksandra and Petroni, Fabio and Karpukhin, Vladimir and Goyal, Naman and K. Retrieval-Augmented Generation for Knowledge-Intensive. Advances in Neural Information Processing Systems , volume=. 2020 , url=

2020

[3] [3]

2023 , url=

Yao, Shunyu and Zhao, Jeffrey and Yu, Dian and Du, Nan and Shafran, Izhak and Narasimhan, Karthik and Cao, Yuan , booktitle=. 2023 , url=

2023

[4] [4]

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions , author=. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=. 2023 , url=

2023

[5] [5]

Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=

Measuring and Narrowing the Compositionality Gap in Language Models , author=. Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=. 2023 , url=

2023

[6] [6]

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

Active Retrieval Augmented Generation , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=. 2023 , url=

2023

[7] [7]

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , pages=

Dense Passage Retrieval for Open-Domain Question Answering , author=. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , pages=. 2020 , url=

2020

[8] [8]

2020 , doi=

Khattab, Omar and Zaharia, Matei , booktitle=. 2020 , doi=

2020