REVIEW 2 major objections 4 minor 48 references
Question Decomposition for Retrieval-Augmented Generation
T0 review · 2 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Pairing an off-the-shelf cross-encoder reranker with LLM-driven question decomposition bridges the retrieval gap on multi-hop questions, without extra training or specialized indexing.
desk verdict Useful drop-in RAG configuration, honestly reported, but the decomposition-specific benefit is confounded by candidate-pool size. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is a two-stage retrieval pipeline built on top of standard dense retrieval. In the first stage, an instruction-tuned LLM with a fixed prompt generates up to five fact-seeking subquestions per original question; dense retrieval is run for each subquestion and for the original query, and the results are merged and deduplicated. In the second stage, a cross-encoder reranker scores every merged candidate against the original question and returns the top-k passages for generation. Question decomposition is the coverage-expanding stage, and reranking is the precision-restoring stage; the original query is always retained in the retrieval set to preserve baseline performance.
What would settle it
On a multi-hop dataset with gold evidence annotations, compare Hits@10 and MRR@10 of (i) single-query retrieval with reranking (retrieve 2k passages, rerank to k) and (ii) question decomposition plus reranking (retrieve k per subquery, merge, rerank to k). If run (i) matches or exceeds run (ii) on the same queries and passage budget, the claim that decomposition expands coverage is falsified, because the reranker alone would then be doing all the work.
Extended reading notes
Core claim
The central claim is that the retrieval gap in multi-hop RAG is bridgeable by pairing LLM-driven question decomposition with a cross-encoder reranker, without any extra training or specialized indexing. Decomposition expands the candidate pool so that each subquestion targets a separate document holding a needed fact; reranking then scores every merged candidate against the original query and keeps only the top-k, restoring precision. The paper's QD+RR system reaches 87.2% Hits@10 and 0.635 MRR@10 on MultiHop-RAG, beating the strongest reported baselines from the dataset paper, and it achieves the best answer EM (28.1) and F1 (35.0) on HotpotQA among naive RAG, QD-only, and reranker-only variants. The paper takes this as evidence that coverage from decomposition and precision from reranking are complementary, not redundant.
Load-bearing premise
The whole method depends on the assumption that a fixed prompt makes an instruction-tuned LLM produce fact-seeking subquestions whose independently retrieved passages, after reranking against the original question, retrieve the gold evidence better than the original query alone.
Editorial extensions
If this is right
- RAG systems operating on multi-hop or multi-entity questions can add this two-module pipeline as a drop-in enhancement, gaining retrieval recall and ranking quality without retraining or reindexing.
- Because both modules are off-the-shelf, the method is model-agnostic: any instruction-following LLM can generate subquestions, and any cross-encoder can rerank.
- The ablation shows that decomposition alone yields only small gains, so the reranker is the component that turns broader coverage into usable precision.
- On MultiHop-RAG, QD+RR outperforms the dataset authors' strongest reranked baselines even with a smaller embedding model, suggesting the method is not tied to a specific retriever.
- Because the LLM almost always emits exactly five subquestions, the effective behavior is a fixed subquery budget; changing that budget is a direct lever that trades latency against coverage.
Reading between the lines
- The near-zero correlation between subquery count and gold evidence count suggests the decomposition LLM is producing a diverse cover rather than counting hops; this predicts that maximizing subquery diversity, rather than number, would be the better objective for the decomposition prompt.
- The paper does not test adaptive decomposition, but its own data suggest a fixed five-subquery budget is wasteful on HotpotQA, where most questions need only two supporting facts; a gating rule that decomposes only when the query is predicted to be multi-hop could retain most of the gain at a fraction of the latency.
- Caching subqueries per raw query, as the paper briefly proposes, makes the amortized latency of QD+RR close to plain RAG, turning the primary overhead into a one-time storage cost.
- The method's gains are demonstrated on English Wikipedia-style corpora; a natural test is whether the same pipeline holds when the reranker is mismatched with the domain, a failure mode the paper itself flags as a limitation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a retrieval-augmented generation pipeline that adds an LLM-based question decomposition (QD) module and an off-the-shelf cross-encoder reranker (RR) to standard dense retrieval. For a multi-hop query, the LLM generates up to five subqueries; top-k passages are retrieved for the original query and each subquery, merged, deduplicated, and reranked against the original query before the top-k passages are passed to the generator. The authors evaluate four system variants (Naive RAG, +QD, +RR, +QD+RR) on MultiHop-RAG and HotpotQA, reporting that QD+RR achieves the highest Hits@10 (87.2%) and MRR@10 (0.635) on MultiHop-RAG and the highest answer EM/F1 on HotpotQA. The paper also reports latency overhead, subquery-count distributions, and a correlation analysis showing no relation between subquery count and gold evidence count.
Significance. If the reported retrieval gains survive a properly controlled comparison, the paper offers a practical and modular enhancement for multi-hop RAG: it requires no fine-tuning, uses only an instruction-tuned LLM for decomposition and a standard reranker, and the authors release their code. The four-way internal ablation is a useful design choice, and using exact FAISS inner-product search removes approximate-search confounds. The central scientific claim, that decomposition-specific subqueries assemble complementary evidence that reranking then distills, is plausible but is not yet cleanly established because the main ablation does not hold the candidate-pool size fixed, and the HotpotQA supporting-fact metrics partially contradict the evidence-selection narrative. The paper is an incremental but potentially useful empirical contribution if these issues are resolved.
major comments (2)
- [Section 5.1, Table 1; Section 4.2] The headline comparison between +RR and +QD+RR does not control for candidate-pool size. As defined in Section 4.2, +RR retrieves top-2k = 20 passages for the original query and reranks them, while +QD+RR merges top-10 passages for the original query plus top-10 passages for each subquery. Table 3 shows that the LLM emits exactly five subqueries in 93.3% of MultiHop-RAG cases and 98.6% of HotpotQA cases, so the QD+RR pool is typically 60 candidates, three times larger than the RR pool. A reranker over a larger candidate pool can improve Hits@10 and MRR@10 even if the additional candidates are generic paraphrases of the original query, so the specific contribution of decomposition-generated subqueries is not isolated. Please add a controlled condition, for example reranking top-60 passages retrieved with the original query alone, or reducing the per-subquery cutoff so the merged pool matches 20 candidates, and report significance or bootstrap intervals across multiple seeds.
- [Section 5.2, Table 2] The claim that QD+RR leads to better evidence-grounded answers is weakened by the supporting-fact and joint metrics on HotpotQA. QD+RR has lower supporting-fact EM and F1 than RR alone (17.9/11.2 vs. 19.6/12.9) and lower joint EM and F1 (8.9/5.8 vs. 9.5/6.4), despite having higher supporting-fact precision (46.8 vs. 44.9). Since the paper's narrative is that decomposition improves coverage and reranking restores precision, the evidence-identification outcome on HotpotQA does not consistently support that narrative. The paper should explicitly discuss this pattern, and should report supporting-fact recall, which is listed in Section 4.3 but omitted from Table 2; without recall it is impossible to tell whether the higher precision comes at the cost of coverage.
minor comments (4)
- [Section 5.1] The comparison against text-ada-002 and voyage-02 baselines from Tang and Yang (2024) uses different embedding models and retrieval settings, so the sentence 'our QD+RR thus improves Hits@10 by 16.5% and MRR@10 by 8.4%' is a cross-paper comparison rather than a controlled head-to-head gain; please label it accordingly.
- [Algorithm 1] Algorithm 1 calls DECOMPOSE(q0) while Section 3.1 defines DECOMPOSE(q, p); the argument q0 is not defined. Please align the pseudocode with the notation in the text.
- [Section 5.4] The claim that 'once decomposed, subqueries can be reused ... so that the latency remains identical to the baseline' depends on an unspecified cache-population assumption; on a cold cache the reported 18.9s/query latency applies, and this tradeoff should be stated more carefully.
- [Table 5] Table 5 labels columns 'Pearson (p)' and 'Spearman (p)' but the values shown appear to be the correlation coefficient with a p-value in parentheses; please rename the columns to avoid ambiguity.
Circularity Check
No significant circularity: the QD+RR pipeline is an empirical system benchmarked on external datasets, with no fitted parameter renamed as a prediction and only minor non-load-bearing self-citations.
full rationale
The paper's central claim is that combining question decomposition with an off-the-shelf reranker improves multi-hop retrieval and answer accuracy. This claim is supported by experiments on MultiHop-RAG and HotpotQA, using fixed public models (bge-large-en-v1.5, bge-reranker-large, Qwen2.5-32B-Instruct) and fixed hyperparameters (k=10, temperature 0.8). No parameter of the pipeline is fitted to the target benchmark labels, and no evaluation metric is defined in terms of the method's own outputs. The retrieval equations R(q) and the reranking score g_phi(q,d) are standard definitions from prior work, not derived from the reported Hits@10 or MRR@10 numbers. The QD module is an LLM call with a fixed prompt; the paper explicitly reports that the LLM almost always emits the maximum of five subqueries, which is a behavioral observation, not a fitted parameter. The self-citations to Golde et al. (2023, 2024) appear only in related work as contextual examples of LLM-based data generation and NER limitations; they are not load-bearing for the main retrieval result, and no uniqueness theorem or prior claim is invoked to forbid alternative approaches. The reviewer-flagged confound that QD+RR reranks up to roughly 60 candidates while RR reranks only 20 is an experimental-design concern about isolating the causal effect of decomposition, but it is not a case where the paper's prediction reduces by construction to its inputs. Consequently, no circular step meets the required evidence standard, and the modest self-citation presence does not affect the validity of the empirical evaluation.
Assumptions & free parameters
free parameters (3)
- k (number of retrieved/reranked passages) =
10
- Subquery budget (LLM prompt cap) =
5
- Sampling temperature and top-p for LLM calls =
0.8 and 0.8
assumptions (5)
- domain assumption bge-large-en-v1.5 dense embeddings are an adequate base retriever for both original queries and subqueries.
- domain assumption bge-reranker-large cross-encoder reliably scores candidate passages by relevance to the original query.
- domain assumption An instruction-tuned LLM can decompose questions into useful fact-seeking subquestions without task-specific training.
- domain assumption Gold evidence annotations in MultiHop-RAG and HotpotQA are complete and correctly define retrieval success.
- domain assumption Dev-split evaluation is representative of hidden test behavior for these benchmarks.
Cite this review
Pith. "Pith review of Question Decomposition for Retrieval-Augmented Generation." pith.science (2026). https://pith.science/paper/KTRUZX56
@misc{pith2026250700355,
author = {Pith},
title = {Pith review of: Question Decomposition for Retrieval-Augmented Generation},
year = {2026},
howpublished = {\url{https://pith.science/paper/KTRUZX56}},
note = {Machine review of arXiv:2507.00355}
}
read the original abstract
Grounding large language models (LLMs) in verifiable external sources is a well-established strategy for generating reliable answers. Retrieval-augmented generation (RAG) is one such approach, particularly effective for tasks like question answering: it retrieves passages that are semantically related to the question and then conditions the model on this evidence. However, multi-hop questions, such as "Which company among NVIDIA, Apple, and Google made the biggest profit in 2023?," challenge RAG because relevant facts are often distributed across multiple documents rather than co-occurring in one source, making it difficult for standard RAG to retrieve sufficient information. To address this, we propose a RAG pipeline that incorporates question decomposition: (i) an LLM decomposes the original query into sub-questions, (ii) passages are retrieved for each sub-question, and (iii) the merged candidate pool is reranked to improve the coverage and precision of the retrieved evidence. We show that question decomposition effectively assembles complementary documents, while reranking reduces noise and promotes the most relevant passages before answer generation. Although reranking itself is standard, we show that pairing an off-the-shelf cross-encoder reranker with LLM-driven question decomposition bridges the retrieval gap on multi-hop questions and provides a practical, drop-in enhancement, without any extra training or specialized indexing. We evaluate our approach on the MultiHop-RAG and HotpotQA, showing gains in retrieval (MRR@10: +36.7%) and answer accuracy (F1: +11.6%) over standard RAG baselines.
Figures
Reference graph
Works this paper leans on
-
[1]
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. https://doi.org/10.48550/arXiv.2310.11511 Self- RAG : Learning to Retrieve , Generate , and Critique through Self-Reflection . Preprint, arXiv:2310.11511
-
[2]
Chi-Min Chan, Chunpu Xu, Ruibin Yuan, Hongyin Luo, Wei Xue, Yike Guo, and Jie Fu. 2024. https://doi.org/10.48550/arXiv.2404.00610 RQ-RAG : Learning to Refine Queries for Retrieval Augmented Generation . Preprint, arXiv:2404.00610
-
[3]
Sukmin Cho, Jeongyeon Seo, Soyeong Jeong, and Jong C. Park. 2023. https://arxiv.org/abs/2310.17490 Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering . Preprint, arXiv:2310.17490
work page Pith review arXiv 2023
-
[4]
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazar \'e , Maria Lomeli, Lucas Hosseini, and Herv \'e J \'e gou. 2024. https://doi.org/10.48550/arXiv.2401.08281 The Faiss library . Preprint, arXiv:2401.08281
-
[5]
Facebookresearch. 2024. Faiss indexes. https://github.com/facebookresearch/faiss/wiki/ Faiss-indexes
work page 2024
-
[6]
Barah Fazili, Koustava Goswami, Natwar Modani, and Inderjeet Nair. 2024. https://doi.org/10.48550/arXiv.2407.10245 GenSco : Can Question Decomposition based Passage Alignment improve Question Answering ? Preprint, arXiv:2407.10245
work page Pith review arXiv doi:10.48550/arxiv.2407.10245 2024
-
[7]
Yair Feldman and Ran El-Yaniv . 2019. https://doi.org/10.48550/arXiv.1906.06606 Multi- Hop Paragraph Retrieval for Open-Domain Question Answering . Preprint, arXiv:1906.06606
work page Pith review arXiv doi:10.48550/arxiv.1906.06606 2019
-
[8]
Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2022. https://doi.org/10.48550/arXiv.2212.10496 Precise Zero-Shot Dense Retrieval without Relevance Labels . Preprint, arXiv:2212.10496
Show all 48 references
-
[9]
Michael Glass, Gaetano Rossiello, Md Faisal Mahbub Chowdhury, Ankita Naik, Pengshan Cai, and Alfio Gliozzo. 2022. https://doi.org/10.18653/v1/2022.naacl-main.194 Re2G : Retrieve , Rerank , Generate . In Proceedings of the 2022 Conference of the North American Chapter of the As...
2022 doi
-
[10]
Jonas Golde, Patrick Haller, Felix Hamborg, Julian Risch, and Alan Akbik. 2023. https://doi.org/10.18653/v1/2023.emnlp-demo.1 Fabricator: An open source toolkit for generating labeled training data with teacher LLM s . In Proceedings of the 2023 Conference on Empirical Methods...
2023 doi
-
[11]
Jonas Golde, Felix Hamborg, and Alan Akbik. 2024. https://aclanthology.org/2024.eacl-long.178/ Large-scale label interpretation learning for few-shot named entity recognition . In Proceedings of the 18th Conference of the European Chapter of the Association for Computational L...
2024
- [12]
- [13]
-
[14]
Vladimir Karpukhin, Barlas O g uz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. https://arxiv.org/abs/2004.04906 Dense Passage Retrieval for Open-Domain Question Answering . Preprint, arXiv:2004.04906
2020 arXiv
-
[15]
LangChain. 2025. LangChain . https://www.langchain.com/
2025
-
[16]
u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, Sebastian Riedel, and Douwe Kiela. 2020. https://arxiv.org/abs/2005.11401 Retrieval- Augmented Generation for Know...
2020 arXiv
-
[17]
Kunze Li and Yu Zhang. 2024. https://doi.org/10.18653/v1/2024.findings-acl.280 Planning first, question second: An LLM -guided method for controllable question generation . In Findings of the Association for Computational Linguistics: ACL 2024, pages 4715--4729, Bangkok, Thail...
2024 doi
-
[18]
Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023. https://arxiv.org/abs/2305.14283 Query Rewriting for Retrieval-Augmented Large Language Models . Preprint, arXiv:2305.14283
2023 arXiv
- [19]
- [20]
-
[21]
Rodrigo Nogueira and Kyunghyun Cho. 2020. https://arxiv.org/abs/1901.04085 Passage Re-ranking with BERT . Technical Report arXiv:1901.04085, arXiv
2020 arXiv
-
[22]
OpenAI . 2022. New and improved embedding model. https://openai.com/index/new-and-improved-embedding-model/
2022
- [23]
- [24]
-
[25]
Qwen Team . 2024. Qwen2.5: A Party of Foundation Models ! https://qwenlm.github.io/blog/qwen2.5/
2024
- [26]
-
[27]
Nils Reimers and Iryna Gurevych. 2019. https://arxiv.org/abs/1908.10084 Sentence- BERT : Sentence Embeddings using Siamese BERT-Networks . Technical Report arXiv:1908.10084, arXiv
2019 arXiv
-
[28]
Chaudhary, Kalpa Gunaratna, Srinivasan Parthasarathy, and Manas Gaur
Yash Saxena, Ankur Padia, Mandar S. Chaudhary, Kalpa Gunaratna, Srinivasan Parthasarathy, and Manas Gaur. 2025. https://doi.org/10.48550/arXiv.2505.16014 Ranking Free RAG : Replacing Re-ranking with Selection in RAG for Sensitive Domains . Preprint, arXiv:2505.16014
- [29]
- [30]
- [31]
- [32]
-
[33]
Voyage AI Innovations Inc. 2024. Voyage AI Home . https://www.voyageai.com/
2024
-
[34]
Shuting Wang, Xin Yu, Mang Wang, Weipeng Chen, Yutao Zhu, and Zhicheng Dou. 2024 a . https://arxiv.org/abs/2406.12566 RichRAG : Crafting Rich Responses for Multi-faceted Queries in Retrieval-Augmented Generation . Preprint, arXiv:2406.12566
2024 arXiv
-
[35]
Xiaohua Wang, Zhenghua Wang, Xuan Gao, Feiran Zhang, Yixin Wu, Zhibo Xu, Tianyuan Shi, Zhengyuan Wang, Shizheng Li, Qi Qian, Ruicheng Yin, Changze Lv, Xiaoqing Zheng, and Xuanjing Huang. 2024 b . https://doi.org/10.18653/v1/2024.emnlp-main.981 Searching for Best Practices in R...
2024 doi
- [36]
- [37]
- [38]
- [39]
-
[40]
Kota Yamaguchi. 2025. Faiss-cpu: A library for efficient similarity search and clustering of dense vectors
2025
-
[41]
Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling. 2024. https://arxiv.org/abs/2401.15884 Corrective Retrieval Augmented Generation . Technical Report arXiv:2401.15884, arXiv
2024 arXiv
- [42]
-
[43]
Cohen, Ruslan Salakhutdinov, and Christopher D
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. https://arxiv.org/abs/1809.09600 HotpotQA : A Dataset for Diverse , Explainable Multi-hop Question Answering . Preprint, arXiv:1809.09600
2018 arXiv
- [44]
- [45]
-
[46]
Rongzhi Zhu, Xiangyu Liu, Zequn Sun, Yiwei Wang, and Wei Hu. 2025. https://arxiv.org/abs/2502.14245 Mitigating lost-in-retrieval problems in retrieval augmented multi-hop question answering . Preprint, arXiv:2502.14245
2025 arXiv
-
[47]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[48]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.