REVIEW 4 major objections 5 minor 41 references
Exp4Fuse: A Rank Fusion Framework for Enhanced Sparse Retrieval using Large Language Model-based Query Expansion
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Exp4Fuse claims that fusing a sparse retriever's ranking of the original query with its ranking of a zero-shot LLM-augmented query consistently improves retrieval and reaches state-of-the-art results on TREC DL 2019 when paired with…
desk verdict A practical fusion recipe for sparse retrieval that works, but the paper's headline claim against query2doc/LameR is not yet demonstrated because those baselines are never run on the same sparse backbones. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is a two-route retrieval plus a modified reciprocal rank fusion score: $FR_{score} = (w_i + \frac{n}{10}) \sum_{i=1}^{2} \frac{1}{k + r_i}$, with $k=60$ fixed from a pilot study, $w_i=1$ for both routes, and $n \in \{1,2\}$ the number of lists containing the document. The $\frac{n}{10}$ bonus preferentially promotes documents that both the original query and the LLM-augmented query retrieve, while the $k$ term dampens rank outliers. A second mechanism is the query-balancing weight $\lambda$: repeating the original query $\lambda$ times before concatenating the LLM-generated passage prevents the long hypothetical document from drowning out the user's terms in a sparse lexical match.
What would settle it
Run Exp4Fuse on new, unseen retrieval datasets with $k=60$ and $\lambda=5$ frozen exactly as reported, and compare the fused ranking against the better of the two individual route lists; if the fused ranking does not consistently beat that single best list, the central fusion claim fails.
Extended reading notes
Core claim
Exp4Fuse establishes that the value of zero-shot LLM query expansion for sparse retrieval is realised only when the expanded query is used in parallel with the original query, not as a replacement. The framework generates two ranked lists from one sparse retriever—one for the original query, one for the original query repeated λ times and concatenated with an LLM-generated hypothetical passage—and fuses them with an adaptive reciprocal rank score that up-weights documents appearing in both lists. On MS MARCO dev, TREC DL 2019/2020, and seven BEIR datasets, fusing improves every sparse retriever tested, from BM25 to SPLADE++-v2, across nearly all metrics; SPLADE++-v1 + Exp4Fuse sets state-of-the-art numbers on TREC DL 2019. The paper also shows the fusion stage is necessary: using the LLM-augmented query alone often hurts learned sparse retrievers, whereas fusing recovers and exceeds the original-query baseline.
Load-bearing premise
The reported gains assume the fusion constants $k=60$ and $\lambda=5$, and the zero-shot prompts, were chosen without tuning on the evaluation test sets; if any of these were selected using MS MARCO dev or TREC DL labels, the improvements would be inflated.
Editorial extensions
If this is right
- Any sparse retriever, from BM25 to learned SPLADE variants, can be improved by fusing original and LLM-expanded result lists, with no retraining.
- Direct zero-shot LLM expansion alone is unreliable for learned sparse retrievers; the fusion ranking stage is what converts LLM output into consistent gains.
- Exp4Fuse with SPLADE++-v1/v2 matches or beats strong dense retrievers and multi-stage retrieval pipelines on several benchmarks, at lower computational cost.
- The benefit extends out-of-domain to low-resource BEIR datasets, with the largest gains on short, ambiguous queries like Touche2020 and NQ.
- Performance grows with additional LLM-based routes up to three, then plateaus or degrades, so two routes (original + hypothetical document) are the cost-effective choice.
Reading between the lines
- The $\frac{n}{10}$ bonus in the fusion formula acts as an implicit consensus prior; a testable extension is to let $n$ carry fractional credit so documents present in both lists are treated as more reliable without over-penalising unique relevant documents.
- Because the framework is retriever-agnostic, it should transfer to any dense or hybrid retriever that produces ranked lists; fusing original-query dense rankings with LLM-augmented dense rankings could show similar or larger gains, assuming the same failure mode of direct expansion.
- The results suggest that LLM hallucination may be less harmful to retrieval when hypothetical documents are used as a second opinion rather than as the query itself; a controlled experiment varying LLM accuracy could test this.
- The paper fixes $k=60$ and $\lambda=5$; a systematic sweep on held-out data could yield larger gains than the reported fixed values, but the fixed values' success across datasets hints at robustness.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes Exp4Fuse, a rank fusion framework that aims to improve sparse retrieval by combining two retrieval routes built on the same sparse retriever: one using the original query and one using a zero-shot LLM-generated hypothetical document appended to the original query. The two top-1000 ranked lists are fused using a modified reciprocal rank fusion (RRF) formula, which adds an occurrence-count term and per-list weights to the standard RRF score. The method is evaluated on MS MARCO dev, TREC DL 2019/2020, and seven BEIR datasets, using GPT-4-mini as the default LLM and LLaMA3-8B-Instruct for a generalizability check. The reported results show consistent improvements of the fused output over the original-query baseline across all tested sparse retrievers and datasets, with the largest gains on TREC DL 2019 and low-resource BEIR datasets. The paper also includes ablations on the number of route retrievals and on the necessity of individual routes.
Significance. If the claimed effect is real, Exp4Fuse offers a simple, low-cost way to leverage zero-shot LLM-based query expansion for sparse and learned-sparse retrievers, avoiding the heavy memory and latency costs of dense retriever pipelines. The strengths of the paper include a clear and reproducible framework, released code, use of public checkpoints for several baselines, consistent positive results across ten benchmarks, and a generalizability experiment with an open-source LLM. However, the central comparative claim—that Exp4Fuse 'surpasses existing LLM-based query expansion methods in enhancing sparse retrievers'—is not directly demonstrated, because the existing methods are not evaluated on the same sparse backbones. The absence of a standard-RRF control also leaves open whether the modified fusion formula, rather than fusion itself, is responsible for the gains.
major comments (4)
- [Section 4.2, Tables 1 and 2] The abstract and Section 1 claim that Exp4Fuse 'surpasses existing LLM-based query expansion methods in enhancing sparse retrievers,' and Section 4.1 states that query2doc and LameR are compared 'under identical experimental conditions.' Yet Tables 1 and 2 contain no sparse-retriever rows for BM25+query2doc, BM25+LameR, SPLADE++-v1+query2doc, or SPLADE++-v1+LameR. The only query2doc/LameR rows use SimLM, a dense retriever. Consequently, the observed advantage of, e.g., SPLADE++-v1+Exp4Fuse over SimLM+query2doc is confounded by retriever strength rather than by the fusion framework, leaving the headline comparative claim unverified. Please add same-backbone runs for query2doc and LameR on at least BM25 and SPLADE++-v1/v2.
- [Section 3.2, Eq. (2), and Table 4] The paper never compares the modified reciprocal rank fusion formula against the standard RRF of Cormack et al. [6] or against any other fusion method. This is a load-bearing omission because Table 4 shows that the LLM-expanded route alone degrades learned sparse retrievers substantially (e.g., SPLADE++-v1 DL19 nDCG@10 drops from 73.1 to 67.8), so the gain to 77.6 comes from the fusion rule itself. Without a standard-RRF baseline, the contribution of the 'adaptive weight strategy' (the n/10 term and the wi weights) is not established. Please include standard RRF on the same two lists, and ideally one additional fusion baseline such as CombSUM or Borda count, as a control.
- [Sections 3.2 and 4.1] The key hyperparameters—k=60 in Eq. (2), λ=5 in Eq. (1), and w1=w2=1—are described as fixed during a pilot study or set empirically, but the pilot study data, tuning procedure, and any sensitivity analysis are not reported. If k or λ were selected using the evaluation benchmarks (MS MARCO dev, TREC DL 2019/2020), the reported improvements would be partly circular. Since these constants directly affect the fusion scores and the augmented queries, please provide a sensitivity analysis on a validation split that is independent of the test sets, or otherwise justify that the values were not chosen on the evaluation data.
- [Tables 1-4] All experiments appear to be single runs with no error bars, per-topic variance, or significance tests. Several reported improvements are small—for example, SLIM++ +Exp4Fuse gains only +0.4 nDCG@10 on DL20 and +0.1 R@1k on DL19, and SPLADE++-v1 +Exp4Fuse gains +0.7 R@1k on MS MARCO dev—yet Section 6 concludes that Exp4Fuse 'significantly improves' performance. Please report standard deviations across the query set or paired significance tests (e.g., two-tailed paired t-test or bootstrap) for the principal comparisons.
minor comments (5)
- [Section 5, Generalizability] The text says the LLaMA3-8B-Instruct results are 'presented in Table 4,' but the table is captioned 'Table 3: Results for Exp4Fuse using LLaMA3-8B-Instruct.' The table numbering is inconsistent and should be corrected.
- [Tables 1 and 3] Metric names are formatted inconsistently: Table 1 uses 'M AP' and 'R@1k,' while Table 3 uses 'R@1K'; the model names 'SPLADE_v1' and 'SPLADE_v2' in Table 3 should match the 'SPLADE++-v1' and 'SPLADE++-v2' used in Table 1.
- [Section 4.1, Implementation Details] The sentence 'Exp4Fuse uses λ = 5for LLM-based query expansion' contains a missing space before 'for,' and later 'nDCG @10' should be 'nDCG@10' for consistency.
- [Equation (2)] The summation notation '2X' is not standard; it should be written as \(\sum_{i=1}^{2}\). Also, the sentence defining \(r_i\) should clarify that \(i=1\) corresponds to \(I_{oq}\) and \(i=2\) to \(I_{eq}\), which is only stated in the surrounding text.
- [Appendix A.1] The dataset name 'Robsut04' is a typo for 'Robust04,' and the inconsistent capitalization of 'Touche2020' versus 'Touche 2020' should be harmonized.
Circularity Check
No significant circularity: Exp4Fuse's gains are empirical and not defined into existence.
full rationale
The paper's claimed derivation chain is entirely empirical: Exp4Fuse's fusion score (Eq. 2) is a heuristic combination of two ranked lists, and the reported gains (e.g., Table 1 and Table 4) are measured, not derived. No equation in the paper defines the output in terms of the input: the fusion formula does not contain the evaluation metric, and the LLM-expanded route is not a fitted surrogate for the final ranking. The hyperparameters k=60 ('fixed during a pilot study') and λ=5 are stated but not shown to have been tuned on the evaluation datasets; without evidence tying those choices to the reported test sets, this is a transparency concern, not a demonstrated circularity. The paper also does not rely on any self-citation: all cited baselines and techniques (RRF [6], query2doc [31], LameR [28], SPLADE [9]) are external prior work. The absence of same-backbone query2doc/LameR rows in Tables 1-2 weakens the comparative claim but is a controlled-comparison flaw, not a reduction of the result to its inputs. Therefore no circular step is exhibited.
Assumptions & free parameters
free parameters (4)
- λ (query repetition weight) =
5
- k (RRF constant) =
60
- w1, w2 (fusion weights) =
1, 1
- n/10 coefficient in FRscore =
0.1
assumptions (3)
- domain assumption The LLM-generated hypothetical document, while weak alone, contains complementary relevant terms that improve recall when fused with the original query.
- ad hoc to paper The modified reciprocal rank fusion formula is at least as effective as standard RRF or other fusion methods.
- ad hoc to paper Hyperparameters were selected without using the evaluation test sets.
Cite this review
Pith. "Pith review of Exp4Fuse: A Rank Fusion Framework for Enhanced Sparse Retrieval using Large Language Model-based Query Expansion." pith.science (2026). https://pith.science/paper/RYXQ4Q32
@misc{pith2026250604760,
author = {Pith},
title = {Pith review of: Exp4Fuse: A Rank Fusion Framework for Enhanced Sparse Retrieval using Large Language Model-based Query Expansion},
year = {2026},
howpublished = {\url{https://pith.science/paper/RYXQ4Q32}},
note = {Machine review of arXiv:2506.04760}
}
read the original abstract
Large Language Models (LLMs) have shown potential in generating hypothetical documents for query expansion, thereby enhancing information retrieval performance. However, the efficacy of this method is highly dependent on the quality of the generated documents, which often requires complex prompt strategies and the integration of advanced dense retrieval techniques. This can be both costly and computationally intensive. To mitigate these limitations, we explore the use of zero-shot LLM-based query expansion to improve sparse retrieval, particularly for learned sparse retrievers. We introduce a novel fusion ranking framework, Exp4Fuse, which enhances the performance of sparse retrievers through an indirect application of zero-shot LLM-based query expansion. Exp4Fuse operates by simultaneously considering two retrieval routes-one based on the original query and the other on the LLM-augmented query. It then generates two ranked lists using a sparse retriever and fuses them using a modified reciprocal rank fusion method. We conduct extensive evaluations of Exp4Fuse against leading LLM-based query expansion methods and advanced retrieval techniques on three MS MARCO-related datasets and seven low-resource datasets. Experimental results reveal that Exp4Fuse not only surpasses existing LLM-based query expansion methods in enhancing sparse retrievers but also, when combined with advanced sparse retrievers, achieves SOTA results on several benchmarks. This highlights the superior performance and effectiveness of Exp4Fuse in improving query expansion for sparse retrieval.
Figures
Reference graph
Works this paper leans on
-
[6]
Gordon V Cormack, Charles LA Clarke, and Stefan Buettcher. 2009. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, pages 758–759
work page 2009
-
[1]
Nasreen Abdul-Jaleel, James Allan, W Bruce Croft, Fernando Diaz, Leah Larkey, Xiaoyan Li, Mark D Smucker, and Courtney Wade. 2004. Umass at trec 2004: Novelty and hard. Computer Science Department Faculty Publication Series, page 189
work page 2004
-
[2]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774
arXiv 2023
-
[3]
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, An- drew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268
arXiv 2016
-
[4]
Nicholas J. Belkin, Paul Kantor, Edward A. Fox, and Joseph A Shaw. 1995. Combining the evidence of multiple query representations for information retrieval. Information Processing & Management, 31(3):431–448
work page 1995
-
[5]
Tao Chen, Mingyang Zhang, Jing Lu, Michael Bendersky, and Marc Najork. 2022. Out-of-domain semantics to the rescue! zero-shot hybrid retrieval models. In European Conference on Information Retrieval, pages 95–110. Springer
work page 2022
-
[7]
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. 2021. Overview of the TREC 2020 deep learning track. CoRR, abs/2102.07662
arXiv 2021
-
[8]
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M V oorhees. 2020. Overview of the trec 2019 deep learning track. arXiv preprint arXiv:2003.07820
arXiv 2020
Show all 41 references
-
[9]
Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. 2022. From distillation to hard negative sampling: Making sparse neural ir models more effective. In Proceedings of the 45th international ACM SIGIR conference on research and development in informa...
2022
-
[10]
Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021. Splade: Sparse lexical and expansion model for first stage ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2288–2292
2021
-
[11]
Luyu Gao and Jamie Callan. 2021. Unsupervised corpus aware language model pre-training for dense passage retrieval. arXiv preprint arXiv:2108.05540
2021 arXiv
-
[12]
Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2022. Precise zero-shot dense retrieval without relevance labels. arXiv preprint arXiv:2212.10496
2022 arXiv
-
[13]
Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. 2020. Improving efficient neural ranking models with cross-architecture knowledge distillation. arXiv preprint arXiv:2010.02666
2020 arXiv
-
[14]
Sebastian Hofstätter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin, and Allan Hanbury. 2021. Efficiently teaching an effective dense retriever with balanced topic aware sampling. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Inform...
2021
-
[15]
Rolf Jagerman, Honglei Zhuang, Zhen Qin, Xuanhui Wang, and Michael Bendersky. 2023. Query expansion by prompting large language models. Preprint, arXiv:2305.03653
2023 arXiv
-
[16]
Minghan Li, Sheng-Chieh Lin, Xueguang Ma, and Jimmy Lin. 2023. Slim: Sparsified late interaction for multi-vector retrieval with inverted indexes. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1954–1959. 9
2023
-
[17]
Jimmy Lin and Xueguang Ma. 2021. A few brief notes on deepimpact, coil, and a conceptual framework for information retrieval techniques. arXiv preprint arXiv:2106.14807
2021 arXiv
-
[18]
Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021. Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th International ACM SIGIR Conference ...
2021
-
[19]
Jing Lu, Gustavo Hernandez Abrego, Ji Ma, Jianmo Ni, and Yinfei Yang. 2021. Multi-stage training with improved negative contrast for neural passage retrieval. In Proceedings of the 2021 conference on empirical methods in natural language processing, pages 6091–6103
2021
-
[20]
Ji Ma, Ivan Korotkov, Keith B Hall, and Ryan T McDonald. 2020. Hybrid first-stage retrieval models for biomedical literature. In CLEF (Working Notes)
2020
-
[21]
Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. 2024. Fine-tuning llama for multi-stage text retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2421–2425
2024
-
[22]
Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. 2020. Document ranking with a pretrained sequence-to- sequence model. arXiv preprint arXiv:2003.06713
2020 arXiv
-
[23]
Rodrigo Nogueira, Jimmy Lin, and AI Epistemic. 2019a. From doc2query to doctttttquery. Online preprint, 6(2)
-
[24]
Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. 2019b. Multi-stage document ranking with bert. arXiv preprint arXiv:1910.14424
1910 arXiv
-
[25]
Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019c. Document expansion by query prediction. arXiv preprint arXiv:1904.08375
1904 arXiv
-
[26]
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2020. Rocketqa: An optimized training approach to dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2010.08191
2020 arXiv
-
[27]
Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389
2009
-
[28]
Tao Shen, Guodong Long, Xiubo Geng, Chongyang Tao, Yibin Lei, Tianyi Zhou, Michael Blumenstein, and Daxin Jiang. 2024. Retrieval-augmented retrieval: Large language models are strong zero-shot retriever. In Findings of the Association for Computational Linguistics ACL 2024, pa...
2024
-
[29]
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. Beir: A het- erogenous benchmark for zero-shot evaluation of information retrieval models. arXiv preprint arXiv:2104.08663
2021 arXiv
-
[30]
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Simlm: Pre-training with representation bottleneck for dense passage retrieval. arXiv preprint arXiv:2207.02578
2022 arXiv
-
[31]
Liang Wang, Nan Yang, and Furu Wei. 2023. Query2doc: Query expansion with large language models. arXiv preprint arXiv:2303.07678
2023 arXiv
-
[32]
Qian Yang, Qian Chen, Wen Wang, Baotian Hu, and Min Zhang. 2023a. Enhancing multi-modal multi-hop question answering via structured knowledge and unified retrieval-generation. In Proceedings of the 31st ACM International Conference on Multimedia, pages 5223–5234
-
[33]
Yingrui Yang, Shanxiu He, Yifan Qiao, Wentai Xie, and Tao Yang. 2023b. Balanced knowledge distillation with contrastive learning for document re-ranking. In Proceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval, pages 247–255
2023
-
[34]
Yingrui Yang, Yifan Qiao, and Tao Yang. 2022. Compact token representations with contextual quantization for efficient document re-ranking. arXiv preprint arXiv:2203.15328
2022 arXiv
-
[35]
Chunyuan Yuan, Wei Zhou, Mingming Li, Shangwen Lv, Fuqing Zhu, Jizhong Han, and Songlin Hu. 2019. Multi-hop selector network for multi-turn response selection in retrieval-based chatbots. In Proceedings of the 2019 conference on empirical methods in natural language processing...
2019
-
[36]
Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Min Zhang, and Shaoping Ma. 2020. Repbert: Contextualized text embeddings for first-stage retrieval. arXiv preprint arXiv:2006.15498
2020 arXiv
-
[37]
Le Zhang, Yihong Wu, Qian Yang, and Jian-Yun Nie. 2024. Exploring the best practices of query expansion with large language models. Preprint, arXiv:2401.06311. 10
2024 arXiv
-
[38]
Yizhe Zhang, Siqi Sun, Xiang Gao, Yuwei Fang, Chris Brockett, Michel Galley, Jianfeng Gao, and Bill Dolan
-
[39]
Zihuai Zhao, Wenqi Fan, Jiatong Li, Yunqing Liu, Xiaowei Mei, Yiqi Wang, Zhen Wen, Fei Wang, Xiangyu Zhao, Jiliang Tang, et al. 2023. Recommender systems in the era of large language models (llms). arXiv preprint arXiv:2307.02046
2023 arXiv
-
[40]
Please write a passage to answer the question. [question_text]
Huaixiu Steven Zheng, Swaroop Mishra, Xinyun Chen, Heng-Tze Cheng, Ed H Chi, Quoc V Le, and Denny Zhou. 2023. Take a step back: Evoking reasoning via abstraction in large language models. arXiv preprint arXiv:2310.06117. A Appendix A.1 Instructions TREC DL19 Instruction messag...
2023 arXiv
-
[2022]
In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11739–11747
Retgen: A joint framework for retrieval and grounded text generation modeling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11739–11747
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.