Pith. sign in

REVIEW 4 major objections 7 minor 1 cited by

RAG in the Wild: On the (In)effectiveness of LLMs with Mixture-of-Knowledge Retrieval Augmentation

T0 review · 4 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Retrieval augmentation mainly helps small language models; large models gain little from external knowledge in mixed-domain settings.

desk verdict A solid, honest empirical study whose main scale-dependence finding survives contact with the per-source data, but the routing negative result needs a defined oracle and a broader model sweep before it can be stated so broadly. read the letter →

arxiv 2507.20059 v1 pith:Y4H43MTS submitted 2025-07-26 cs.CL cs.AIcs.LG

Ran Xu , Yuchen Zhuang , Yue Yu , Haoyu Wang , Wenqi Shi , Carl Yang This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords retrieval-augmentedgenerationmixture-of-knowledgeRAGevaluationqueryroutingmodelscalererankingfactualityQAMassiveDS
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that retrieval-augmented generation (RAG) helps mainly when the backbone language model is small: over a heterogeneous datastore mixing many knowledge domains, the relative gains from retrieval shrink steadily as model size grows, and for the largest models the gains are near zero or negative on general, scientific, and computer-science QA. The exception is factuality-focused QA, where even large models improve. The authors reach this by comparing seven instruction-tuned LLMs from three families on six benchmarks, using a fixed dense retriever and reranker over a trillion-token datastore with ten corpora, and by measuring relative gain over a no-retrieval baseline. They also find that reranking adds little and that prompting LLMs to route queries to the best corpus usually underperforms simply retrieving from all sources. A sympathetic reader would care because real deployments must serve queries over messy, heterogeneous corpora, and these results suggest that static all-source retrieval is currently a safer default than prompt-based routing.

What carries the argument

The load-bearing machinery is the mixture-of-knowledge evaluation protocol: a trillion-token datastore (MassiveDS) partitioned into ten corpora, a fixed dense retriever (bge-base-en-v1.5) and a reranker (bge-reranker-v2-m3), zero-shot prompting on off-the-shelf instruction-tuned LLMs, and the relative-gain measure $\Delta(p_s)=(p_s-\rho)/\rho$, where $p_s$ is RAG performance with retrieved passages and $\rho$ is the no-retrieval baseline. This normalized measure is what turns 'does retrieval help?' into a model-scale comparison, and the instance-level analysis, which counts queries answerable only through one specific corpus, supplies the motivation for the routing experiments. The heterogeneous corpora plus the relative-gain measure together carry the paper's main claim that retrieval benefits concentrate in smaller models.

What would settle it

Re-run the six benchmarks with a substantially stronger retrieval stack (for example, a large reranker-based dense retriever, $k=20$ passages, and a RAG-fine-tuned backbone). If GPT-4o or Qwen3-32B then shows large relative gains, such as more than ten percent on MMLU-Pro or SciQ, the paper's scale-dependence claim collapses. Alternatively, train a supervised router on a few thousand labeled queries; if it matches the oracle upper bound on MMLU and MMLU-Pro, the claim that current LLMs cannot route effectively would need to be restricted to the zero-shot setting.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is a scale-dependent effectiveness curve for RAG under mixture-of-knowledge conditions. Retrieval produces large relative gains for Llama-3.2-3B (up to about +23% on MMLU subdomains) and Llama-3.1-8B, smaller gains for Qwen3-4B and Qwen3-8B, and near-zero or negative gains for Qwen3-32B, GPT-4o-mini, and GPT-4o on MMLU, MMLU-Pro, ARC Challenge, SciQ, and CSBench; the counterexample is SimpleQA factuality, where even GPT-4o improves with retrieval (0.343 to 0.463 using all sources). The paper further claims that reranking the top-30 retrieved passages to the top 5 with a reranker yields only marginal improvements, that no individual corpus consistently beats the no-retrieval baseline across datasets, and that LLM-based corpus routing with plain or chain-of-thought prompting fails to beat static all-source retrieval and sometimes falls below the no-retrieval baseline.

Load-bearing premise

The load-bearing premise is that the specific retrieval pipeline the paper used—the bge-base-en-v1.5 retriever with five passages and zero-shot prompting on off-the-shelf instruction-tuned models—is representative enough of real RAG deployments that the observed scale-dependent benefit pattern is a property of retrieval augmentation itself rather than an artifact of this setup.

Editorial extensions

If this is right

  • With this exact setup, upgrading the backbone from 3B to 32B or to GPT-4o reduces or eliminates the accuracy lift from retrieval on MMLU, MMLU-Pro, ARC-C, SciQ, and CSBench, so RAG pipelines cannot assume retrieval will pay off at large model scale.
  • Factuality-style QA remains the clear exception, so retrieval investment is better targeted at factual recall than at broad knowledge or reasoning benchmarks.
  • Reranking the top 30 passages down to the top 5 does not close the gap, which suggests the bottleneck under mixed knowledge is not simply retriever ranking quality.
  • Prompt-based corpus routing, with or without chain-of-thought, is currently an unreliable strategy; retrieving from all sources at once is a safer default in the paper's experiments.
  • Oracle routing upper bounds are substantially higher than any prompt-based router, indicating concrete headroom for learned routing modules.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same scale-dependence may extend to other non-parametric memory methods, such as long-context stuffing or memory banks; a testable extension is sweeping context length or memory size against these six benchmarks with the backbone held fixed.
  • Editorial inference: the routing failures may be an artifact of zero-shot prompting; a router fine-tuned on a few thousand labeled query-to-corpus pairs, or trained with reinforcement learning on the downstream QA reward, could approach the oracle upper bound and overturn the finding that static all-source retrieval wins.
  • Editorial inference: because no single corpus dominates, cheap retrieval-side signals—such as per-corpus top-$k$ score margins or query-corpus density estimates—might serve as more reliable routing evidence than LLM meta-reasoning, and could be tested without any new training.
  • Editorial inference: the negative gains from wrong-corpus retrieval suggest a conservative default: when confidence in source selection is low, answering with no retrieval may beat retrieving from a mismatched domain.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper empirically evaluates retrieval-augmented generation (RAG) in a multi-domain, mixture-of-knowledge setting built on the MassiveDS datastore. It benchmarks six QA datasets (MMLU, MMLU-Pro, ARC-C, SciQ, SimpleQA, CSBench) across seven instruction-tuned LLMs (Llama-3.2-3B, Llama-3.1-8B, Qwen3-4B/8B/32B, GPT-4o-mini/4o), using bge-base-en-v1.5 as the retriever and optionally bge-reranker-v2-m3. The main reported findings are that retrieval gains are largely confined to smaller models, that reranking adds only marginal value, that no single knowledge source consistently outperforms others, and that prompt-based LLM routing of queries across heterogeneous corpora is ineffective. The authors release code and data.

Significance. If the headline trend is robust, the paper provides a useful caution for practitioners: in heterogeneous, multi-source retrieval scenarios, a fixed one-size-fits-all RAG pipeline may benefit small models while adding noise for larger ones, and adaptive routing remains an unsolved problem. The study is commendably non-circular: benchmarks, datastore, retrievers, and models all come from prior or independent work, and the appendix tables provide detailed per-source accuracy results. The release of code and data supports reproducibility. However, the significance of the conclusions depends critically on the definition of the oracle router, the fairness of the 'All' retrieval condition, and the statistical support for the scale-dependence trend; these issues are the subject of the major comments below.

major comments (4)
  1. [Section 2.5, Figure 4] The 'oracle router upper bound' is never defined, and its numerical values are not reported in the text or tables. The routing conclusion — that LLM-prompted routing often underperforms static retrieval — is only meaningful relative to a well-defined upper bound, but the reader cannot tell whether the oracle is the per-query best source, the best single source across the dataset, or an upper envelope over all sources. In addition, the routing experiment uses only Qwen-3 models (4B, 8B, 32B), so the claim that 'current LLMs struggle to route queries' is not tested across the Llama and GPT families used elsewhere in the paper. Please define the oracle operationally, report its accuracy for MMLU and MMLU-Pro, and extend the routing evaluation to at least one non-Qwen model.
  2. [Section 2.2, Table 5, Figure 1] The headline scale-dependence finding is measured against the 'All' retrieval condition, which is a fixed, non-routed retrieval from the concatenation of ten heterogeneous corpora. The paper does not establish that this condition is a fair or representative RAG setup. For example, Table 5 shows that on MMLU with Llama-3.2-3B, the 'All' gain over no retrieval (0.552 vs 0.481, about +14.8% relative) is largely captured by Wikipedia alone (0.534), and for the larger models the per-source columns are mostly at or below the no-retrieval baseline. Consequently, the observed diminishing returns could reflect the noise added by a one-size-fits-all retrieval pipeline rather than an intrinsic property of retrieval augmentation at larger scale. To separate these explanations, report for each large model the accuracy achievable with the best single source per query (or an oracle-routed source) and show whether the scale trend persists under that condition.
  3. [Section 2.2, Table 1 and Tables 2-7] No measure of variability is reported anywhere in the paper. Several differences used to support a monotone scale trend are small — e.g., Table 1 MMLU Humanities relative gains of +7.70% for Llama-3.2-3B versus +7.85% for Llama-3.1-8B, and Table 5 GPT-4o MMLU with retrieval (0.828) versus without (0.833) — and Figure 1 contains many near-zero values. Without confidence intervals, bootstrap estimates, or significance tests, the claim that gains 'diminish significantly' with model scale is not statistically supported. At minimum, provide bootstrap CIs over question-level predictions, or rephrase the claim as a descriptive trend rather than a significant effect.
  4. [Abstract, Introduction, and Limitations] The abstract and introduction state the findings as unconditional properties of RAG ('retrieval mainly benefits smaller models', 'rerankers add minimal value'), but the experiments use a single dense retriever (bge-base-en-v1.5), a single reranker (bge-reranker-v2-m3), k=5, and zero-shot prompting of off-the-shelf instruction-tuned models. The Limitations section appropriately acknowledges that alternative retrieval paradigms and larger open-source models are not covered, yet this qualification is absent from the high-level claims and from the title's 'in the Wild' framing. Either narrow the claims to the tested pipeline or add a sensitivity analysis (e.g., a stronger retriever, a larger k, or a RAG-finetuned backbone) to show that the scale-dependence pattern is not an artifact of the specific setup.
minor comments (7)
  1. [Throughout] The naming is inconsistent: 'MASSIVE DS' appears in the abstract while 'MassiveDS' is used in Section 2.1; standardize one form.
  2. [Section 2.1] The relative-gain formula is typeset as 'ps−ρ ρ' and is ambiguous; clarify it as (p_s − ρ)/ρ and state whether reported gains are fractions (as in Figure 1) or percentages (as in Table 1).
  3. [Section 2.1] 'k = 5 top passage' should be 'k = 5 top passages'; it would also help to state explicitly that in the 'All' condition the retrieved passages may come from multiple corpora.
  4. [Figure 1 caption] The caption says 'relevance performance', but the figure actually plots relative gain over the no-retrieval baseline; the color scale and units should be defined, and negative values should be explicitly interpreted as retrieval hurting performance.
  5. [Section 2.5, Figure 4] Please clarify in the text or caption whether the router model is the same model that subsequently answers the question; the legend lists only model names and prompting variants without making the pipeline explicit.
  6. [Appendix A, Figures 7 and 8] There are grammatical errors ('The model have enough knowledge', 'Please concise reasoning'), and the 'no' option's description ('The model have enough knowledge to answer the question') may bias the router toward the no-retrieval choice; consider a more neutral phrasing.
  7. [Section 3] 'we also focuses on routing' should be 'we also focus on routing'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the study is an external, parameter-free evaluation whose conclusions are not fitted to its own inputs.

full rationale

The paper's claims are empirical generalizations from a fixed evaluation protocol: six external benchmarks (MMLU, MMLU-Pro, ARC-C, SciQ, CSBench, SimpleQA), an external datastore (MassiveDS), an external retriever (bge-base-en-v1.5), an external reranker (bge-reranker-v2-m3), and off-the-shelf instruction-tuned backbones. No parameter is fitted to any subset of the results and then reported as a prediction; the only derived quantity is the definitional relative gain Delta(ps) = (ps - rho) / rho (Section 2.1), which is a reporting metric, not a fitted claim. The scale-dependent benefit finding is read directly from Table 1 and Figures 1-3 and is independently checkable against the full accuracy tables in Appendix B. No load-bearing step reduces to a self-citation: the author team's prior work cited in Related Work (Xu et al. 2024a, 2024b) is not used to establish any central premise, and the MassiveDS setup follows Shao et al. (2024), an external group, not the present authors. Two non-circular reporting weaknesses are flagged per the review rule: (i) Section 2.5 and Figure 4 introduce an 'oracle router upper bound' that is never defined or numerically reported, so the routing conclusion rests on the plain versus all-sources comparison rather than on the oracle; (ii) the Limitations section explicitly concedes that the study targets short-form QA and does not cover larger open-source models or alternative retrieval paradigms, which bounds generalizability without making any derivation circular. Neither issue makes a result equivalent-by-construction to its inputs, so the circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities or fitted parameters. The central claims rest on domain assumptions about representativeness of datasets, corpora, prompting, and the unstated oracle router.

free parameters (2)
  • k (number of retrieved passages) = 5
    Hand-chosen constant in Section 2.1; the paper does not test sensitivity of the main conclusions to k, though RAG benefit can depend on retrieval depth.
  • k' (pre-rerank retrieved passages) = 30
    Hand-chosen in Section 2.1 for reranking experiments; only one depth is tested.
assumptions (4)
  • domain assumption MassiveDS with 10 named sources is a valid proxy for real-world heterogeneous knowledge.
    The paper's mixture-of-knowledge claim assumes these sources cover the query distribution; Section 2.1 Retrieval Corpora.
  • domain assumption Zero-shot prompting with the provided templates captures the models' actual RAG capability.
    No RAG-specific fine-tuning is applied; Appendix A and Section 2.1 Implementation Details. A RAG-finetuned model could show different gains.
  • ad hoc to paper The oracle router in Figure 4 provides a valid upper bound for source routing.
    The construction of the oracle is not specified; the routing conclusions depend on this unstated assumption; Section 2.5 and Figure 4.
  • domain assumption The six selected benchmarks are representative knowledge-intensive QA tasks.
    Only short-form QA and multiple-choice tasks are evaluated; the Limitations section notes this scope, so generalization to open-ended generation is an assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RAG in the Wild: On the (In)effectiveness of LLMs with Mixture-of-Knowledge Retrieval Augmentation." pith.science (2026). https://pith.science/paper/Y4H43MTS

@misc{pith2026250720059,
  author       = {Pith},
  title        = {Pith review of: RAG in the Wild: On the (In)effectiveness of LLMs with Mixture-of-Knowledge Retrieval Augmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y4H43MTS}},
  note         = {Machine review of arXiv:2507.20059}
}
read the original abstract

Retrieval-augmented generation (RAG) enhances large language models (LLMs) by integrating external knowledge retrieved at inference time. While RAG demonstrates strong performance on benchmarks largely derived from general-domain corpora like Wikipedia, its effectiveness under realistic, diverse retrieval scenarios remains underexplored. We evaluated RAG systems using MassiveDS, a large-scale datastore with mixture of knowledge, and identified critical limitations: retrieval mainly benefits smaller models, rerankers add minimal value, and no single retrieval source consistently excels. Moreover, current LLMs struggle to route queries across heterogeneous knowledge sources. These findings highlight the need for adaptive retrieval strategies before deploying RAG in real-world settings. Our code and data can be found at https://github.com/ritaranx/RAG_in_the_Wild.

Figures

Figures reproduced from arXiv: 2507.20059 by the authors.

Figure 1
Figure 1. The relevance performance of different LLMs compared to non-retrieval baselines on six datasets. ‘All’ [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Number of cases (in %) specifically resolved by retrieving from an individual corpus. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Performance with rerank on retrieval effectiveness across different datasets and models. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Performance comparison of routing strategies across MMLU and MMLU-Pro datasets. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Prompt for answer generation on multi-choice questions (e.g. MMLU, MMLU-Pro). [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Prompt for answer generation on span-based questions (e.g. SciQ, SimpleQA). [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Prompt for question routing without chain-of-thought prompting. [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Prompt for question routing with chain-of-thought prompting ( [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Improving Language Agents through BREW: Bootstrapping expeRientially-learned Environmental knoWledge

    cs.AI 2025-11 conditional novelty 5.0 of 10

    An LLM agent can improve by storing and Monte-Carlo-refining concept-level natural-language recipes mined from its own past trajectories.

Reference graph

Works this paper leans on

53 extracted references · 11 canonical work pages · cited by 1 Pith paper

  1. [1]

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. https://openreview.net/forum?id=hSyW5go0v8 Self- RAG : Learning to retrieve, generate, and critique through self-reflection . In The Twelfth International Conference on Learning Representations

    2024

  2. [2]

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216

    arXiv 2024

  3. [3]

    Zhe Chen, Yusheng Liao, Shuyang Jiang, Pingjie Wang, Yiqiu Guo, Yanfeng Wang, and Yu Wang. 2025. Towards omni-rag: Comprehensive retrieval-augmented generation for large language models in medical applications. arXiv preprint arXiv:2501.02460

    arXiv 2025

  4. [4]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    arXiv 2018

  5. [5]

    Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. 2024. The power of noise: Redefining retrieval for rag systems. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 719--729

    2024

  6. [6]

    Evan Frick, Connor Chen, Joseph Tennyson, Tianle Li, Wei-Lin Chiang, Anastasios N Angelopoulos, and Ion Stoica. 2025. Prompt-to-leaderboard. arXiv preprint arXiv:2502.14855

    arXiv 2025

  7. [7]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

  8. [8]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. https://openreview.net/forum?id=d7KBjmI3GmQ Measuring massive multitask language understanding . In International Conference on Learning Representations

    2021

Show all 53 references
  1. [9]

    Jie Huang, Wei Ping, Peng Xu, Mohammad Shoeybi, Kevin Chang, and Bryan Catanzaro. 2024. https://openreview.net/forum?id=GMalvQu0XL RAVEN : In-context learning with retrieval-augmented encoder-decoder language models . In First Conference on Language Modeling

    2024

  2. [10]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  3. [11]

    Shayekh Bin Islam, Md Asib Rahman, K S M Tozammel Hossain, Enamul Hoque, Shafiq Joty, and Md Rizwan Parvez. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.831 Open- RAG : Enhanced retrieval augmented reasoning with open-source large language models . In Findings of the ...

    2024 doi

  4. [12]

    Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2023. Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research, 24(251):1--43

    2023

  5. [13]

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. 2025. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516

    2025 arXiv

  6. [14]

    Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. https://doi.org/10.18653/v1/P17-1147 T rivia QA : A large scale distantly supervised challenge dataset for reading comprehension . In Proceedings of the 55th Annual Meeting of the Association for Computational...

    2017 doi

  7. [15]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav...

    2019

  8. [16]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information proc...

    2020

  9. [17]

    Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. 2025. Search-o1: Agentic search-enhanced large reasoning models. arXiv preprint arXiv:2501.05366

    2025 arXiv

  10. [18]

    Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Richard James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih. 2024. https://openreview.net/forum?id=22OTbutug9 RA - DIT : Retrieval-augmented dual instruction tun...

    2024

  11. [19]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

    2024 arXiv

  12. [20]

    Xinbei Ma, Yeyun Gong, Pengcheng He, hai zhao, and Nan Duan. 2023. https://openreview.net/forum?id=gXq1cwkUZc Query rewriting in retrieval-augmented large language models . In The 2023 Conference on Empirical Methods in Natural Language Processing

    2023

  13. [21]

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. https://doi.org/10.18653/v1/2023.acl-long.546 When not to trust language models: Investigating effectiveness of parametric and non-parametric memories . In Proceedings of the 6...

    2023 doi

  14. [22]

    AI Meta. 2025. https://ai.meta.com/blog/llama-4-multimodal-intelligence/ The llama 4 herd: The beginning of a new era of natively multimodal ai innovation

    2025

  15. [23]

    Feiteng Mu, Yong Jiang, Liwen Zhang, Liuchu Liuchu, Wenjie Li, Pengjun Xie, and Fei Huang. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.598 Query routing for homogeneous tools: An instantiation in the RAG scenario . In Findings of the Association for Computational Lin...

    2024 doi

  16. [24]

    Feiteng Mu, Liwen Zhang, Yong Jiang, Wenjie Li, Zhen Zhang, Pengjun Xie, and Fei Huang. 2025. Unsupervised query routing for retrieval augmented generation. arXiv preprint arXiv:2501.07793

    2025 arXiv

  17. [25]

    Cheng Niu, Yuanhao Wu, Juno Zhu, Siliang Xu, KaShun Shum, Randy Zhong, Juntong Song, and Tong Zhang. 2024. https://doi.org/10.18653/v1/2024.acl-long.585 RAGT ruth: A hallucination corpus for developing trustworthy retrieval-augmented language models . In Proceedings of the 62n...

    2024 doi

  18. [26]

    Gonzalez, M Waleed Kadous, and Ion Stoica

    Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica. 2025. https://openreview.net/forum?id=8sSqNntaMr Route LLM : Learning to route LLM s from preference data . In The Thirteenth International Conference on Le...

    2025

  19. [27]

    Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rockt \"a schel, and Sebastian Riedel. 2021. https://doi.org/10.18653/v1/2021.naacl-main.200 KIL...

    2021 doi

  20. [28]

    Rulin Shao, Jacqueline He, Akari Asai, Weijia Shi, Tim Dettmers, Sewon Min, Luke Zettlemoyer, and Pang Wei W Koh. 2024. Scaling retrieval-based language models with a trillion-token datastore. Advances in Neural Information Processing Systems, 37:91260--91299

    2024

  21. [29]

    Rulin Shao, Rui Qiao, Varsha Kishore, Niklas Muennighoff, Xi Victoria Lin, Daniela Rus, Bryan Kian Hsiang Low, Sewon Min, Wen-tau Yih, Pang Wei Koh, et al. 2025. Reasonir: Training retrievers for reasoning tasks. arXiv preprint arXiv:2504.20595

    2025 arXiv

  22. [30]

    Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.620 Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy . In Findings of the Association for Comput...

    2023 doi

  23. [31]

    Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Richard James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2024. https://doi.org/10.18653/v1/2024.naacl-long.463 REPLUG : Retrieval-augmented black-box language models . In Proceedings of the 2024 Conference of the Nor...

    2024 doi

  24. [32]

    Xiaoshuai Song, Muxi Diao, Guanting Dong, Zhengyang Wang, Yujia Fu, Runqi Qiao, Zhexu Wang, Dayuan Fu, Huangxuan Wu, Bin Liang, Weihao Zeng, Yejie Wang, Zhuoma GongQue, Jianing Yu, Qiuna Tan, and Weiran Xu. 2025. https://openreview.net/forum?id=fjEZ2LPceZ CS -bench: A comprehe...

    2025

  25. [33]

    James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. https://aclanthology.org/N18-1074 FEVER : a large-scale dataset for fact extraction and VER ification . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association f...

    2018

  26. [34]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2023. https://doi.org/10.18653/v1/2023.acl-long.557 Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions . In Proceedings of the 61st Annual Meeting of th...

    2023 doi

  27. [35]

    Haoyu Wang, Ruirui Li, Haoming Jiang, Jinjin Tian, Zhengyang Wang, Chen Luo, Xianfeng Tang, Monica Xiao Cheng, Tuo Zhao, and Jing Gao. 2024 a . https://doi.org/10.18653/v1/2024.emnlp-main.58 B lend F ilter: Advancing retrieval-augmented large language models via query generati...

    2024 doi

  28. [36]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024 b . https://openreview.net/forum?id=y10DM6R2r3 MMLU -pro: A m...

    2024

  29. [37]

    Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. 2024. Measuring short-form factuality in large language models. arXiv preprint arXiv:2411.04368

    2024 arXiv

  30. [38]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  31. [39]

    Johannes Welbl, Nelson F Liu, and Matt Gardner. 2017. Crowdsourcing multiple choice science questions. arXiv preprint arXiv:1707.06209

    2017 arXiv

  32. [40]

    Di Wu, Jia-Chen Gu, Kai-Wei Chang, and Nanyun Peng. 2025. Self-routing rag: Binding selective retrieval with knowledge verbalization. arXiv preprint arXiv:2504.01018

    2025

  33. [41]

    Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. 2024. https://doi.org/10.18653/v1/2024.findings-acl.372 Benchmarking retrieval-augmented generation for medicine . In Findings of the Association for Computational Linguistics: ACL 2024, pages 6233--6251, Bangkok, Thailan...

    2024 doi

  34. [42]

    Ran Xu, Wenqi Shi, Yue Yu, Yuchen Zhuang, Bowen Jin, May Dongmei Wang, Joyce Ho, and Carl Yang. 2024 a . https://doi.org/10.18653/v1/2024.acl-short.68 RAM - EHR : Retrieval augmentation meets clinical predictions on electronic health records . In Proceedings of the 62nd Annual...

    2024 doi

  35. [43]

    Ho, Chao Zhang, and Carl Yang

    Ran Xu, Wenqi Shi, Yue Yu, Yuchen Zhuang, Yanqiao Zhu, May Dongmei Wang, Joyce C. Ho, Chao Zhang, and Carl Yang. 2024 b . https://doi.org/10.18653/v1/2024.emnlp-main.1241 Bmretriever: Tuning large language models as better biomedical text retrievers . In Proceedings of the 202...

    2024 doi

  36. [44]

    Shicheng Xu, Liang Pang, Mo Yu, Fandong Meng, Huawei Shen, Xueqi Cheng, and Jie Zhou. 2024 c . https://doi.org/10.18653/v1/2024.acl-long.9 Unsupervised information refinement training of large language models for retrieval-augmented generation . In Proceedings of the 62nd Annu...

    2024 doi

  37. [45]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388

    2025 arXiv

  38. [46]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. https://aclanthology.org/D18-1259 H otpot QA : A dataset for diverse, explainable multi-hop question answering . In Proceedings of the 2018 Conference on...

    2018

  39. [47]

    Zijun Yao, Weijian Qi, Liangming Pan, Shulin Cao, Linmei Hu, Weichuan Liu, Lei Hou, and Juanzi Li. 2024. Seakr: Self-aware knowledge retrieval for adaptive retrieval augmented generation. arXiv preprint arXiv:2406.19215

    2024 arXiv

  40. [48]

    Yue Yu, Wei Ping, Zihan Liu, Boxin Wang, Jiaxuan You, Chao Zhang, Mohammad Shoeybi, and Bryan Catanzaro. 2024. https://openreview.net/forum?id=S1fc92uemC Rank RAG : Unifying context ranking with retrieval-augmented generation in LLM s . In The Thirty-eighth Annual Conference o...

    2024

  41. [49]

    Jiarui Zhang, Xiangyu Liu, Yong Hu, Chaoyue Niu, Fan Wu, and Guihai Chen. 2025. Query routing for retrieval-augmented language models. arXiv preprint arXiv:2505.23052

    2025

  42. [50]

    Gonzalez

    Tianjun Zhang, Shishir G Patil, Naman Jain, Sheng Shen, Matei Zaharia, Ion Stoica, and Joseph E. Gonzalez. 2024. https://openreview.net/forum?id=rzQGHXNReU RAFT : Adapting language model to domain specific RAG . In First Conference on Language Modeling

    2024

  43. [51]

    Shengyao Zhuang, Xueguang Ma, Bevan Koopman, Jimmy Lin, and Guido Zuccon. 2025. Rank-r1: Enhancing reasoning in llm-based document rerankers via reinforcement learning. arXiv preprint arXiv:2503.06034

    2025 arXiv

  44. [52]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  45. [53]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.