{"id":"e8804b1c-f3ec-4c03-bd86-f651c8c2b4cf","arxiv_id":"2507.20059","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Retrieval augmentation mainly helps smaller LLMs, rerankers add marginal value, and current LLMs cannot reliably route queries across heterogeneous knowledge sources.","lead":"This paper tests retrieval-augmented generation (RAG) with a ten-source knowledge store across seven language models and six question-answering benchmarks. It finds that retrieval mainly helps smaller models, that rerankers add little, and that language models cannot reliably choose which source to search.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Scale-dependent benefit pattern is confounded by a ceiling in all-sources retrieval quality; the headroom between the 'All' column and the oracle upper bound is not shown to be reachable without fixed 'no' routing.","rationale":"The reader's weakest assumption focuses on retriever/k representativeness, which is a valid external-validity threat. My concern is narrower and internal: the paper's headline finding that retrieval benefits are largely confined to smaller models is operationalized through un-routed 'All' retrieval, while the paper's own Section 2.3 instance-level analysis (Figure 2) demonstrates that per-source unique value exists across model sizes, and the routing experiment (Figure 4) shows that an oracle upper bound is intended but never defined or reported. The load-bearing logical step is: because the 'All' pipeline cannot realize the per-source unique value the paper documents, the observed shrinking gain with model scale conflates retrieval-augmentation value with the noise/inefficiency of the fixed, non-adaptive mixture-of-knowledge pipeline. The paper explicitly flags routing as a failure mode (Section 2.5), but the routing experiment itself lacks the precise oracle needed to serve as the counterfactual. Consequently, the strong conclusion is underdetermined, not contradicted: the per-source gains in Figure 2 are computed for small (Llama-3.2-3B) and 8B models, not for Qwen3-32B or GPT-4o, so the data as presented cannot rule out that bigger models would still benefit from correctly routed per-source retrieval. The paper's own Limitations section concedes it does not cover larger open-source models or alternative retrieval paradigms, and the scarcity of error bars and the undefined oracle were already flagged. I therefore agree with the CONDITIONAL verdict, not because the internal tables are inconsistent, but because the central inference requires a counterfactual the paper does not fully provide. The concrete test above would either rescue the scale-dependence claim (if the oracle gap is small for large models) or transform the paper's contribution into a router-deficit finding.","tokens_in":21693,"tokens_out":2336,"duration_ms":20917,"concrete_test":"Define the oracle router precisely (e.g., for each test query, select the single corpus whose retrieved passages maximize downstream accuracy as measured by exhaustive per-source forward passes, never selecting 'no' unless no source exceeds the no-retrieval baseline). Then, on MMLU and MMLU-Pro, compute the per-model oracle accuracy for Llama-3.2-3B, Qwen3-8B, Qwen3-32B, and GPT-4o, and compare it against the plain and 'All' columns reported in Figure 4 and Tables 5-6. If the oracle-minus-plain gap remains large and positive at Qwen3-32B and GPT-4o while the 'All'-minus-plain gap shrinks, the scale-dependent conclusion is an artifact of un-routed retrieval and the paper's routing claim becomes the central finding rather than diminishing RAG value.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that retrieval gains shrink with model scale. The tables show the 'All' (retrieve from every corpus) column as the RAG condition, and the Figure 4 oracle upper bound is stated in the routing experiment but never defined nor reported numerically; the reader flagged this as an undefined oracle. The load-bearing issue is that the 'All' condition is a fixed, un-routed concatenation of retrieved passages over ten heterogeneous corpora. With k=5, a query from any single domain will usually receive zero or one relevant passage from its true source plus four irrelevant passages from other domains. Empirically, Table 5 shows 'All' as 0.552 for Llama-3.2-3B versus plain 0.481, while single-source English Wikipedia alone gives 0.534, which is close to the entire gain. For large models, the comparison is between plain and 'All'; if per-source retrieval quality is the binding constraint, the correct floor for the scale-dependence claim is the per-source maximum with an optimal router, which the paper shows (Figure 2) to be well above 'All' for several datasets (e.g., 8-39% of queries solved only by specific corpora for Llama-3.1-8B). Without a defined oracle and without routing, the paper cannot distinguish the strong claim 'retrieval augmentation itself has diminishing returns for large models' from the weaker claim 'a fixed, non-adaptive one-size-fits-all retrieval pipeline fails to capture per-source value, and larger models discount the noise it adds.' The reader's weakest_assumption identified the representativeness of the retriever and k; the internal point here is that the paper's own instance-level analysis (Section 2.3) shows that source-specific unique value exists for large models (e.g., Qwen3-32B has per-source contributions, though small), so the routing failure is itself entangled with the headline scale-dependent finding.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper empirically evaluates retrieval-augmented generation (RAG) in a multi-domain, mixture-of-knowledge setting built on the MassiveDS datastore. It benchmarks six QA datasets (MMLU, MMLU-Pro, ARC-C, SciQ, SimpleQA, CSBench) across seven instruction-tuned LLMs (Llama-3.2-3B, Llama-3.1-8B, Qwen3-4B/8B/32B, GPT-4o-mini/4o), using bge-base-en-v1.5 as the retriever and optionally bge-reranker-v2-m3. The main reported findings are that retrieval gains are largely confined to smaller models, that reranking adds only marginal value, that no single knowledge source consistently outperforms others, and that prompt-based LLM routing of queries across heterogeneous corpora is ineffective. The authors release code and data.","tokens_in":22042,"tokens_out":9428,"duration_ms":79791,"significance":"If the headline trend is robust, the paper provides a useful caution for practitioners: in heterogeneous, multi-source retrieval scenarios, a fixed one-size-fits-all RAG pipeline may benefit small models while adding noise for larger ones, and adaptive routing remains an unsolved problem. The study is commendably non-circular: benchmarks, datastore, retrievers, and models all come from prior or independent work, and the appendix tables provide detailed per-source accuracy results. The release of code and data supports reproducibility. However, the significance of the conclusions depends critically on the definition of the oracle router, the fairness of the 'All' retrieval condition, and the statistical support for the scale-dependence trend; these issues are the subject of the major comments below.","major_comments":[{"comment":"The 'oracle router upper bound' is never defined, and its numerical values are not reported in the text or tables. The routing conclusion — that LLM-prompted routing often underperforms static retrieval — is only meaningful relative to a well-defined upper bound, but the reader cannot tell whether the oracle is the per-query best source, the best single source across the dataset, or an upper envelope over all sources. In addition, the routing experiment uses only Qwen-3 models (4B, 8B, 32B), so the claim that 'current LLMs struggle to route queries' is not tested across the Llama and GPT families used elsewhere in the paper. Please define the oracle operationally, report its accuracy for MMLU and MMLU-Pro, and extend the routing evaluation to at least one non-Qwen model.","section":"Section 2.5, Figure 4"},{"comment":"The headline scale-dependence finding is measured against the 'All' retrieval condition, which is a fixed, non-routed retrieval from the concatenation of ten heterogeneous corpora. The paper does not establish that this condition is a fair or representative RAG setup. For example, Table 5 shows that on MMLU with Llama-3.2-3B, the 'All' gain over no retrieval (0.552 vs 0.481, about +14.8% relative) is largely captured by Wikipedia alone (0.534), and for the larger models the per-source columns are mostly at or below the no-retrieval baseline. Consequently, the observed diminishing returns could reflect the noise added by a one-size-fits-all retrieval pipeline rather than an intrinsic property of retrieval augmentation at larger scale. To separate these explanations, report for each large model the accuracy achievable with the best single source per query (or an oracle-routed source) and show whether the scale trend persists under that condition.","section":"Section 2.2, Table 5, Figure 1"},{"comment":"No measure of variability is reported anywhere in the paper. Several differences used to support a monotone scale trend are small — e.g., Table 1 MMLU Humanities relative gains of +7.70% for Llama-3.2-3B versus +7.85% for Llama-3.1-8B, and Table 5 GPT-4o MMLU with retrieval (0.828) versus without (0.833) — and Figure 1 contains many near-zero values. Without confidence intervals, bootstrap estimates, or significance tests, the claim that gains 'diminish significantly' with model scale is not statistically supported. At minimum, provide bootstrap CIs over question-level predictions, or rephrase the claim as a descriptive trend rather than a significant effect.","section":"Section 2.2, Table 1 and Tables 2-7"},{"comment":"The abstract and introduction state the findings as unconditional properties of RAG ('retrieval mainly benefits smaller models', 'rerankers add minimal value'), but the experiments use a single dense retriever (bge-base-en-v1.5), a single reranker (bge-reranker-v2-m3), k=5, and zero-shot prompting of off-the-shelf instruction-tuned models. The Limitations section appropriately acknowledges that alternative retrieval paradigms and larger open-source models are not covered, yet this qualification is absent from the high-level claims and from the title's 'in the Wild' framing. Either narrow the claims to the tested pipeline or add a sensitivity analysis (e.g., a stronger retriever, a larger k, or a RAG-finetuned backbone) to show that the scale-dependence pattern is not an artifact of the specific setup.","section":"Abstract, Introduction, and Limitations"}],"minor_comments":[{"comment":"The naming is inconsistent: 'MASSIVE DS' appears in the abstract while 'MassiveDS' is used in Section 2.1; standardize one form.","section":"Throughout"},{"comment":"The relative-gain formula is typeset as 'ps−ρ ρ' and is ambiguous; clarify it as (p_s − ρ)/ρ and state whether reported gains are fractions (as in Figure 1) or percentages (as in Table 1).","section":"Section 2.1"},{"comment":"'k = 5 top passage' should be 'k = 5 top passages'; it would also help to state explicitly that in the 'All' condition the retrieved passages may come from multiple corpora.","section":"Section 2.1"},{"comment":"The caption says 'relevance performance', but the figure actually plots relative gain over the no-retrieval baseline; the color scale and units should be defined, and negative values should be explicitly interpreted as retrieval hurting performance.","section":"Figure 1 caption"},{"comment":"Please clarify in the text or caption whether the router model is the same model that subsequently answers the question; the legend lists only model names and prompting variants without making the pipeline explicit.","section":"Section 2.5, Figure 4"},{"comment":"There are grammatical errors ('The model have enough knowledge', 'Please concise reasoning'), and the 'no' option's description ('The model have enough knowledge to answer the question') may bias the router toward the no-retrieval choice; consider a more neutral phrasing.","section":"Appendix A, Figures 7 and 8"},{"comment":"'we also focuses on routing' should be 'we also focus on routing'.","section":"Section 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is an honest empirical study with public code and data, and the topic is timely. The main problems — an undefined oracle, a potentially unrepresentative 'All' retrieval condition, and missing statistical support for the headline trend — are load-bearing but fixable in revision. I would not reject, but the claims as currently stated outrun the evidence; a revision that defines the oracle, adds per-source or oracle-routed comparisons for large models, and reports uncertainty would make the paper's conclusions much more credible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline claim is mostly right but needs a qualifier: retrieval helps smaller models a lot, large models little, except on factuality tasks. The paper's own tables show this pattern holds even when you compare against the best single-source retrieval, not just the fixed 'All' mixture. For MMLU with GPT-4o, plain accuracy is 0.833 and no retrieval condition, including the best single source, gets above 0.828. So the strong version of the scale-dependence claim is not an artifact of a weak 'All' baseline; it survives contact with the per-source columns.\n\nWhat is genuinely new here is the evaluation setting itself: a mixture-of-knowledge datastore with ten heterogeneous corpora, seven LLMs, and six QA benchmarks. The instance-level analysis of per-source contributions (Figure 2) and the negative results on reranking and LLM-as-router are useful evaluation angles. The experiments are fully external, with no circularity, and the code and data are promised. The paper also gets credit for stating its own scope in the limitations section: short-form QA, no efficiency analysis, no larger open-source models.\n\nThe soft spots are real but addressable. There are no error bars or significance tests, so small differences like +7.70% versus +7.85% are indistinguishable from noise. The oracle router in Figure 4 is never defined or reported numerically, which matters because the routing conclusion is supposed to show there is headroom. That conclusion is also drawn from Qwen-3 models only, with zero-shot prompting, and then generalized to 'current LLMs' — that is too broad. The stress-test concern about the 'All' condition being a ceiling is mostly wrong, because All usually beats every single source in the tables, but the entanglement between routing failure and scale dependence is real: Figure 2 shows that even large models get unique value from specific sources on some queries, which is precisely where the undefined oracle and the narrow router set leave the biggest gap.\n\nThis paper is for people building multi-source RAG systems and for readers interested in empirical scaling behavior of retrieval augmentation. It deserves a serious referee. My recommendation: send it to peer review, and ask for error bars, a defined and numerically reported oracle, a broader router model sweep, and a robustness check with a stronger retriever or larger k. With those fixes, it becomes a useful reference point for the field.","headline":"A solid, honest empirical study whose main scale-dependence finding survives contact with the per-source data, but the routing negative result needs a defined oracle and a broader model sweep before it can be stated so broadly.","tokens_in":22614,"tokens_out":2494,"would_cite":true,"duration_ms":27554,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Retrieval augmentation mainly helps small language models; large models gain little from external knowledge in mixed-domain settings.","keywords":["retrieval-augmented generation","mixture-of-knowledge","RAG evaluation","query routing","model scale","reranking","factuality QA","MassiveDS"],"falsifier":"Re-run the six benchmarks with a substantially stronger retrieval stack (for example, a large reranker-based dense retriever, $k=20$ passages, and a RAG-fine-tuned backbone). If GPT-4o or Qwen3-32B then shows large relative gains, such as more than ten percent on MMLU-Pro or SciQ, the paper's scale-dependence claim collapses. Alternatively, train a supervised router on a few thousand labeled queries; if it matches the oracle upper bound on MMLU and MMLU-Pro, the claim that current LLMs cannot route effectively would need to be restricted to the zero-shot setting.","tokens_in":21514,"feed_emoji":"🔍","tokens_out":10806,"duration_ms":87654,"temperature":0.7,"pith_summary":"This paper tries to establish that retrieval-augmented generation (RAG) helps mainly when the backbone language model is small: over a heterogeneous datastore mixing many knowledge domains, the relative gains from retrieval shrink steadily as model size grows, and for the largest models the gains are near zero or negative on general, scientific, and computer-science QA. The exception is factuality-focused QA, where even large models improve. The authors reach this by comparing seven instruction-tuned LLMs from three families on six benchmarks, using a fixed dense retriever and reranker over a trillion-token datastore with ten corpora, and by measuring relative gain over a no-retrieval baseline. They also find that reranking adds little and that prompting LLMs to route queries to the best corpus usually underperforms simply retrieving from all sources. A sympathetic reader would care because real deployments must serve queries over messy, heterogeneous corpora, and these results suggest that static all-source retrieval is currently a safer default than prompt-based routing.","feed_headline":"Retrieval gains shrink as language models grow in multi-corpus RAG","feed_subtitle":"A multi-benchmark study over a trillion-token datastore finds reranking and routing add little for large models.","key_machinery":"The load-bearing machinery is the mixture-of-knowledge evaluation protocol: a trillion-token datastore (MassiveDS) partitioned into ten corpora, a fixed dense retriever (bge-base-en-v1.5) and a reranker (bge-reranker-v2-m3), zero-shot prompting on off-the-shelf instruction-tuned LLMs, and the relative-gain measure $\\Delta(p_s)=(p_s-\\rho)/\\rho$, where $p_s$ is RAG performance with retrieved passages and $\\rho$ is the no-retrieval baseline. This normalized measure is what turns 'does retrieval help?' into a model-scale comparison, and the instance-level analysis, which counts queries answerable only through one specific corpus, supplies the motivation for the routing experiments. The heterogeneous corpora plus the relative-gain measure together carry the paper's main claim that retrieval benefits concentrate in smaller models.","core_discovery":"On the paper's own terms, the discovery is a scale-dependent effectiveness curve for RAG under mixture-of-knowledge conditions. Retrieval produces large relative gains for Llama-3.2-3B (up to about +23% on MMLU subdomains) and Llama-3.1-8B, smaller gains for Qwen3-4B and Qwen3-8B, and near-zero or negative gains for Qwen3-32B, GPT-4o-mini, and GPT-4o on MMLU, MMLU-Pro, ARC Challenge, SciQ, and CSBench; the counterexample is SimpleQA factuality, where even GPT-4o improves with retrieval (0.343 to 0.463 using all sources). The paper further claims that reranking the top-30 retrieved passages to the top 5 with a reranker yields only marginal improvements, that no individual corpus consistently beats the no-retrieval baseline across datasets, and that LLM-based corpus routing with plain or chain-of-thought prompting fails to beat static all-source retrieval and sometimes falls below the no-retrieval baseline.","pith_inferences":["Editorial inference: the same scale-dependence may extend to other non-parametric memory methods, such as long-context stuffing or memory banks; a testable extension is sweeping context length or memory size against these six benchmarks with the backbone held fixed.","Editorial inference: the routing failures may be an artifact of zero-shot prompting; a router fine-tuned on a few thousand labeled query-to-corpus pairs, or trained with reinforcement learning on the downstream QA reward, could approach the oracle upper bound and overturn the finding that static all-source retrieval wins.","Editorial inference: because no single corpus dominates, cheap retrieval-side signals—such as per-corpus top-$k$ score margins or query-corpus density estimates—might serve as more reliable routing evidence than LLM meta-reasoning, and could be tested without any new training.","Editorial inference: the negative gains from wrong-corpus retrieval suggest a conservative default: when confidence in source selection is low, answering with no retrieval may beat retrieving from a mismatched domain."],"forward_implications":["With this exact setup, upgrading the backbone from 3B to 32B or to GPT-4o reduces or eliminates the accuracy lift from retrieval on MMLU, MMLU-Pro, ARC-C, SciQ, and CSBench, so RAG pipelines cannot assume retrieval will pay off at large model scale.","Factuality-style QA remains the clear exception, so retrieval investment is better targeted at factual recall than at broad knowledge or reasoning benchmarks.","Reranking the top 30 passages down to the top 5 does not close the gap, which suggests the bottleneck under mixed knowledge is not simply retriever ranking quality.","Prompt-based corpus routing, with or without chain-of-thought, is currently an unreliable strategy; retrieving from all sources at once is a safer default in the paper's experiments.","Oracle routing upper bounds are substantially higher than any prompt-based router, indicating concrete headroom for learned routing modules."],"supporting_citations":[{"why":"Supplies MassiveDS, the trillion-token multi-corpus datastore that defines the mixture-of-knowledge setting, along with document filtering and deduplication.","marker":"Shao et al. (2024)"},{"why":"Supplies the bge retriever and bge reranker models used for all retrieval and reranking experiments.","marker":"Chen et al. (2024)"},{"why":"Provides MMLU, one of the two general-knowledge QA benchmarks used to measure retrieval gains across domains.","marker":"Hendrycks et al. (2021)"},{"why":"Provides MMLU-Pro, the second general-knowledge benchmark, which also anchors the query-routing comparison.","marker":"Wang et al. (2024b)"},{"why":"Provides SimpleQA, the factuality benchmark where even large models show retrieval gains.","marker":"Wei et al. (2024)"},{"why":"Provides ARC Challenge, a scientific reasoning task included in the six-dataset evaluation.","marker":"Clark et al. (2018)"},{"why":"Provides CSBench, the computer-science benchmark with mixed answer formats.","marker":"Song et al. (2025)"},{"why":"Supplies the Llama-3.2-3B and Llama-3.1-8B backbone models whose gains anchor the small-model side of the scale comparison.","marker":"Grattafiori et al. (2024)"},{"why":"Supplies the Qwen3-4B/8B/32B backbones used both as generation models and as the routers in the routing experiments.","marker":"Yang et al. (2025)"},{"why":"Supplies the GPT-4o and GPT-4o-mini backbones whose near-zero or negative gains anchor the large-model side of the scale comparison.","marker":"Hurst et al. (2024)"}],"fun_headline_variants":["RAG helps small LLMs, stalls large ones in wild tests","Multi-corpus RAG: reranking and routing add little value","Retrieval gains shrink as LLMs scale—except for fact checks","Big language models see little benefit from RAG in the wild","RAG routing and rerankers fail to beat plain retrieval"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the specific retrieval pipeline the paper used—the bge-base-en-v1.5 retriever with five passages and zero-shot prompting on off-the-shelf instruction-tuned models—is representative enough of real RAG deployments that the observed scale-dependent benefit pattern is a property of retrieval augmentation itself rather than an artifact of this setup.","fun_headline_variants_meta":{"raw":{"variants":["RAG helps small LLMs, stalls large ones in wild tests","Multi-corpus RAG: reranking and routing add little value","Retrieval gains shrink as LLMs scale—except for fact checks","Big language models see little benefit from RAG in the wild","RAG routing and rerankers fail to beat plain retrieval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000602,"raw_usage":{"total_tokens":2802,"prompt_tokens":929,"completion_tokens":1873,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":1765}},"tokens_in":545,"tokens_out":1873,"duration_ms":13704,"temperature":1.0,"reasoning_tokens":1765,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:50:04.409733+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the six benchmarks with a substantially stronger retrieval stack (for example, a large reranker-based dense retriever, $k=20$ passages, and a RAG-fine-tuned backbone). If GPT-4o or Qwen3-32B then shows large relative gains, such as more than ten percent on MMLU-Pro or SciQ, the paper's scale-dependence claim collapses. Alternatively, train a supervised router on a few thousand labeled queries; if it matches the oracle upper bound on MMLU and MMLU-Pro, the claim that current LLMs cannot route effectively would need to be restricted to the zero-shot setting.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies MassiveDS, the trillion-token multi-corpus datastore that defines the mixture-of-knowledge setting, along with document filtering and deduplication."}],"review_version":1}