Pith. sign in

REVIEW 8 cited by

Inference Scaling for Long-Context Retrieval Augmented Generation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.04343 v2 pith:LAVSSA4M submitted 2024-10-06 cs.CL

Inference Scaling for Long-Context Retrieval Augmented Generation

Zhenrui Yue , Honglei Zhuang , Aijun Bai , Kai Hui , Rolf Jagerman , Hansi Zeng , Zhen Qin , Dong Wang
show 2 more authors
Xuanhui Wang Michael Bendersky
This is my paper
classification cs.CL
keywords inferencecomputationscalingperformancecomputegenerationincreasingknowledge
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

The scaling of inference computation has unlocked the potential of long-context large language models (LLMs) across diverse settings. For knowledge-intensive tasks, the increased compute is often allocated to incorporate more external knowledge. However, without effectively utilizing such knowledge, solely expanding context does not always enhance performance. In this work, we investigate inference scaling for retrieval augmented generation (RAG), exploring the combination of multiple strategies beyond simply increasing the quantity of knowledge, including in-context learning and iterative prompting. These strategies provide additional flexibility to scale test-time computation (e.g., by increasing retrieved documents or generation steps), thereby enhancing LLMs' ability to effectively acquire and utilize contextual information. We address two key questions: (1) How does RAG performance benefit from the scaling of inference computation when optimally configured? (2) Can we predict the optimal test-time compute allocation for a given budget by modeling the relationship between RAG performance and inference parameters? Our observations reveal that increasing inference computation leads to nearly linear gains in RAG performance when optimally allocated, a relationship we describe as the inference scaling laws for RAG. Building on this, we further develop the computation allocation model to estimate RAG performance across different inference configurations. The model predicts optimal inference parameters under various computation constraints, which align closely with the experimental results. By applying these optimal configurations, we demonstrate that scaling inference compute on long-context LLMs achieves up to 58.9% gains on benchmark datasets compared to standard RAG.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Latent Abstraction for Retrieval-Augmented Generation

    cs.CL 2026-04 unverdicted novelty 7.0

    LAnR unifies retrieval-augmented generation inside a single LLM by deriving dense retrieval vectors from a [PRED] token's hidden states and using entropy to adaptively stop retrieval, outperforming prior RAG on six QA...

  2. Retrieval Augmented Conversational Recommendation with Reinforcement Learning

    cs.IR 2026-04 unverdicted novelty 7.0

    RAR retrieves candidate items from a 300k-movie corpus then uses LLM generation with RL feedback to produce context-aware recommendations that outperform baselines on benchmarks.

  3. MemSearcher: Training LLMs to Reason, Search and Manage Memory via End-to-End Reinforcement Learning

    cs.CL 2025-11 unverdicted novelty 7.0

    MemSearcher trains LLMs to manage compact memory in multi-turn searches via multi-context GRPO for end-to-end RL, outperforming ReAct-style baselines with stable token counts.

  4. Retrieving a Set, Not Independent Passages: Set-Level Compatibility Learning for Efficient Set Exploration

    cs.IR 2026-07 conditional novelty 6.5

    Set-level margin training that ranks complete gold-compatible passage sets above incomplete or noisy ones, implemented with ParaSet exploration and SetCE reranking, improves multi-hop retrieval and QA and complements ...

  5. Efficient Multi-round LLM Inference over Disaggregated Serving

    cs.DC 2026-02 conditional novelty 6.0

    AMPD adaptively routes incremental prefill tasks between prefill and decode workers and reorders queued prefill jobs to improve SLO attainment for multi-round LLM inference under prefill-decode disaggregation.

  6. DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments

    cs.AI 2025-04 conditional novelty 6.0

    End-to-end RL in authentic web environments produces LLM research agents that outperform prompt-engineering and RAG-based baselines by up to 28.9 and 7.2 points respectively while exhibiting emergent planning, cross-v...

  7. Two AI Metrics Diverged: Will it Make All the Difference?

    cs.AI 2026-07 unverdicted novelty 5.0

    Bounded performance metrics always favor convergence of AI capabilities to meek models while unbounded metrics allow frontier models to maintain leads indefinitely, with policy implications for capability concentration.

  8. Advancing Multi-Agent RAG Systems with Minimalist Reinforcement Learning

    cs.CL 2025-05 unverdicted novelty 5.0

    Mujica-MyGo decomposes multi-turn RAG interactions via multi-agent workflows and applies minimalist policy gradient optimization to improve performance on QA benchmarks while avoiding long-context problems.