REVIEW 4 major objections 4 minor 26 references
SAGE: SLO-Aware Adaptive Retrieval for Production RAG Systems
T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read This paper claims that a learned per-query retrieval budget, trained by imitating an oracle's budget choices, lets production RAG systems meet P95 latency SLOs at roughly half the retrieval cost of the best fixed budget, with only a small…
desk verdict A plausible and well-motivated adaptive retrieval policy, but the headline SLO claims rest on load-free measurements and the method description is internally inconsistent; worth a serious revision, not a desk reject. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is an offline imitation-learning pipeline built on a per-query latency–quality frontier. For each training query, the paper runs the full hybrid RAG stack at every budget in a discrete set K={2,3,5,7,10,15,20,25,30}, records end-to-end latency L(q,k) and answer quality, and defines an oracle label k*(q): the smallest budget that meets the 5s P95 SLO while maximizing quality (or, if none qualifies, the one that violates least). A RandomForest classifier (100 trees, depth 10) is trained with cross-entropy loss to reproduce these labels from a feature vector $\varphi$(q) derived from a lightweight k=2 probe: score statistics, rank gaps, sparse–dense agreement, and lexical signals. A temperature on the classifier's logits is swept on a validation set to trade a little SLO compliance for higher EM. At inference the policy is a single argmax over budget classes taking under 1 ms, requiring no LLM calls.
What would settle it
Deploy SAGE and the best static budget side by side in a serving system with sustained concurrent traffic, measure P95 end-to-end latency and SLO compliance under the same 5-second target, and compare the compliance gap. If the gap between SAGE and static k=20 shrinks materially as load increases, or SAGE's compliance drops below target at high concurrency, the load-independent oracle assumption is falsified.
Extended reading notes
Core claim
The paper's central claim is that the fixed retrieval budget is the wrong control variable in production RAG. Because query difficulty varies, any global k either over-retrieves on easy queries (wasting latency and cost) or under-serves hard ones (hurting quality), and no static choice lands in the high-SLO, acceptable-EM operating region. SAGE establishes that a per-query budget can be predicted from a small set of retrieval-side features—score drop, rank gaps, and lexical signals from a k=2 probe—and that supervision can come entirely from an offline oracle that runs every training query at every budget, records latency and quality, and picks the smallest k meeting the P95 SLO while maximizing quality. The resulting RandomForest policy reaches 95% SLO compliance versus 30% for the best static budget, cuts P95 latency from 5.6s to 3.6s, halves relative retrieval cost, and loses 2 EM points; the same frozen policy is reported to improve SLO compliance by 45–52 points on two other datasets and on three other LLM families without retraining.
Load-bearing premise
The oracle labels are built from latencies measured one query at a time in isolation, and the learned policy assumes those single-run measurements predict real production latency under concurrent load; if queueing, batching, and inter-query interference dominate the P95 tail, then a per-query threshold policy trained on isolated latencies may not control the distributional SLO.
Editorial extensions
If this is right
- Operators can replace a single global k with a policy that raises SLO compliance from 30% to 95% while halving retrieval cost, at a 2-point EM cost.
- A policy trained once on Natural Questions transfers to HotpotQA and UnSeenTimeQA and to four LLM families without retraining, giving +45–52 point SLO improvements at unchanged EM.
- Because the policy adds under 1 ms and no LLM calls, it can be deployed as a stateless service alongside the retriever and rolled back independently of model weights or prompts.
- The average budget drops to 9.8, translating to a 51% retrieval-cost reduction; the paper's cost model puts this at about $132,000 per year at 10M queries per day.
- Calibration on a held-out validation set lets operators trade a small amount of SLO compliance for higher EM.
Reading between the lines
- Editorial extension: the same retrieval-side feature set could tune other retrieval-stage knobs—reranker depth, number of iterative retrieval rounds, or fusion weights—turning SAGE's decision rule into a general retrieval-resource allocator.
- Editorial extension: because the oracle uses isolated latencies, a production deployment would likely need to re-calibrate the temperature under changing load; the paper's calibration is a one-time validation-set step, not an online controller.
- Editorial extension: the cost model implies savings scale linearly with query volume, so the claimed 51% cost reduction at 10M queries/day depends on the learned budget distribution remaining stable as traffic and corpus change over time.
- Editorial extension: a direct testable follow-up is whether the score-drop and rank-gap features retain their signal on out-of-domain corpora (for example, legal or biomedical text) without retraining, since the paper only evaluates on Wikipedia-based QA datasets.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SAGE, a learned adaptive retrieval policy that selects a per-query retrieval budget k from a discrete set, using lightweight retrieval-side features and an offline imitation-learning objective. The labels come from an oracle that, for each query, sweeps all budgets and picks the smallest k satisfying a 5s per-query latency threshold while preserving quality. On Natural Questions, the authors report that SAGE achieves 95% SLO compliance and 22% EM, versus 30% compliance and 24% EM for static k=20, with roughly half the retrieval cost, and that a single NQ-trained policy generalizes to HotpotQA, UnSeenTimeQA, and four LLM families without retraining. The paper frames this as a production-relevant solution to the fixed-budget problem in RAG systems.
Significance. If the production claim were established, SAGE would be a practical and valuable contribution: it adds negligible inference overhead, makes no additional LLM calls, uses interpretable retrieval features, and the cross-model generalization is a genuine operational convenience. The problem formulation as a constrained optimization over SLO compliance, quality, and cost is clean, and the evaluation is not circular: oracle labels are used only for supervision, while the reported compliance and EM are direct held-out measurements. However, the production significance currently rests on a load-free latency measurement and on a baseline choice that obscures the trade-off, so the significance is conditional on the major revisions below.
major comments (4)
- [Section III.A, Section IV.C, Section VI.A (Table I)] The reported SLO compliance is computed from per-query end-to-end latency L(q,k), which Section III.A defines from request arrival to LLM completion and Section IV.C uses to choose the smallest k satisfying the 5s threshold. No concurrency level, arrival process, batching policy, or queueing model is described anywhere in the manuscript, so the 95% compliance figure is a per-query threshold-satisfaction rate under isolated execution, not a P95 tail-latency SLO under production load. Queueing and inter-query interference are precisely what dominate tail latency in the production RAG setting asserted in Sections I and VI.D, so the central production claim is currently unsupported. Please add a concurrent-load evaluation with varying arrival rates and batch sizes, or explicitly restrict the claims to single-request latency control and revise the title and abstract accordingly.
- [Abstract and Table I] The abstract's '95% versus 30%' comparison implicitly treats k=20 as the best static baseline, but Table I shows static k=2 already achieves 95% SLO compliance (2.1s P95) under the same metric, and static k=10 achieves the same 22% EM as SAGE at 45% compliance. The value of SAGE is therefore better stated as moving from 30% to 95% compliance at roughly constant EM relative to k=20, while the more natural SLO-satisfying static comparison is k=2, against which SAGE raises EM from 11% to 22% at higher cost (49% vs 10%). Please report the full static frontier and justify the baseline selection explicitly.
- [Section VI.A, Figures 4-5] The NQ test set is only 334 queries, and no confidence intervals or significance tests are reported anywhere. The headline 2-point EM difference between SAGE (22%) and static k=20 (24%) corresponds to roughly seven queries and is well within sampling noise, and the cross-dataset and cross-model 'no quality degradation' claims are made without per-model numbers or error bars. Please provide bootstrap intervals or other uncertainty estimates for all headline percentages, and report the actual per-dataset and per-model values behind Figures 4 and 5.
- [Section IV.C] The oracle policy that generates the imitation labels is never evaluated on the test set, so the reader cannot see how much of the achievable latency-quality frontier is lost by behavior cloning. Report oracle SLO compliance, EM, average k, and oracle-SAGE agreement on the NQ test set; without this ceiling, the 'near-optimal' interpretation of Eq. (2) is an assumption rather than a demonstrated property.
minor comments (4)
- [Section V.E] The sentence 'details are omitted for space' removes training hyperparameters, the calibration temperature grid, and early-stopping criteria; these are needed for reproducibility and should be supplied in an appendix or supplemental material.
- [Section VI.B (Table II)] The '– Hybrid (dense only)' ablation row reports 0.8s P95 and 15% EM, which is qualitatively different from the full system; clarify what retrieval stack this ablation uses and why dense-only latency is so much lower than the hybrid configuration.
- [Section III.B and Section IV.D] The calibration step selects a temperature to maximize validation EM subject to SLO compliance, but the target α in Eq. (2) is never given a numerical value; state the numeric α used in the experiments.
- [Figures 4 and 5] The figures would be much easier to verify if accompanied by tables of exact SLO compliance, EM, and cost values for each dataset and model family.
Circularity Check
No significant circularity: reported SLO compliance and EM are measured on a held-out test set, and the oracle only defines the supervision target.
full rationale
The paper's derivation chain is self-contained against the possibility of circularity. The oracle in Section IV.C selects, for each training query, the smallest budget satisfying the 5s latency threshold using measured L(q,k), and the policy is trained by imitation; but the evaluation in Section VI.A runs the full pipeline on a held-out test set and computes SLO compliance, P95 latency, and EM from those test measurements. The calibration temperature in Section IV.D is selected on a held-out validation set, not on the test set, so the reported 95% SLO compliance is not a fitted value. No load-bearing argument rests on a self-citation: the cited prior work (Ross et al., Dean and Barroso, etc.) supplies standard methodology and background, not the paper's central claims. The closest concern is that the oracle labels are constructed to satisfy the same SLO metric used in evaluation; that makes high SLO compliance a property of the supervision target rather than a surprise, but it is not circular because the test evaluation is statistically independent of the labels and the policy is not evaluated on the training queries. The absence of concurrency/queueing in the latency model is a real validity limitation for the production claim, but it is a correctness risk, not a circularity: the measured SLO compliance is what it claims to be under the stated load-free definition, and nothing in that definition is imported from the conclusion.
Assumptions & free parameters
free parameters (2)
- calibration temperature =
not reported
- policy hyperparameters =
not reported
assumptions (3)
- domain assumption Per-query end-to-end latency L(q,k) is a stable, load-independent quantity that can be measured once per budget and used to derive oracle decisions.
- ad hoc to paper The feature vector phi(q) from a k=2 probe retrieval (score drop, top1 score, rank gaps, lexical signals) is sufficient to predict the oracle-optimal budget.
- ad hoc to paper Imitating the smallest-k-satisfying-SLO oracle via behavior cloning yields a policy that optimizes the constrained objective of Eq. (2).
Cite this review
Pith. "Pith review of SAGE: SLO-Aware Adaptive Retrieval for Production RAG Systems." pith.science (2026). https://pith.science/paper/NUSHDWQL
@misc{pith2026260808237,
author = {Pith},
title = {Pith review of: SAGE: SLO-Aware Adaptive Retrieval for Production RAG Systems},
year = {2026},
howpublished = {\url{https://pith.science/paper/NUSHDWQL}},
note = {Machine review of arXiv:2608.08237}
}
read the original abstract
Retrieval-Augmented Generation (RAG) systems in production operate under strict service level objectives (SLOs) on tail latency and infrastructure cost. However, standard retrieval pipelines rely on fixed retrieval budgets that ignore query difficulty, over-retrieving for easy queries and under-serving hard ones, forcing operators to trade answer quality against SLO compliance. This paper proposes SAGE, a learned SLO-aware adaptive retrieval policy that dynamically selects the number of passages k per query. SAGE uses lightweight features derived from initial retrieval (e.g., score distributions, rank gaps, lexical signals) and is trained offline via imitation learning from an oracle that approximates optimal latency-quality trade-offs. At inference, it adds no LLM calls and minimal overhead. On Natural Questions, under a 5s P95 latency SLO, SAGE achieves 95% SLO compliance versus 30% for the best static baseline (k=20), reduces P95 latency by 36% and retrieval cost by 51% with only 2 percentage points Exact Match (EM) loss. A single policy trained on Natural Questions generalizes across HotpotQA, UnSeenTimeQA, and four LLM families (Llama, Qwen, Mistral, Gemma), consistently yielding +45-52 point SLO improvements without quality degradation.
Figures
Reference graph
Works this paper leans on
-
[1]
Retrieval-augmented generation for knowledge-intensive NLP tasks,
P. Lewis, E. Perez, A. Piktuset al., “Retrieval-augmented generation for knowledge-intensive NLP tasks,” inAdvances in Neural Information Processing Systems (NeurIPS), 2020. [Online]. Available: https: //doi.org/10.48550/arXiv.2005.11401
-
[2]
REALM: Retrieval-augmented language model pre-training,
K. Guu, K. Lee, Z. Tunget al., “REALM: Retrieval-augmented language model pre-training,” inProceedings of the 37th International Conference on Machine Learning (ICML), 2020. [Online]. Available: https://doi.org/10.48550/arXiv.2002.08909
-
[3]
Dense passage retrieval for open- domain question answering,
V . Karpukhin, B. Oguz, S. Minet al., “Dense passage retrieval for open- domain question answering,” inProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020. [Online]. Available: https://doi.org/10.48550/arXiv.2004.04906
-
[4]
Retrieval-augmented generation for large language models: A survey,
Y . Gao, Y . Xiong, X. Gaoet al., “Retrieval-augmented generation for large language models: A survey,” 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2312.10997
-
[5]
J. Dean and L. A. Barroso, “The tail at scale,”Communications of the ACM, vol. 56, no. 2, pp. 74–80, 2013. [Online]. Available: https://doi.org/10.1145/2408776.2408794
arXiv 2013
-
[6]
Active retrieval augmented generation,
Z. Jiang, F. F. Xu, L. Gaoet al., “Active retrieval augmented generation,” inProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2305.06983
-
[7]
Self-RAG: Learning to retrieve, generate, and critique through self-reflection,
A. Asai, Z. Wu, Y . Wanget al., “Self-RAG: Learning to retrieve, generate, and critique through self-reflection,” inProceedings of the International Conference on Learning Representations (ICLR), 2024. [Online]. Available: https://openreview.net/forumid=hSyW5go0v8
work page 2024
-
[8]
Efficient context selection for long-context qa: No tuning, no iteration, just adaptive- k,
C. Taguchi, S. Maekawa, and N. Bhutani, “Efficient context selection for long-context qa: No tuning, no iteration, just adaptive- k,”arXiv preprint arXiv:2506.08479, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2506.08479
Show all 26 references
-
[9]
Stop-RAG: Value-based retrieval control for iterative rag,
J.-Y . L. Jaewan Park, Solbee Cho, “Stop-RAG: Value-based retrieval control for iterative rag,”arXiv preprint arXiv:2510.14337, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2510.14337
2025 doi
-
[10]
Leveraging passage retrieval with generative models for open domain question answering,
G. Izacard and E. Grave, “Leveraging passage retrieval with generative models for open domain question answering,” inProceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, 2021, pp. 874–880. [Online]. Available: https://acl...
2021
-
[11]
Fido: Fusion-in- decoder optimized for stronger performance and faster inference,
M. de Jong, Y . Zemlyanskiy, J. Ainslieet al., “Fido: Fusion-in- decoder optimized for stronger performance and faster inference,” arXiv preprint arXiv:2212.08153, 2023. [Online]. Available: https: //doi.org/10.48550/arXiv.2212.08153
-
[12]
The probabilistic relevance framework: BM25 and beyond,
S. Robertson and H. Zaragoza, “The probabilistic relevance framework: BM25 and beyond,”Foundations and Trends in Information Retrieval, vol. 3, no. 4, pp. 333–389, 2009. [Online]. Available: https: //doi.org/10.1561/1500000019
2009 doi
-
[13]
Reciprocal rank fusion outperforms condorcet and individual rank learning methods,
G. V . Cormack, C. L. A. Clarke, and S. B ¨uttcher, “Reciprocal rank fusion outperforms condorcet and individual rank learning methods,” inProceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, 2009, pp. 758–759. [Onlin...
2009
-
[14]
Interleaving retrieval with chain-of-thought reasoning for knowledge- intensive multi-step questions,
H. Trivedi, N. Balasubramanian, T. Khot, and A. Sabharwal, “Interleaving retrieval with chain-of-thought reasoning for knowledge- intensive multi-step questions,” inProceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023. [Online]. Availab...
-
[15]
DRAGIN: Dynamic retrieval augmented generation based on the information needs of large language models,
W. Su, Y . Tang, Q. Aiet al., “DRAGIN: Dynamic retrieval augmented generation based on the information needs of large language models,”arXiv preprint arXiv:2403.10081, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2403.10081
-
[16]
Seakr: Self-aware knowledge retrieval for adaptive retrieval augmented generation,
Z. Yao, W. Qi, L. Panet al., “Seakr: Self-aware knowledge retrieval for adaptive retrieval augmented generation,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, 2025. [Online]. Available: https://doi.org/10.18653/v1/2025.acl-long.1312
2025 doi
-
[17]
Efficient memory management for large language model serving with pagedattention,
W. Kwon, Z. Li, S. Zhuanget al., “Efficient memory management for large language model serving with pagedattention,” inProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles (SOSP), 2023. [Online]. Available: https://doi.org/10.1145/3600006. 3613165
2023 doi
-
[18]
Slo-power: SLO and power- aware elastic scaling for web services,
M. Savasci, A. Souza, L. Wuet al., “Slo-power: SLO and power- aware elastic scaling for web services,” inProceedings of the 24th IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing (CCGrid), 2024, pp. 136–147. [Online]. Available: https://doi.org/10.1109/...
2024
-
[19]
JITServe: SLO-aware LLM serving with imprecise request information,
W. Zhang, Z. Wu, Y . Muet al., “JITServe: SLO-aware LLM serving with imprecise request information,”arXiv preprint arXiv:2504.20068,
-
[20]
A reduction of imitation learning and structured prediction to no-regret online learning,
S. Ross, G. J. Gordon, and D. Bagnell, “A reduction of imitation learning and structured prediction to no-regret online learning,” inProceedings of the 14th International Conference on Artificial Intelligence and Statistics (AISTATS), ser. JMLR Workshop and Conference Proceedi...
2011
-
[21]
Imitation learning: A survey of learning methods,
A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne, “Imitation learning: A survey of learning methods,”ACM Computing Surveys, vol. 50, no. 2, pp. 21:1–21:35, 2017. [Online]. Available: https://doi.org/10.1145/3054912
2017 doi
-
[22]
Natural questions: A benchmark for question answering research,
T. Kwiatkowski, J. Palomaki, O. Redfieldet al., “Natural questions: A benchmark for question answering research,”Transactions of the Association for Computational Linguistics, vol. 7, pp. 453–466, 2019. [Online]. Available: https://doi.org/10.1162/tacl a 00276
2019 doi
-
[23]
HOTPOTQA: A dataset for diverse, explainable multi-hop question answering,
Z. Yang, P. Qi, S. Zhanget al., “HOTPOTQA: A dataset for diverse, explainable multi-hop question answering,” inProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, pp. 2369–2380. [Online]. Available: https://doi.org/10.18653/v1/ D18-1259
2018 doi
-
[24]
UnSeenTimeQA: Time-sensitive Question-Answering beyond LLMs’ Memorization,
M. N. Uddin, A. Saeidi, D. Handaet al., “UnSeenTimeQA: Time-sensitive Question-Answering beyond LLMs’ Memorization,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, 2025. [Online]. Available: https: //doi.org/10.18653/v1/2025.acl-long.94
2025 doi
-
[25]
M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self- knowledge distillation,
J. Chen, S. Xiao, P. Zhanget al., “M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self- knowledge distillation,”arXiv preprint arXiv:2402.03216, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2402.03216
-
[2025]
Available: https://doi.org/10.48550/arXiv.2504.20068
[Online]. Available: https://doi.org/10.48550/arXiv.2504.20068
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.