{"id":"69f1576b-5e77-4cc0-b909-6232ded69f49","arxiv_id":"2608.08237","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"SAGE uses offline imitation learning from a latency-quality oracle to choose per-query retrieval budgets, reporting 95% SLO compliance and 51% lower retrieval cost on Natural Questions with 2 points of EM loss.","lead":"SAGE is a learned system that picks how many passages a RAG pipeline retrieves for each query, choosing small budgets for easy questions and larger ones for hard questions. Under a 5-second tail-latency target it kept 95% of queries within the target and halved retrieval cost, with only a small drop in answer quality.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central production claim is untested: SLO compliance is measured on isolated per-query latencies, not under contention, so the reported 95% figure may not correspond to a real P95 SLO in deployment.","rationale":"The paper's central contribution is a production-oriented SLO claim, but its measurement protocol never exercises the system under load. Section IV.C uses per-query latencies measured in isolation to build the oracle, and Section VI evaluates SLO compliance on the same kind of isolated runs. The reader's weakest assumption identifies exactly this load-independence issue, and I agree with it. This is more load-bearing than the RandomForest-versus-Adam/logits inconsistency because even a correctly implemented policy would not address queueing and batching effects. It also matters more than the small 334-query test set: the 30% versus 95% compliance gap is far beyond sampling noise, but it is a gap on a metric that may not correspond to the claimed production P95. Table I itself shows that static k=2 achieves 95% compliance in the load-free setting, which suggests the 5s threshold is not binding for small budgets and that the headline comparison to k=20 is not a comparison to a production-realistic operating point. If the proposed load test shows SAGE preserves its advantage under contention, the concern would be resolved; until then, the production claim should be treated as conditional on load validation. The critique is about the experimental protocol, not about the authors' integrity, and the paper deserves credit for a clearly stated offline imitation-learning framework and a broad evaluation across datasets and model families.","tokens_in":9083,"tokens_out":5266,"duration_ms":52935,"concrete_test":"Reproduce Table I under a controlled load test: serve the NQ test queries through vLLM with a fixed concurrency (e.g., 8 in-flight requests) and a Poisson arrival rate that drives the static k=20 configuration to roughly 50% GPU utilization, recording end-to-end latency including queueing. Recompute SLO compliance and P95 for SAGE, static k=2, k=10, and k=20 under this load. Additionally, recompute the oracle labels using loaded latencies L_load(q,k) instead of isolated L(q,k), and measure the fraction of queries whose isolated-optimal budget differs from the loaded-optimal budget. If SAGE's loaded SLO compliance is not appreciably above static k=10, or if the two oracle labelings diverge on more than roughly 20% of queries, the production claim in the abstract is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III.A defines L(q,k) as end-to-end latency measured from request arrival to LLM completion, and Section IV.C constructs the oracle by selecting, for each query, the smallest k satisfying the 5s threshold using these isolated L(q,k) values. Section VI then reports SLO compliance and P95 latency from the same load-free evaluation; no concurrency level, arrival process, or queueing model is described anywhere in the manuscript. In a production RAG service, queueing, batching, and inter-query interference dominate the tail, so a policy trained to meet a per-query threshold on isolated runs has no mechanism to control the distributional P95 SLO. Table I makes the gap concrete: static k=2 already achieves 95% SLO compliance and 2.1s P95 under this measurement, while static k=20 drops to 30%. The headline '95% versus 30%' compares SAGE against a slow static configuration under a metric that does not measure production tail latency. Because the oracle and the evaluation share the same load-free assumption, the central claim that SAGE meets the P95 SLO in production is currently unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SAGE, a learned adaptive retrieval policy that selects a per-query retrieval budget k from a discrete set, using lightweight retrieval-side features and an offline imitation-learning objective. The labels come from an oracle that, for each query, sweeps all budgets and picks the smallest k satisfying a 5s per-query latency threshold while preserving quality. On Natural Questions, the authors report that SAGE achieves 95% SLO compliance and 22% EM, versus 30% compliance and 24% EM for static k=20, with roughly half the retrieval cost, and that a single NQ-trained policy generalizes to HotpotQA, UnSeenTimeQA, and four LLM families without retraining. The paper frames this as a production-relevant solution to the fixed-budget problem in RAG systems.","tokens_in":9376,"tokens_out":5466,"duration_ms":51953,"significance":"If the production claim were established, SAGE would be a practical and valuable contribution: it adds negligible inference overhead, makes no additional LLM calls, uses interpretable retrieval features, and the cross-model generalization is a genuine operational convenience. The problem formulation as a constrained optimization over SLO compliance, quality, and cost is clean, and the evaluation is not circular: oracle labels are used only for supervision, while the reported compliance and EM are direct held-out measurements. However, the production significance currently rests on a load-free latency measurement and on a baseline choice that obscures the trade-off, so the significance is conditional on the major revisions below.","major_comments":[{"comment":"The reported SLO compliance is computed from per-query end-to-end latency L(q,k), which Section III.A defines from request arrival to LLM completion and Section IV.C uses to choose the smallest k satisfying the 5s threshold. No concurrency level, arrival process, batching policy, or queueing model is described anywhere in the manuscript, so the 95% compliance figure is a per-query threshold-satisfaction rate under isolated execution, not a P95 tail-latency SLO under production load. Queueing and inter-query interference are precisely what dominate tail latency in the production RAG setting asserted in Sections I and VI.D, so the central production claim is currently unsupported. Please add a concurrent-load evaluation with varying arrival rates and batch sizes, or explicitly restrict the claims to single-request latency control and revise the title and abstract accordingly.","section":"Section III.A, Section IV.C, Section VI.A (Table I)"},{"comment":"The abstract's '95% versus 30%' comparison implicitly treats k=20 as the best static baseline, but Table I shows static k=2 already achieves 95% SLO compliance (2.1s P95) under the same metric, and static k=10 achieves the same 22% EM as SAGE at 45% compliance. The value of SAGE is therefore better stated as moving from 30% to 95% compliance at roughly constant EM relative to k=20, while the more natural SLO-satisfying static comparison is k=2, against which SAGE raises EM from 11% to 22% at higher cost (49% vs 10%). Please report the full static frontier and justify the baseline selection explicitly.","section":"Abstract and Table I"},{"comment":"The NQ test set is only 334 queries, and no confidence intervals or significance tests are reported anywhere. The headline 2-point EM difference between SAGE (22%) and static k=20 (24%) corresponds to roughly seven queries and is well within sampling noise, and the cross-dataset and cross-model 'no quality degradation' claims are made without per-model numbers or error bars. Please provide bootstrap intervals or other uncertainty estimates for all headline percentages, and report the actual per-dataset and per-model values behind Figures 4 and 5.","section":"Section VI.A, Figures 4-5"},{"comment":"The oracle policy that generates the imitation labels is never evaluated on the test set, so the reader cannot see how much of the achievable latency-quality frontier is lost by behavior cloning. Report oracle SLO compliance, EM, average k, and oracle-SAGE agreement on the NQ test set; without this ceiling, the 'near-optimal' interpretation of Eq. (2) is an assumption rather than a demonstrated property.","section":"Section IV.C"}],"minor_comments":[{"comment":"The sentence 'details are omitted for space' removes training hyperparameters, the calibration temperature grid, and early-stopping criteria; these are needed for reproducibility and should be supplied in an appendix or supplemental material.","section":"Section V.E"},{"comment":"The '– Hybrid (dense only)' ablation row reports 0.8s P95 and 15% EM, which is qualitatively different from the full system; clarify what retrieval stack this ablation uses and why dense-only latency is so much lower than the hybrid configuration.","section":"Section VI.B (Table II)"},{"comment":"The calibration step selects a temperature to maximize validation EM subject to SLO compliance, but the target α in Eq. (2) is never given a numerical value; state the numeric α used in the experiments.","section":"Section III.B and Section IV.D"},{"comment":"The figures would be much easier to verify if accompanied by tables of exact SLO compliance, EM, and cost values for each dataset and model family.","section":"Figures 4 and 5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a reasonable systems/ML contribution if reframed as a study of per-query latency-quality trade-offs in a load-free setting. The main risk is that the production framing overreaches the measurement. I would encourage a revision that either adds a concurrent-load experiment or softens the production claims, and that corrects the baseline-selection narrative around Table I."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper goes after a real operational problem: fixed-k retrieval in production RAG wastes latency and cost on easy queries while starving hard ones. The core idea, a lightweight classifier that maps retrieval-side features to a per-query budget k, trained by imitating an oracle built from budget sweeps, is clean and sensible. The cross-dataset and cross-LLM generalization is genuinely useful if it holds, and the evaluation is extensive in breadth (174 configurations, 58k queries), even if the primary NQ test set is only 334 queries. The oracle is used only for supervision, not evaluation, so there is no circularity in the main result.\n\nThe soft spots are real and one is load-bearing. The SLO compliance and P95 numbers are measured on isolated per-query latencies. Section III.A defines L(q,k) from request arrival to LLM completion, and Section IV.C builds the oracle from these single-run measurements; Section VI then reports the same load-free metric as if it were a production P95 SLO. No concurrency level, arrival process, or queueing model appears anywhere. In a real deployment, queueing and batching dominate the tail, so a policy trained to meet a per-query threshold on isolated runs has no mechanism to control the distributional SLO. The stress-test note is correct: the headline '95% versus 30%' compares against a slow static configuration under a metric that does not measure production tail latency.\n\nThe abstract also mischaracterizes the baseline. It calls k=20 the 'best static baseline' for the 30% SLO comparison, but Table I shows static k=2 already achieves 95% SLO compliance (with 11% EM). The fair comparison is against the static frontier, and SAGE does beat k=10 on both SLO and EM, so the improvement is real, but the framing is cherry-picked.\n\nThe method description has a concrete internal inconsistency: Section IV.B says the policy is a RandomForest classifier, but Section IV.D describes calibrating a temperature on policy logits and Section V.E says the policy is optimized with Adam. That is not a minor typo; it makes the experimental setup ambiguous. Also missing: oracle ceiling, error bars on the 334-query test set, and the exact hyperparameters.\n\nThis paper deserves a serious referee, but not acceptance as is. A revised version that measures latency under realistic contention (or explicitly scopes the claims to the load-free setting), corrects the baseline framing, fixes the method description, and reports uncertainty would be a solid contribution to the production RAG systems literature. I would not cite the current version in my own work, but I would read the revision.","headline":"A plausible and well-motivated adaptive retrieval policy, but the headline SLO claims rest on load-free measurements and the method description is internally inconsistent; worth a serious revision, not a desk reject.","tokens_in":9843,"tokens_out":2254,"would_cite":false,"duration_ms":21966,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a learned per-query retrieval budget, trained by imitating an oracle's budget choices, lets production RAG systems meet P95 latency SLOs at roughly half the retrieval cost of the best fixed budget, with only a small…","keywords":["SLO-aware retrieval","adaptive retrieval","RAG","tail latency","imitation learning","retrieval budget","service level objective","question answering"],"falsifier":"Deploy SAGE and the best static budget side by side in a serving system with sustained concurrent traffic, measure P95 end-to-end latency and SLO compliance under the same 5-second target, and compare the compliance gap. If the gap between SAGE and static k=20 shrinks materially as load increases, or SAGE's compliance drops below target at high concurrency, the load-independent oracle assumption is falsified.","tokens_in":8895,"feed_emoji":"⚡","tokens_out":5846,"duration_ms":50662,"temperature":0.7,"pith_summary":"SAGE claims that a RAG system can meet strict tail-latency service-level objectives by choosing how many passages to retrieve per query instead of using a fixed budget for every query. It trains a lightweight classifier offline on decisions made by an oracle that has measured, for each training query, the latency and answer quality at every budget size; at runtime the classifier reads only retrieval-side scores and rank statistics, so it adds no LLM calls. On Natural Questions under a 5-second P95 SLO, the paper reports SLO compliance rising from 30% with the best static budget to 95%, with retrieval cost roughly halved and a 2-point Exact Match drop. The same policy, trained once, is claimed to transfer to multi-hop and time-sensitive QA datasets and to four LLM families without retraining. If true, this gives production operators a way to satisfy latency SLOs while cutting infrastructure cost, without changing the LLM or the retriever.","feed_headline":"Adaptive retrieval lifts RAG SLO compliance from 30% to 95%","feed_subtitle":"One learned policy halves retrieval cost with a 2-point accuracy dip, and transfers across models and datasets.","key_machinery":"The load-bearing mechanism is an offline imitation-learning pipeline built on a per-query latency–quality frontier. For each training query, the paper runs the full hybrid RAG stack at every budget in a discrete set K={2,3,5,7,10,15,20,25,30}, records end-to-end latency L(q,k) and answer quality, and defines an oracle label k*(q): the smallest budget that meets the 5s P95 SLO while maximizing quality (or, if none qualifies, the one that violates least). A RandomForest classifier (100 trees, depth 10) is trained with cross-entropy loss to reproduce these labels from a feature vector $\\varphi$(q) derived from a lightweight k=2 probe: score statistics, rank gaps, sparse–dense agreement, and lexical signals. A temperature on the classifier's logits is swept on a validation set to trade a little SLO compliance for higher EM. At inference the policy is a single argmax over budget classes taking under 1 ms, requiring no LLM calls.","core_discovery":"The paper's central claim is that the fixed retrieval budget is the wrong control variable in production RAG. Because query difficulty varies, any global k either over-retrieves on easy queries (wasting latency and cost) or under-serves hard ones (hurting quality), and no static choice lands in the high-SLO, acceptable-EM operating region. SAGE establishes that a per-query budget can be predicted from a small set of retrieval-side features—score drop, rank gaps, and lexical signals from a k=2 probe—and that supervision can come entirely from an offline oracle that runs every training query at every budget, records latency and quality, and picks the smallest k meeting the P95 SLO while maximizing quality. The resulting RandomForest policy reaches 95% SLO compliance versus 30% for the best static budget, cuts P95 latency from 5.6s to 3.6s, halves relative retrieval cost, and loses 2 EM points; the same frozen policy is reported to improve SLO compliance by 45–52 points on two other datasets and on three other LLM families without retraining.","pith_inferences":["Editorial extension: the same retrieval-side feature set could tune other retrieval-stage knobs—reranker depth, number of iterative retrieval rounds, or fusion weights—turning SAGE's decision rule into a general retrieval-resource allocator.","Editorial extension: because the oracle uses isolated latencies, a production deployment would likely need to re-calibrate the temperature under changing load; the paper's calibration is a one-time validation-set step, not an online controller.","Editorial extension: the cost model implies savings scale linearly with query volume, so the claimed 51% cost reduction at 10M queries/day depends on the learned budget distribution remaining stable as traffic and corpus change over time.","Editorial extension: a direct testable follow-up is whether the score-drop and rank-gap features retain their signal on out-of-domain corpora (for example, legal or biomedical text) without retraining, since the paper only evaluates on Wikipedia-based QA datasets."],"forward_implications":["Operators can replace a single global k with a policy that raises SLO compliance from 30% to 95% while halving retrieval cost, at a 2-point EM cost.","A policy trained once on Natural Questions transfers to HotpotQA and UnSeenTimeQA and to four LLM families without retraining, giving +45–52 point SLO improvements at unchanged EM.","Because the policy adds under 1 ms and no LLM calls, it can be deployed as a stateless service alongside the retriever and rolled back independently of model weights or prompts.","The average budget drops to 9.8, translating to a 51% retrieval-cost reduction; the paper's cost model puts this at about $132,000 per year at 10M queries per day.","Calibration on a held-out validation set lets operators trade a small amount of SLO compliance for higher EM."],"supporting_citations":[{"why":"Defines retrieval-augmented generation, the substrate that SAGE controls.","marker":"[1]"},{"why":"Establishes tail-latency SLOs as the production constraint that SAGE targets.","marker":"[5]"},{"why":"Supplies the BM25 sparse retriever in the hybrid stack.","marker":"[12]"},{"why":"Reciprocal Rank Fusion combines sparse and dense rankings to form the feature substrate SAGE reads.","marker":"[13]"},{"why":"The serving layer used in the experiments, which SAGE claims to complement rather than replace.","marker":"[17]"},{"why":"Provides the imitation-learning reduction that grounds SAGE's offline behavior-cloning approach.","marker":"[20]"},{"why":"Natural Questions is the training and primary evaluation corpus.","marker":"[22]"},{"why":"HotpotQA is the multi-hop generalization target.","marker":"[23]"},{"why":"UnSeenTimeQA is the temporal-QA generalization target.","marker":"[24]"},{"why":"Supplies the dense-retriever embeddings used in the hybrid retrieval pipeline.","marker":"[25]"}],"fun_headline_variants":["Adaptive RAG retrieval: SLO compliance from 30% to 95%","Per-query retrieval budgets lift RAG SLOs by 65 points","SAGE: Adaptive retrieval hits 95% SLO with 36% lower latency","RAG retrieval that adapts to query difficulty: 95% SLO, 51% less cost","One learned policy for RAG: 95% SLO compliance, 2 EM drop"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The oracle labels are built from latencies measured one query at a time in isolation, and the learned policy assumes those single-run measurements predict real production latency under concurrent load; if queueing, batching, and inter-query interference dominate the P95 tail, then a per-query threshold policy trained on isolated latencies may not control the distributional SLO.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive RAG retrieval: SLO compliance from 30% to 95%","Per-query retrieval budgets lift RAG SLOs by 65 points","SAGE: Adaptive retrieval hits 95% SLO with 36% lower latency","RAG retrieval that adapts to query difficulty: 95% SLO, 51% less cost","One learned policy for RAG: 95% SLO compliance, 2 EM drop"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000797,"raw_usage":{"total_tokens":3543,"prompt_tokens":1014,"completion_tokens":2529,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":630,"completion_tokens_details":{"reasoning_tokens":2415}},"tokens_in":630,"tokens_out":2529,"duration_ms":14541,"temperature":1.0,"reasoning_tokens":2415,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:14:09.233357+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Deploy SAGE and the best static budget side by side in a serving system with sustained concurrent traffic, measure P95 end-to-end latency and SLO compliance under the same 5-second target, and compare the compliance gap. If the gap between SAGE and static k=20 shrinks materially as load increases, or SAGE's compliance drops below target at high concurrency, the load-independent oracle assumption is falsified.","supporting_citations":[{"cited_title":"A reduction of imitation learning and structured prediction to no-regret online learning,","cited_arxiv_id":null,"evidence_quote":"Provides the imitation-learning reduction that grounds SAGE's offline behavior-cloning approach."}],"review_version":1}