{"id":"3f539fad-1955-4c82-97b9-65caf6cc6e3e","arxiv_id":"2506.19677","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SABER uses a fitted scalability curve and deadline-aware admission control to dynamically size continuous batches, improving SLA-compliant throughput by up to 26% over the best static configuration.","lead":"This paper presents SABER, an adaptive scheduler that decides which coding AI requests to admit to the GPU batch based on predicted speed and whether they can finish by their deadline. It reports up to 26% more requests meeting their target response times and lower latency variability than the best fixed batch settings in single-GPU CodeLLM serving.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The single-variable speed estimator f(L) is never validated on held-out workload mixes or on per-request speed variance at fixed concurrency; SABER's admission decisions inherit that error, so the 26% goodput claim may be calibration-specific.","rationale":"The reader's weakest assumption identifies exactly the condition on which the central claim rests: f(L) must accurately predict generation speed for every admission decision. My review sharpens this into a testable generalization requirement: the estimator is a mean curve with no reported variance, and the evaluation does not demonstrate that the workload mixes are disjoint from the offline profiling data. Section VII already shows that estimator choice can flip the sign of the improvement, so the mechanism's benefit is contingent on estimator fidelity. This is a substantive concern, but it does not disprove the mechanism; it makes the paper's empirical claim conditional on held-out validation. The reader's CONDITIONAL verdict remains appropriate, so no verdict change is needed.","tokens_in":16486,"tokens_out":12619,"duration_ms":134677,"concrete_test":"Run a leave-one-workload-out test: train f(L) exclusively on two of the three workload mixes, then run SABER on the held-out mix (cyclically) under the RQ2 load sweep, comparing against the best static configuration at each RPS. Report the per-request speed residual distribution at fixed L for each held-out mix. If SABER's average goodput advantage over best static is negative on any held-out mix, or if the residual spread at fixed L exceeds the speed margin required by Algorithm 1, the single-variable estimator does not transfer and the headline gain is calibration-specific.","verdict_should_be":"UNCHANGED","load_bearing_attack":"SABER's central mechanism is the admission predicate in Algorithm 1, lines 8-13: it admits a request only if f(curLoad+1), a USL fit of per-request token-generation speed against concurrency L (Sec. IV-B), is at least the required speed of the incoming request and of every active request. The paper does not establish that this single-variable estimator is valid for the request mix being admitted. It reports R^2=0.99 for the USL fit but never reports the residual spread of per-request speeds at a fixed L; work the paper itself cites (Sun et al., [35]) documents up to 2.6x speed variation at a given batch size depending on composition, so a mean curve can be accurate on average yet wrong for the specific prefill/decode mix. Moreover, the offline profiling and the RQ1/RQ2 evaluation are not shown to use disjoint workload mixes; if the evaluation mixes overlap the training data, the goodput gains are in-distribution. Section VII's sensitivity analysis shows the mechanism is fragile: replacing USL with logistic or linear estimators changes W2's average advantage from +1.2% to -5.3%/-6.7%. Since every admission decision inherits f(L)'s error, the headline 26% goodput improvement over the best static configuration is not established for workload mixes that differ from the calibration distribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SABER, an adaptive request-scheduling layer for CodeLLM serving systems that use continuous batching. In an offline phase, SABER fits a Universal Scalability Law (USL) curve to measured per-request token-generation speed as a function of the number of concurrently executing requests. Online, a two-tier queue classifies requests by their required generation speed, and an admission-control loop admits a request only if the predicted speed at the increased concurrency is sufficient for both the incoming request and all active requests. The evaluation uses Qwen-Coder-2.5B on vLLM with three workload mixes, compares against the best static batch-size configuration for each request rate, and reports goodput, completion-time ratios, and coefficient-of-variation metrics. The paper claims up to 26% goodput improvement and up to 45% latency-variability reduction over the best static configurations.","tokens_in":16817,"tokens_out":5201,"duration_ms":52041,"significance":"If the claims hold, SABER is a useful and practical contribution: it is a lightweight, non-intrusive admission-control layer that can be added to existing serving engines, and it directly targets the realistic problem that a static batch-size configuration cannot adapt to changing request rates and workload mixes. The study has clear strengths: it performs a systematic sweep over 360 static configurations, uses the per-RPS best static configuration as a conservative oracle baseline, and includes a sensitivity analysis of the estimation function. The empirical comparison is the core of the paper and is directionally reasonable. However, several load-bearing points need attention before the headline claims can be accepted: the reported 45% variability-reduction figure is not supported by the CV data in the paper, the admission decision rests entirely on a single-variable speed estimator whose residual spread and out-of-distribution behavior are not validated, and the SLA thresholds are defined from the system's own mean completion times rather than from external requirements.","major_comments":[{"comment":"The abstract states that SABER \"reduces latency variability by up to 45%\", but the CV values reported in Figure 6 do not support this number. The largest relative reduction is in Workload 1, from 52.4% to 33.7%, which is a 35.7% reduction; Workload 2 is 20.8% to 17.5% (15.9%) and Workload 3 is 36.4% to 25.1% (31.0%). The claim should be corrected to the largest observed reduction or an explicit computation of the 45% figure should be provided.","section":"Abstract; Section VI-C, Figure 6"},{"comment":"Every admission decision is based on f(L), a single-variable USL curve fitted to mean per-request token-generation speed as a function of concurrency. The paper reports R^2=0.99 for the fit but does not report residual spread, per-request speed variance at fixed L, or validation on held-out workload mixes. Since the cited work of Sun et al. [35] documents up to 2.6x speed variation with batch composition, a high R^2 on the mean curve does not establish that the admission predicate is safe for the specific prefill/decode mix being admitted. This is load-bearing because Section VII shows that replacing the estimator changes the Workload 2 average advantage from +1.2% to -5.3% (logistic) and -6.7% (linear); the headline goodput gains are therefore not established outside the calibration distribution.","section":"Section IV-B; Algorithm 1, lines 8-13"},{"comment":"The admission check compares the predicted speed against each active request's reqSpd, but reqSpd is recorded once at admission as maxTokens/(deadline - currentTime) and is never recomputed from the remaining number of tokens. An active request that has generated fewer tokens than its schedule requires can therefore have an actual remaining speed requirement higher than the stored reqSpd, and the algorithm will admit new requests that push it past its SLA without detecting the violation. The algorithm should update active-request requirements on each admission decision, or check the remaining tokens, for the SLA-guarantee claim to hold.","section":"Section IV-C, Step 2; Algorithm 1, line 10"},{"comment":"The SLA thresholds in Table I are not external requirements but the system's own mean completion times measured at 10 RPS on the same hardware and model. Because SABER is evaluated against thresholds derived from the same serving stack it controls, the goodput numbers measure compliance with a self-referential target. The paper should either anchor the SLA values to independent, user-defined requirements or explicitly state that the evaluation demonstrates relative improvement under a simulated SLA regime, not an absolute SLA guarantee.","section":"Section III-B"},{"comment":"The evaluation appears to use the same task types, datasets, and SLA thresholds from which the offline profile of Section IV-B is built; the paper does not state that evaluation workloads are disjoint from calibration data. Without a held-out workload mix or an explicit statement of overlap, the goodput improvements cannot be separated from in-distribution calibration effects. A validation on workload compositions or request-length distributions not seen during profiling is needed.","section":"Section V-B; Section VI-B"}],"minor_comments":[{"comment":"The abstract contains a grammatical error: \"systems employs\" should be \"systems employ\"; also, \"We propose\" is capitalized mid-sentence and should be \"we propose\" for consistency.","section":"Abstract"},{"comment":"The Figure 4 caption says the CDF analysis is at RPS=10, but Section V-C reports comparisons at RPS=5 for Workload 1 and RPS=20 for Workload 2; the caption should match the text.","section":"Figure 4"},{"comment":"The name is written inconsistently as \"SABER\" in the text and \"Saber\" in figures and some figure captions; one capitalization convention should be used throughout.","section":"Throughout"},{"comment":"Equation (1) defines the coefficient of variation, but the text does not explicitly state that sigma and mu are computed over the per-request completion-time-to-SLA ratios across the different RPS values; this should be clarified.","section":"Section VI-B, Equation (1)"},{"comment":"Each configuration is evaluated with only 100 requests and three repetitions; reporting only the mean hides run-to-run variability, which is especially relevant for the goodput differences of a few percentage points in Figure 5. Confidence intervals or per-run points should be reported.","section":"Section III-B"}],"recommendation":"major_revision","confidential_remarks":"The central idea is sound and the empirical methodology is mostly careful, but the unsupported 45% claim, the unvalidated single-variable estimator, and the self-referential SLA definition are load-bearing for the paper's advertised contributions. These issues are fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuine, well-structured systems paper that deserves a serious referee, but the abstract overclaims and the SLA setup is self-referential. The core idea—an admission-control layer over continuous batching using a fitted speed-versus-concurrency curve and a two-tier deadline queue—is a sensible lightweight addition for single-GPU CodeLLM serving. It requires no engine modification, which is a real practical advantage over ExeGPT or DistServe-style interventions. The motivation study is the strongest part: it cleanly shows that no static batch size wins across workload mixes and load levels, and that the penalty for picking wrong can be severe. The experimental design is also careful in one important respect: SABER is compared against the best static configuration per workload and RPS, so static gets the benefit of hindsight.\n\nThe soft spots are real but not fatal. First, the abstract claims 'up to 45% latency variability reduction,' but the CV numbers in Fig. 6 give 35.7% as the largest reduction (W1). The 'up to 26% goodput' is also a single operating point (W3, 20 RPS); average gains are 1.2–10.2%, and W2 gains are basically noise. Second, SLAs are defined as the system's own mean completion time at moderate load. That makes goodput a measure of self-consistency rather than external requirements; the relative comparison against static remains meaningful, but the absolute numbers should not be read as real-world SLA compliance. Third, the estimator f(L) is a mean curve with no reported residual spread at fixed concurrency. The paper itself cites Llumnix's 2.6x composition-dependent speed variation, so a mean curve can be wrong for a specific prefill/decode mix. The sensitivity analysis in Section VII confirms this fragility: substituting logistic (R^2=0.97) for USL (R^2=0.99) flips W2 from +1.2% to -5.3%. Fourth, the paper never states that offline profiling data are disjoint from evaluation traces, so we cannot rule out in-distribution gains. Finally, no artifacts are provided, which is a real liability for a systems paper.\n\nWho benefits: researchers and practitioners working on lightweight scheduling for self-hosted LLM serving. It won't change the world, but it is a usable empirical datapoint. I would accept it for peer review, with a request for major revision: fix the abstract numbers, add a held-out evaluation or at least report residual variance, discuss the self-referential SLA definition, and release code and configs. The central mechanism is plausible and the paper is honest about many of its limitations; with those fixes it would be a solid conference paper.","headline":"SABER is a solid engineering study of adaptive admission control for CodeLLM serving, but the headline numbers overshoot the data and the SLA setup is self-referential; with a corrected abstract and held-out evaluation it would be a reasonable systems paper.","tokens_in":17346,"tokens_out":4108,"would_cite":false,"duration_ms":41589,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SABER shows that replacing a static batch-size setting with an SLA-aware admission controller improves goodput by up to 26% and cuts latency variability by up to 45% in single-GPU CodeLLM serving.","keywords":["large language models","request scheduling","SLA compliance","adaptive systems","continuous batching","admission control","code generation","goodput"],"falsifier":"Feed SABER a workload whose output-token distribution is deliberately shifted after the offline profile is fixed, for example all code-translation requests with double the profiled output length, or a mix that alternates between heavy and light tasks every few minutes. If goodput then falls below the best static configuration chosen for each phase, or SLA violations concentrate on requests the controller predicted would succeed, the claim that the concurrency-only estimator generalizes is falsified.","tokens_in":16328,"feed_emoji":"⚙️","tokens_out":10619,"duration_ms":93915,"temperature":0.7,"pith_summary":"Continuous batching in LLM serving engines relies on a fixed maximum batch size, but no single fixed size stays optimal as the mix of coding tasks and arrival rate changes. SABER replaces that static knob with an admission-control layer that predicts, for each incoming request, whether it can finish before its SLA deadline without slowing already-running requests past theirs. On a single-GPU CodeLLM workload, the paper reports that SABER beats the best static configuration in goodput — the fraction of requests completed within their SLA — by up to 26%, with up to 45% lower latency variability, and does so without restarts or manual tuning. The paper's case is that SLA-aware adaptive scheduling, not better static tuning, is the path to reliable self-hosted CodeLLM serving.","feed_headline":"Adaptive batching lifts CodeLLM SLA goodput by up to 26%","feed_subtitle":"A speed-prediction admission layer beats every fixed batch size without restarts or manual tuning.","key_machinery":"The load-bearing object is the estimation function $\\hat{v}=f(L)$, a Universal Scalability Law curve fitted offline to measured pairs of concurrency and token-generation speed; it predicts the per-request generation speed when $L$ requests run together. The admission control loop uses $f(\\text{current load} + 1)$ to test two conditions before letting a request join the batch: the predicted speed must cover the incoming request's required speed (max tokens divided by remaining time to deadline), and it must not push any already-executing request below its own required speed. A two-tier queue supports the loop: requests whose required speed already exceeds the engine's fastest speed are demoted to a low-priority best-effort queue, and a random window over the high-priority queue prevents head-of-line blocking.","core_discovery":"On its own terms, the paper's discovery is that the failure of static continuous batching under heterogeneous CodeLLM loads is avoidable: a lightweight admission controller can match or beat the best static configuration. The authors show experimentally that the optimal maximum batch size shifts with workload composition (30 for heavy tasks, 70 for light tasks, 80 for a balanced mix) and with request rate, so no fixed setting can serve a production workload well. SABER instead fits a Universal Scalability Law curve to offline measurements of token-generation speed at different concurrency levels, then online admits a request only if the predicted generation speed at current load plus one is enough to meet both that request's required speed and the required speeds of requests already executing. Requests whose deadlines have become impossible are demoted to a best-effort low-priority queue. Under contention the policy becomes a deliberate triage: short tasks are accelerated, hopeless long tasks are postponed, and aggregate goodput rises while completion-time variability falls.","pith_inferences":["The paper's estimator uses only concurrency as input; feeding it additional features such as average sequence length or prefill/decode split could let the same admission rule survive workload drift outside the offline profile.","SABER's triage appears to optimize perceived responsiveness rather than raw SLA count; a user study could test whether deferring heavy tasks is experienced as better service, not just measured as higher goodput.","The single-GPU framing suggests the method could lift to multi-GPU serving by replacing $f(L)$ with a distributed cost model, a setting the paper explicitly leaves unexplored."],"forward_implications":["Operators no longer need to restart the serving engine to retune batch size when the workload shifts, because SABER adjusts admission decisions continuously.","Under heavy load, light interactive tasks such as short Q&A and summaries are protected at the expense of heavy long-generation tasks, matching what users tolerate waiting for.","The goodput advantage over the best static configuration grows with load, from near zero when capacity is ample to as much as 26 percentage points under saturation.","SABER can be added to an existing continuous-batching engine as an admission layer plus an offline profile, without changing the inference engine's internals.","The quality of the speed estimator determines the gain: replacing the fitted USL curve with less accurate linear or logistic regressors eroded or reversed the goodput advantage in light workloads."],"supporting_citations":[{"why":"It is the Qwen-Coder-2.5B model served in every experiment.","marker":"[3]"},{"why":"It is the continuous-batching engine whose static max-concurrency setting SABER replaces.","marker":"[14]"},{"why":"It introduced iteration-level continuous batching, the execution model SABER assumes.","marker":"[17]"},{"why":"It quantifies decode-speed degradation as batch tokens grow, motivating the speed-estimation approach.","marker":"[35]"},{"why":"It supplies the goodput framing and SLA-aware serving perspective SABER adopts.","marker":"[36]"},{"why":"It is the Code QnA dataset, one of the four SLA task types used in the workloads.","marker":"[42]"},{"why":"It is the Code Generation dataset, a heavy-output task in the workload mixes.","marker":"[43]"},{"why":"It is the Code Summary dataset, a light task in the workload mixes.","marker":"[44]"},{"why":"It is the Code Translation dataset, the longest-output task type.","marker":"[45]"},{"why":"It is the Universal Scalability Law, the fitted curve behind the estimation function $f(L)$.","marker":"[50]"}],"fun_headline_variants":["SLA-aware batching cuts CodeLLM latency variance 45%","No-restart adaptive batching beats static CodeLLM serving","SABER's adaptive admission lifts CodeLLM goodput 26%","Adaptive SLA-aware batching: 26% better CodeLLM goodput","SABER adaptive batching lifts CodeLLM goodput up to 26%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the fitted speed estimator $f(L)$, which predicts token-generation rate from the number of concurrent requests alone; if a live workload mix behaves differently from the offline profile, the admission decisions will either admit requests that bust their SLAs or reject requests that would have made it.","fun_headline_variants_meta":{"raw":{"variants":["SLA-aware batching cuts CodeLLM latency variance 45%","No-restart adaptive batching beats static CodeLLM serving","SABER's adaptive admission lifts CodeLLM goodput 26%","Adaptive SLA-aware batching: 26% better CodeLLM goodput","SABER adaptive batching lifts CodeLLM goodput up to 26%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000989,"raw_usage":{"total_tokens":4162,"prompt_tokens":882,"completion_tokens":3280,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":3180}},"tokens_in":498,"tokens_out":3280,"duration_ms":21153,"temperature":1.0,"reasoning_tokens":3180,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:28:03.027004+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Feed SABER a workload whose output-token distribution is deliberately shifted after the offline profile is fixed, for example all code-translation requests with double the profiled output length, or a mix that alternates between heavy and light tasks every few minutes. If goodput then falls below the best static configuration chosen for each phase, or SLA violations concentrate on requests the controller predicted would succeed, the claim that the concurrency-only estimator generalizes is falsified.","supporting_citations":[{"cited_title":"Orca: A distributed serving system for{Transformer-Based}generative models,","cited_arxiv_id":null,"evidence_quote":"It introduced iteration-level continuous batching, the execution model SABER assumes."},{"cited_title":"Hl-codellama-chat-response dataset,","cited_arxiv_id":null,"evidence_quote":"It is the Code QnA dataset, one of the four SLA task types used in the workloads."},{"cited_title":"Synthetic code generations dataset,","cited_arxiv_id":null,"evidence_quote":"It is the Code Generation dataset, a heavy-output task in the workload mixes."},{"cited_title":"Code summary java dataset,","cited_arxiv_id":null,"evidence_quote":"It is the Code Summary dataset, a light task in the workload mixes."},{"cited_title":"Code translation dataset,","cited_arxiv_id":null,"evidence_quote":"It is the Code Translation dataset, the longest-output task type."},{"cited_title":"Integrating concurrency control in n-tier application scaling management in the cloud,","cited_arxiv_id":null,"evidence_quote":"It is the Universal Scalability Law, the fitted curve behind the estimation function $f(L)$."}],"review_version":2}