Pith. sign in

REVIEW 4 major objections 6 minor 51 references

Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single per-request latency budget can jointly drive scheduling and KV-cache movement, lifting SLO-satisfied goodput up to 2.4x and cutting violations by 40%.

desk verdict Clean core idea, but the published algorithm and Eq. 5 don't implement the claimed fairness mechanism, and the sim-based evidence needs artifacts before the goodput numbers can be trusted. read the letter →

arxiv 2608.06557 v1 pith:AZX77UFR submitted 2026-08-06 cs.DC cs.LG

Muhammad Adnan , Rohan Mahapatra , Prashant J. Nair , Daniel Berger , Pantea Zardoshti , Rodrigo Fonseca , Esha Choukse This is my paper · ORCID
classification cs.DCcs.LG
keywords per-requestlatencybudgetLLMinferenceservingSLO-awareschedulingKVcachemanagementgoodputprefixcachingfairnessmemoryhierarchy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that requests governed by the same latency service-level objective have very different amounts of slack, because their input length, generation length, and reusable key-value cache state make their execution times differ by orders of magnitude. It defines the per-request latency budget as the SLO minus the predicted remaining service time, and claims that this single number, updated continuously, is the right common resource for both request scheduling and cache placement decisions. A scheduler that runs least-remaining-budget-first and a memory manager that only restores or prefetches cache state when the transfer fits the budget can push queueing and data-movement overhead onto requests that can absorb it. On production traces across three large language models, this jointly coordinated design is reported to improve SLO-satisfied goodput by up to 2.4x and reduce SLO violations by 40% relative to the default first-come, first-served scheduler, while keeping fairness across request classes high.

What carries the argument

The central object is the per-request latency budget, defined as the difference between the request's SLO target and its predicted TTFT, with the prediction averaged over current and maximum load and scaled by a per-model guardband $\gamma$. The budget is updated as time elapses, and it drives three decisions: least-remaining-budget-first queue ordering; a budget-feasible prefetch bound $M_{r,k} = [B^{\mathrm{rem}}_r - \sum_{\ell \in P(k)} \delta_\ell]^+ / \sum_{\ell \in P(k)} 1/B^{\mathrm{eff}}_\ell$ that caps how many bytes of prefix KV state may be restored from tier $k$; and preemption of the request with the largest remaining budget. This single number links scheduling and memory management so that overhead is charged to the same account that has room to pay for it.

What would settle it

Measure the same traces under Cascade with an oracle estimator that uses actual observed prefill times; if the gap in goodput and violation rate between the oracle and the learned estimator is small, the budget mechanism itself carries the gains, and if it is large, the gains are an artifact of the estimator. A second check is to feed the system long-context requests whose prefill times are deliberately underestimated and observe whether violation rates climb.

Watch

Extended reading notes

Core claim

Cascade establishes that the latency budget—the difference between a request's TTFT service-level target and its predicted prefill time, $B_r = S^{\mathrm{TTFT}}_{c(r)} - L_r$—can serve as a unified control variable. The scheduler orders queued requests by ascending remaining budget $B^{\mathrm{rem}}_r = B_r - (t_{\mathrm{curr}} - a_r)$, so the most urgent requests run first without systematically slowing long-context work. The memory manager admits a deep-tier KV restore only when its transfer time fits the remaining budget, falling back to recomputation otherwise, and preempts the request with the largest remaining budget under HBM pressure. Because queueing, cache movement, and preemption all consume the same budget, the two subsystems no longer work at cross-purposes; the paper claims this removes the dominant sources of SLO violations and delivers up to 2.4x higher SLO-satisfied goodput and 40% fewer violations compared with a first-come, first-served baseline, with near-ideal cross-class fairness.

Load-bearing premise

Everything rests on the offline-trained TTFT estimator predicting prefill latency accurately; if it is systematically optimistic, budgets inflate and requests accept queuing and cache-transfer delays they cannot afford, so SLO violations rise despite the scheduling machinery.

Editorial extensions

If this is right

  • SLO attainment stops being a single aggregate number: the per-class fairness index stays above 0.98 across models and traces, so long-context and reasoning workloads are not silently starved.
  • Deep-tier KV caches become usable for latency-critical requests: a cache hit is only acted on when the restore fits the budget, otherwise the prefix is recomputed, turning previously violating NVMe hits into compliant ones.
  • The system serves the same load with fewer GPUs: at fixed load, it sustains 2.4x goodput with 22% fewer serving instances than the baseline needs.
  • Under rising load, the scheduler degrades gracefully: at 48 QPS the budget-driven policy holds 1.5x goodput and a 14% violation rate where FCFS and EDF collapse to 0.05x goodput and over 90% violations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the budget view is right, the same accounting could price other forms of overhead — batching interference, PD-disaggregation network transfers, speculative recomputation — as debits against the same per-request account, turning SLO management into a budget-constrained optimization across the whole serving stack.
  • The guardband $\gamma$ is a single fitted constant per model; a natural extension is to make it load- and class-dependent, or to learn it online from observed violations, which would likely make the estimator robust to distribution shift.
  • The budget's accuracy hinges on output-length prediction; incorporating conservative worst-case generation estimates could make the mechanism safe for hard SLOs, at the cost of some goodput.
  • Because the mechanism is orthogonal to prefill-decode disaggregation, the same budget could be used to decide whether a request's KV state should migrate to a decode instance or be recomputed, connecting to disaggregated serving designs.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces a per-request latency budget, defined as the difference between a request's class-level SLO and its predicted remaining service time, and builds a serving system (Cascade) around that single quantity. The budget is used to order requests (least-remaining-budget-first), to decide whether to restore or prefetch KV cache from DRAM/NVMe or recompute the prefix, and to select preemption victims under HBM pressure. The authors implement Cascade on top of vLLM and evaluate it in an extension of the Vidur simulator, using production traces from Aliyun Bailian across Qwen-2.5-72B, Llama-3-70B, and Llama-3-405B. Relative to the default vLLM FCFS scheduler, they report goodput improvements of up to 2.4x and a 40% reduction in SLO violations, together with high Jain fairness across input-length classes.

Significance. The observation that request scheduling and KV-cache movement consume the same latency headroom is valuable and, to my knowledge, is not jointly exploited by prior serving systems. If the claimed results hold, the budget-feasible restore bound in Eq. (7) is a clean, falsifiable design rule, and the paper's fairness metric over input-length classes is a useful evaluation tool. The paper also deserves credit for evaluating across three models and ten workload traces, and for showing per-cause SLO violation breakdowns in Figure 11. However, the evaluation rests entirely on an unreleased simulator extension, and the central TTFT estimator is a black box with an undisclosed fitted guardband; the manuscript is therefore not independently verifiable in its current form. There is also an internal inconsistency between the text's claim that budgets are continuously re-estimated and the actual update law in Eq. (5).

major comments (4)
  1. [§IV-C (Eq. 5), Algorithm 1] The remaining-budget update law is only B_rem_r = B_r - (t_curr - a_r), with B_r = S_TTFT - L_r computed once at arrival. Nothing in Eq. (5) or Algorithm 1 refreshes L_r or B_r from current queue occupancy, load, or KV-cache state. This contradicts the abstract and Section IV's claim that the budget is continuously updated from current system load, and it makes the promised Tier-2-to-Tier-1 promotion impossible: a request admitted with B_r <= 0 has non-increasing B_rem_r and can never turn positive. The same staleness also undercuts the restore-admission bound in Eq. (7), which is evaluated with a stale B_rem when load rises after arrival. Finally, Algorithm 1 contains no reserved fraction of each batch for Tier-2 requests, although the text says this is how Tier-2 starvation is prevented. These are not presentation issues: the published algorithm is not the algorithm described. Please give an explicit re-estimation step (e.g., recompute L_r under the current system state each iteration and update B_r and B_rem accordingly), add the Tier-2 batch fraction to the pseudocode, and re-evaluate with that algorithm.
  2. [§IV-B (Eq. 3), Fig. 7] The TTFT estimator is the foundation of every budget in the paper. The paper states that 'the budget is only as reliable as this estimate,' yet Eq. (3) is a weighted average of two regime predictions multiplied by a guardband gamma whose value is never reported and whose estimator model is never described. Fig. 7 validates the estimator only for Qwen-2.5-72B on the Mixed trace, with MAE and correlation; no per-model or per-load validation is shown, and no sensitivity to gamma is given. Because all end-to-end results come from an extended simulator, an optimistic estimator could, by the paper's own admission, inflate budgets and produce the reported goodput gains without the scheduler itself contributing anything. I ask for disclosure of the estimator and gamma values, per-model validation with error distributions, and a sensitivity study of gamma and of the 10x/5x SLO multipliers.
  3. [§V-A.a and §V-B] The evaluation is performed entirely in an extension of the Vidur simulator, populated with profiles from GB200 NVL72 hardware, but neither the simulator patch nor the profiling and configuration files are released. This makes the headline quantitative claims (2.4x goodput, 40% fewer violations) unverifiable from the manuscript alone. In particular, I cannot determine whether the extended simulator correctly models multi-tier KV transfer and LMCache behavior at the claimed bandwidths, which is exactly where the paper's novelty lies. Please release the simulator extension and traces, or provide a detailed validation of the simulator against the real vLLM implementation on at least a subset of the traces.
  4. [§IV-A and §V-A.c] The SLO targets are defined relative to the system's own isolated execution (10x isolated TTFT, 5x isolated TPOT), and the budget is computed as S_TTFT - L_r. If the isolated baselines are measured with the same execution model used to predict L_r, the SLO target, the estimated latency, and the resulting budget are not fully independent, so the 'violation' numbers partly measure consistency with an internally defined target rather than an external contract. I am not claiming this is circular in the sense of being vacuous, but it needs clarification: state how the absolute numbers in Table III were derived and report at least one workload under a fixed absolute SLO (e.g., a 2-second TTFT target) to separate the effect of the budget mechanism from the choice of normalization.
minor comments (6)
  1. [Algorithm 1] The comments in Algorithm 1 say 'via Eq. (2)' and 'via Eq. (3)'; these should refer to Eq. (3) for L_r and Eq. (4) for B_r.
  2. [Section IV title] The section title 'CASCADE: DEADLINE DRIVEN LLM SERVING' uses 'deadline' even though the paper consistently contrasts budgets with deadlines; consider renaming it to 'Budget-Driven LLM Serving'.
  3. [§V-B.1] The text says 'On average Cascade attains 2.4x the goodput of FCFS' while the abstract says 'up to 2.4x'; report the per-trace range so the reader knows whether 2.4x is the mean or the maximum.
  4. [§IV-D.e] The spill decision for a preempted request ('spilled to DRAM when that fits its remaining budget') is not specified by Eq. (7), which is a restore-admission bound; please state the spill rule explicitly or point to the equation that governs it.
  5. [§II-B.b] There is a typo: 'accomodate' should be 'accommodate'.
  6. [§VI] The sentences introducing Nitsum [35] and PolyServe [45] are missing punctuation and read awkwardly; please proofread this paragraph.

Circularity Check

1 steps flagged · score 2.0 of 10

Budget equations are definitions and goodput results are empirical; the only mild pattern is that "latency budget" renames classic laxity, and self-citations are not load-bearing.

  1. renaming known result [Section IV-B/IV-C, Eqs. (4)-(5); Introduction]
    "We define this headroom—the difference between a request’s service level objective and its predicted remaining service time—as its per-request latency budget. ... Brem_r = B_r − (t_curr − a_r) ... requests are dispatched in ascending order of remaining headroom (least-remaining-budget-first)."

    By Eq. (4), B_r = S_TTFT − L_r, and by Eq. (5), B_rem = B_r − elapsed time. This is exactly the classic real-time scheduling concept of laxity/slack: deadline minus remaining execution time minus elapsed time. Dispatching in ascending B_rem is therefore least-laxity-first scheduling, a known policy, presented here as a new "per-request latency budget" mechanism and contrasted with deadline-only ordering. The scheduling half of the claimed budget mechanism thus reduces by construction to a renamed prior concept. The KV-restore admission bound (Eq. 7) is likewise the definitional feasibility check T_k(x) ≤ B_rem. The goodput/fairness evaluation is empirical and independent of the naming, so this is a minor issue rather than a forced result.

full rationale

Most of the paper is a systems/empirical comparison, not a derivation. Eqs. (3)-(7) are definitions: a conservative TTFT estimate, a budget defined as SLO minus that estimate, a remaining-budget update, and an algebraic inversion for the maximum restorable bytes. None of these equations is used to derive the claimed 2.4x goodput or 40% violation reduction; those numbers come from the Vidur-based evaluation against FCFS/EDF/SJF under the same SLOs, so no fitted parameter is renamed as a prediction. Self-citations (Medha [4], Splitwise [29]) appear only in background/related work and are not load-bearing. The only pattern of concern is terminological: B_rem is classic laxity, and least-remaining-budget-first is least-laxity-first; the paper presents this as a new concept. That is a renaming, not a circular derivation, and the KV-cache admission integration plus the empirical evaluation are independent content. Separately, the prose claims Tier-2 budgets are "re-estimated as load falls" with promotion back to Tier-1, but Eq. (5) only subtracts elapsed time from the arrival-time B_r and Algorithm 1 contains no re-estimation step; this is an internal specification gap (a Tier-2 request with B_r ≤ 0 can never have B_rem turn positive under the stated update), not a circularity. The manuscript also honestly flags the estimator risk ("The budget is only as reliable as this estimate..."), which supports the low circularity score.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on one learned estimator and one scalar guardband, both undisclosed, plus a simulator whose fidelity is unverified. No new physical or conceptual entities are introduced; the per-request latency budget is a defined arithmetic quantity.

free parameters (3)
  • guardband gamma = not reported
    Scales the estimated TTFT in Eq. (3) to absorb estimation errors. Set once per model from offline profiling and held fixed across traces; its value is not disclosed or swept, and it directly shifts every budget B_r and therefore every scheduling and prefetch decision.
  • TTFT estimator model = undisclosed learned weights
    The offline-trained predictive model in Section IV-B predicts prefill latency L_actual, L_max, etc. Architecture, training data, and the profiling/calibration procedure are not specified, so the estimator is an opaque fitted function that determines all budgets.
  • SLO multipliers (10x TTFT, 5x TPOT) = 10x, 5x
    These targets are chosen by the authors following prior work and are defined relative to isolated execution. They are experiment settings rather than constants fitted to data, but they define what counts as a violation and materially affect the reported goodput.
assumptions (4)
  • domain assumption The extended Vidur simulator, populated with GB200 NVL72 profiles, faithfully models cluster-scale scheduling and multi-tier KV movement.
    All reported goodput numbers are produced by this simulator (Section V-A); if the simulator is optimistic, the 2.4x claim does not transfer to real systems.
  • ad hoc to paper The TTFT estimator, with guardband gamma, provides a conservative upper bound on actual prefill latency.
    The paper sets L_r as the average of actual and maximum load times gamma (Eq. 3) and asserts this absorbs estimation error; no proof or sensitivity analysis is given.
  • domain assumption Chunked prefill with fixed chunk size 512 and batch size 128 preserves TPOT SLOs regardless of budget-driven dispatch.
    The budget governs TTFT only; the paper assumes TPOT is protected by chunking, but the evaluation does not separate TPOT attainment from TTFT attainment in the goodput metric.
  • domain assumption The SLO targets (TTFTp90 = 10x, TPOTp90 = 5x of isolated execution latency) are the right contract for measuring serving quality.
    The targets are inherited from prior work but are loose enough that requests have 9x their intrinsic prefill time as slack, which is what the budget figures in Fig. 5 exploit.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving." pith.science (2026). https://pith.science/paper/AZX77UFR

@misc{pith2026260806557,
  author       = {Pith},
  title        = {Pith review of: Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AZX77UFR}},
  note         = {Machine review of arXiv:2608.06557}
}
read the original abstract

The reasoning and agentic capabilities of large language models have expanded the range of applications they support, from short interactive exchanges to long, compute-heavy requests. LLM serving platforms today define response-latency service-level objectives, even though requests within the same service can differ by orders of magnitude in input length, generation length, execution cost, and the availability of reusable KV-cache state. As a result, requests governed by the same service level objective have different urgency: after accounting for the time required to execute them, some have substantial latency headroom while others have almost none. We define this headroom---the difference between a request's service level objective and its predicted remaining service time---as its per-request latency budget. We present Cascade, an LLM serving system that estimates and continuously updates this budget from request characteristics, KV-cache state, and current system load. Unlike prior SLO-aware schedulers that use deadlines to govern request ordering alone, Cascade uses a single per-request budget to jointly coordinate request scheduling and KV-cache management across the memory hierarchy. Its scheduler prioritizes requests with little remaining budget, while its memory manager uses the same budget to decide whether non-resident KV state should be restored or prefetched from a deeper tier, retained in HBM, or recomputed. By directing queueing and data-movement overhead toward requests that can absorb it, Cascade improves SLO-satisfied goodput while preserving fairness across heterogeneous request classes. On production traces across three large language models, Cascade improves goodput by up to2.4x and reduces SLO violations by 40% relative to the default vLLM first-come, first-served scheduler.

Figures

Figures reproduced from arXiv: 2608.06557 by the authors.

Figure 1
Figure 1. Goodput and fairness trade-offs across scheduling [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Workload heterogeneity in multi-instance LLM clus [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. System view of LLM Serving system. User requests encompassing multiple tasks are routed to an instance within the [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Impact of deep-tier KV retrieval on TTFT for a [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 6
Figure 6. Figure 6: Overview of Cascade [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: Validation of estimated latency (Lr) against actual execution TTFT (Tprefill) on Qwen-2.5-72B. Performance is evaluated across the Mixed trace using Mean Absolute Error (MAE) and Pearson correlation (R) [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: System goodput and SLO violation rate across all three models and ten workload traces. Goodput (output tokens per [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Time-To-First-Token (TTFT) tail latency distribution for Qwen-2.5-72B across application classes in the [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: Jain’s Fairness Index (J ) across all workloads and models. CASCADE achieves uniformly high fairness across all workloads [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 12
Figure 12. Figure 12: SLO violation rate grouped by input-sequence-length [PITH_FULL_IMAGE:figures/full_fig_p010_12.png]
Figure 14
Figure 14. Figure 14: Goodput and SLO violation under varying load for [PITH_FULL_IMAGE:figures/full_fig_p011_14.png]
Figure 15
Figure 15. Figure 15: TTFT at the p50, p90, and p95 percentiles under [PITH_FULL_IMAGE:figures/full_fig_p011_15.png]
Figure 16
Figure 16. Figure 16: SLO attainment per application class across models [PITH_FULL_IMAGE:figures/full_fig_p012_16.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 24 canonical work pages

  1. [1]

    Qwen2 technical report,

    “Qwen2 technical report,” 2024

    2024

  2. [2]

    Vidur: A large-scale simulation framework for llm inference,

    A. Agrawal, N. Kedia, J. Mohan, A. Panwar, N. Kwatra, B. S. Gula- vani, R. Ramjee, and A. Tumanov, “Vidur: A large-scale simulation framework for llm inference,” inProceedings of Machine Learning and Systems, vol. 6, 2024, pp. 351–366

    work page 2024

  3. [3]

    Taming throughput-latency tradeoff in llm inference with sarathi-serve,

    A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. S. Gulavani, A. Tumanov, and R. Ramjee, “Taming throughput-latency tradeoff in llm inference with sarathi-serve,” inProceedings of the 18th USENIX Conference on Operating Systems Design and Implementation, ser. OSDI’24. USA: USENIX Association, 2024

    2024

  4. [4]

    No request left behind: Tackling heterogeneity in long-context llm inference with medha,

    A. Agrawal, H. Qiu, J. Chen, ´I˜nigo Goiri, C. Zhang, R. Shahid, R. Ramjee, A. Tumanov, and E. Choukse, “No request left behind: Tackling heterogeneity in long-context llm inference with medha,”

    work page

  5. [5]

    Llama 3 model card,

    AI@Meta, “Llama 3 model card,” 2024. [Online]. Available: https://github.com/meta-lama/llama3/blob/main/MODEL CARD.md

    work page 2024

  6. [6]

    GQA: Training generalized multi-query transformer models from multi-head checkpoints,

    J. Ainslie, J. Lee-Thorp, M. de Jong, Y . Zemlyanskiy, F. Lebron, and S. Sanghai, “GQA: Training generalized multi-query transformer models from multi-head checkpoints,” inProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Association for Computational Linguistics, Dec....

    2023

  7. [7]

    Qwen-bailian anonymous dataset,

    Bailian, “Qwen-bailian anonymous dataset,” 2025. [Online]. Available: https://github.com/alibaba-edu/qwen-bailian-usagetraces-anon

    work page 2025

  8. [8]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Her...

    arXiv 2021

Show all 51 references
  1. [9]

    Slos-serve: Optimized serving of multi-slo llms,

    S. Chen, Z. Jia, S. Khan, A. Krishnamurthy, and P. B. Gibbons, “Slos-serve: Optimized serving of multi-slo llms,” 2025. [Online]. Available: https://arxiv.org/abs/2504.08784

    2025 arXiv

  2. [10]

    Atp: Adaptive tensor parallelism for foundation models,

    S. Cheng, Z. Liu, J. Du, and Y . You, “Atp: Adaptive tensor parallelism for foundation models,” 2023. [Online]. Available: https: //arxiv.org/abs/2301.08658

    2023 arXiv

  3. [11]

    Jockey: guaranteed job latency in data parallel clusters,

    A. D. Ferguson, P. Bodik, S. Kandula, E. Boutin, and R. Fonseca, “Jockey: guaranteed job latency in data parallel clusters,” in Proceedings of the 7th ACM European Conference on Computer Systems, ser. EuroSys ’12. New York, NY , USA: Association for Computing Machinery, 2012, ...

    2012

  4. [12]

    Prompt cache: Modular attention reuse for low-latency inference,

    I. Gim, G. Chen, S. seob Lee, N. Sarda, A. Khandelwal, and L. Zhong, “Prompt cache: Modular attention reuse for low-latency inference,”

  5. [13]

    Qoserve: Breaking the silos of llm inference serving,

    K. Goel, J. Mohan, N. Kwatra, R. S. Anupindi, and R. Ramjee, “Qoserve: Breaking the silos of llm inference serving,” inProceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ser. ASPLOS ’26. New ...

    2026

  6. [14]

    [Online]

    Google, “llm-d,” 2025. [Online]. Available: https://github.com/llm- d/llm-d

    2025

  7. [15]

    Kvquant: towards 10 million context length llm inference with kv cache quantization,

    C. Hooper, S. Kim, H. Mohammadzadeh, M. W. Mahoney, Y . S. Shao, K. Keutzer, and A. Gholami, “Kvquant: towards 10 million context length llm inference with kv cache quantization,” inProceedings of the 38th International Conference on Neural Information Processing Systems, ser....

    2024

  8. [16]

    Inference without interference: Disaggregate llm inference for mixed downstream workloads,

    C. Hu, H. Huang, L. Xu, X. Chen, J. Xu, S. Chen, H. Feng, C. Wang, S. Wang, Y . Bao, N. Sun, and Y . Shan, “Inference without interference: Disaggregate llm inference for mixed downstream workloads,” 2024. [Online]. Available: https://arxiv.org/abs/2401.11181

    2024 arXiv

  9. [17]

    A quantitative measure of fairness and discrimination for resource allocation in shared computer systems,

    R. Jain, D. Chiu, and W. Hawe, “A quantitative measure of fairness and discrimination for resource allocation in shared computer systems,”

  10. [18]

    Sageserve: Optimizing llm serving on cloud data centers with forecast aware auto-scaling,

    S. Jaiswal, K. Jain, Y . Simmhan, A. Parayil, A. Mallick, R. Wang, R. S. Amant, C. Bansal, V . Ruhle, A. Kulkarni, S. Kofsky, and S. Rajmohan, “Sageserve: Optimizing llm serving on cloud data centers with forecast aware auto-scaling,”Proc. ACM Meas. Anal. Comput. Syst., vol. 9...

    2025 doi

  11. [19]

    Learned best-effort llm serving,

    S. Jha, C. Hooper, X. Liu, S. Kim, and K. Keutzer, “Learned best-effort llm serving,” 2024. [Online]. Available: https://arxiv.org/abs/2401.07886

    2024 arXiv

  12. [20]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” inProceedings of the 29th Symposium on Operating Systems Principles, ser. SOSP ’23. New York, NY ...

    2023

  13. [21]

    Tokenscale: Timely and accurate autoscaling for disaggregated llm serving with token velocity,

    R. Lai, H. Liu, C. Lu, Z. Liu, S. Cao, S. Shao, Y . Zhang, L. Mai, and D. Ustiugov, “Tokenscale: Timely and accurate autoscaling for disaggregated llm serving with token velocity,” 2025. [Online]. Available: https://arxiv.org/abs/2512.03416

    2025

  14. [22]

    Revisiting disaggregated large language model serving for performance and energy implications,

    J. Li, Y . Zhu, B. Chen, E. K. Lee, and K. Nahrstedt, “Revisiting disaggregated large language model serving for performance and energy implications,” inProceedings of the Sixth European Workshop on Machine Learning and Systems, ser. EuroMLSys ’26. New York, NY , USA: Associat...

    2026

  15. [23]

    AlpaServe: Statistical multiplexing with model parallelism for deep learning serving,

    Z. Li, L. Zheng, Y . Zhong, V . Liu, Y . Sheng, X. Jin, Y . Huang, Z. Chen, H. Zhang, J. E. Gonzalez, and I. Stoica, “AlpaServe: Statistical multiplexing with model parallelism for deep learning serving,” in17th USENIX Symposium on Operating Systems Design and Implementation (...

    2023

  16. [24]

    Scheduling algorithms for multiprogramming in a hard-real-time environment,

    C. L. Liu and J. W. Layland, “Scheduling algorithms for multiprogramming in a hard-real-time environment,”J. ACM, vol. 20, no. 1, p. 46–61, Jan. 1973. [Online]. Available: https://doi.org/10.1145/321738.321743

    1973

  17. [25]

    Lmcache: An efficient kv cache layer for enterprise-scale llm inference,

    Y . Liu, Y . Cheng, J. Yao, Y . An, X. Chen, S. Feng, Y . Huang, S. Shen, R. Zhang, K. Du, and J. Jiang, “Lmcache: An efficient kv cache layer for enterprise-scale llm inference,” 2025. [Online]. Available: https://arxiv.org/abs/2510.09665

    2025

  18. [26]

    Ai-dynamo,

    NVIDIA, “Ai-dynamo,” 2025. [Online]. Available: https://github.com/ aidynamo/dynamo

    2025

  19. [27]

    Nvidia gb200 nvl partition,

    NVIDIA, “Nvidia gb200 nvl partition,” 2025. [Online]. Available: https: //docs.nvidia.com/multi-node-nvlink-systems/partition-guide-v1-2.pdf

    2025

  20. [28]

    Nvidia gb200 nvl72 delivers trillion-parameter llm training and real-time inference,

    NVIDIA, “Nvidia gb200 nvl72 delivers trillion-parameter llm training and real-time inference,” 2025. [Online]. Avail- able: https://developer.nvidia.com/blog/nvidia-gb200-nvl72-delivers- trillion-parameter-llm-training-and-real-time-inference/

    2025

  21. [29]

    Splitwise: Efficient generative llm inference using phase splitting,

    P. Patel, E. Choukse, C. Zhang, A. Shah, I. n. Goiri, S. Maleki, and R. Bianchini, “Splitwise: Efficient generative llm inference using phase splitting,” inProceedings of the 51st Annual International Symposium on Computer Architecture, ser. ISCA ’24. IEEE Press, 2025, p. 118–...

    2025

  22. [30]

    Conserve: Fine-grained gpu harvesting for llm online and offline co-serving,

    Y . Qiao, S. Anzai, S. Yu, H. Ma, S. Yang, Y . Wang, M. Kim, Y . Wu, Y . Zhou, J. Xing, J. E. Gonzalez, I. Stoica, and H. Xu, “Conserve: Fine-grained gpu harvesting for llm online and offline co-serving,”

  23. [31]

    Mooncake: trading more storage for less computation — a kvcache-centric architecture for serving llm chatbot,

    R. Qin, Z. Li, W. He, J. Cui, F. Ren, M. Zhang, Y . Wu, W. Zheng, and X. Xu, “Mooncake: trading more storage for less computation — a kvcache-centric architecture for serving llm chatbot,” inProceedings of the 23rd USENIX Conference on File and Storage Technologies, ser. FAST ...

    2025

  24. [32]

    Tutti: Making ssd-backed kv cache practical for long-context llm serving,

    S. Qiu, Y . Hu, X. Wang, W. Zhu, J. Yan, H. Chen, K. Xu, K. Chen, and Y . Zhang, “Tutti: Making ssd-backed kv cache practical for long-context llm serving,” 2026. [Online]. Available: https://arxiv.org/abs/2605.03375

    2026 arXiv

  25. [33]

    Timecard: controlling user-perceived delays in server-based mobile applications,

    L. Ravindranath, J. Padhye, R. Mahajan, and H. Balakrishnan, “Timecard: controlling user-perceived delays in server-based mobile applications,” inProceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles, ser. SOSP ’13. New York, NY , USA: Association for ...

    2013

  26. [34]

    Available: https://arxiv.org/abs/2410.01228

    [Online]. Available: https://arxiv.org/abs/2410.01228

    arXiv

  27. [35]

    Nitsum: Serving tiered llm requests with adaptive tensor parallelism,

    V . Srivatsa, Z. He, P. Guo, D. Li, and Y . Zhang, “Nitsum: Serving tiered llm requests with adaptive tensor parallelism,” 2026. [Online]. Available: https://arxiv.org/abs/2605.05467

    2026 arXiv

  28. [36]

    Kvcache cache in the wild: Characterizing and optimizing kvcache cache at a large cloud provider,

    J. Wang, J. Han, X. Wei, S. Shen, D. Zhang, C. Fang, R. Chen, W. Yu, and H. Chen, “Kvcache cache in the wild: Characterizing and optimizing kvcache cache at a large cloud provider,” in2025 USENIX Annual Technical Conference (USENIX ATC 25). USENIX Association, Jul. 2025. [Onli...

    2025

  29. [37]

    Better never than late: meeting deadlines in datacenter networks,

    C. Wilson, H. Ballani, T. Karagiannis, and A. Rowtron, “Better never than late: meeting deadlines in datacenter networks,” inProceedings of the ACM SIGCOMM 2011 Conference, ser. SIGCOMM ’11. New York, NY , USA: Association for Computing Machinery, 2011, p. 50–61. [Online]. Ava...

    2011

  30. [38]

    Preble: Efficient distributed prompt scheduling for llm serving,

    V . Srivatsa, Z. He, R. Abhyankar, D. Li, and Y . Zhang, “Preble: Efficient distributed prompt scheduling for llm serving,” 2024. [Online]. Available: https://arxiv.org/abs/2407.00023

    2024 arXiv

  31. [39]

    Aegaeon: Effective gpu pooling for concurrent llm serving on the market,

    Y . Xiang, X. Li, K. Qian, Y . Yang, D. Zhu, W. Yu, E. Zhai, X. Liu, X. Jin, and J. Zhou, “Aegaeon: Effective gpu pooling for concurrent llm serving on the market,” inProceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles, ser. SOSP ’25. New York, NY , US...

    2025

  32. [40]

    Orca: A distributed serving system for transformer-based generative models,

    G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, “Orca: A distributed serving system for transformer-based generative models,” in16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). Carlsbad, CA: USENIX Association, Jul. 2022, pp. 521–538. [...

    2022

  33. [41]

    Superinfer: Slo-aware rotary scheduling and memory management for llm inference on superchips,

    J. Yu, M. Hu, Z. Lin, and M. Zhang, “Superinfer: Slo-aware rotary scheduling and memory management for llm inference on superchips,”

  34. [42]

    FastServe: Iteration-Level preemptive scheduling for large language model inference,

    B. Wu, Y . Zhong, Z. Zhang, S. Liu, F. Liu, Y . Sun, G. Huang, X. Liu, and X. Jin, “FastServe: Iteration-Level preemptive scheduling for large language model inference,” in23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26). Renton, W A: USENIX Assoc...

    2026

  35. [43]

    Sglang: efficient execution of structured language model programs,

    L. Zheng, L. Yin, Z. Xie, C. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez, C. Barrett, and Y . Sheng, “Sglang: efficient execution of structured language model programs,” in Proceedings of the 38th International Conference on Neural Information Proc...

    2024

  36. [44]

    Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving,

    Y . Zhong, S. Liu, J. Chen, J. Hu, Y . Zhu, X. Liu, X. Jin, and H. Zhang, “Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving,” in18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). USENIX Association, ...

    2024

  37. [45]

    Polyserve: Efficient multi-slo serving at scale,

    K. Zhu, H. Shi, L. Xu, J. Shan, A. Krishnamurthy, B. Kasikci, and L. Xie, “Polyserve: Efficient multi-slo serving at scale,” 2025. [Online]. Available: https://arxiv.org/abs/2507.17769 AI USE We acknowledge the use of AI in the preparation of this submission. AI assistants wer...

    2025 arXiv

  38. [47]

    Jitserve: Slo-aware llm serving with imprecise request information,

    W. Zhang, Z. Wu, Y . Mu, R. Ning, B. Liu, N. Sarda, M. Lee, and F. Lai, “Jitserve: Slo-aware llm serving with imprecise request information,”

  39. [48]

    Available: https://arxiv.org/abs/2504.20068

    [Online]. Available: https://arxiv.org/abs/2504.20068

  40. [1998]

    Available: https://arxiv.org/abs/cs/9809099

    [Online]. Available: https://arxiv.org/abs/cs/9809099

    arXiv

  41. [2024]

    Available: https://arxiv.org/abs/2311.04934

    [Online]. Available: https://arxiv.org/abs/2311.04934

    arXiv

  42. [2025]

    Available: https://arxiv.org/abs/2409.17264

    [Online]. Available: https://arxiv.org/abs/2409.17264

  43. [2026]

    Available: https://arxiv.org/abs/2601.20309

    [Online]. Available: https://arxiv.org/abs/2601.20309

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.