Hierarchical Autoscaling for Large Language Model Serving with Chiron

Archit Patke; Chandra Narayanaswami; Dhemath Reddy; Ravishankar Iyer; Saurabh Jha; Zbigniew Kalbarczyk

REVIEW 3 major objections 6 minor 21 references

Hierarchical Autoscaling for Large Language Model Serving with Chiron

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Chiron claims that SLO-aware hierarchical backpressure—local batch-size scaling plus global queue-based instance scaling—achieves up to 90% higher SLO attainment and 70% better GPU efficiency than Llumnix for LLM serving.

desk verdict Chiron is a plausible and useful hierarchical autoscaler for LLM serving with genuinely new SLO-aware design, but reproducibility gaps and an unmodeled feedback loop in the waiting-time estimator keep it at conditional acceptance. read the letter →

arxiv 2501.08090 v1 pith:SIBYRZFK submitted 2025-01-14 cs.DC cs.AI

Archit Patke , Dhemath Reddy , Saurabh Jha , Chandra Narayanaswami , Zbigniew Kalbarczyk , Ravishankar Iyer This is my paper · ORCID

classification cs.DCcs.AI

keywords LLMservingautoscalingSLOattainmenthierarchicalbackpressurecontinuousbatchingbatchsizeadaptationqueuewaitingtimeestimationGPUefficiency

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that LLM serving autoscalers should act on what requests actually need—their time-to-first-token and per-token latency targets—rather than on raw GPU utilization. Chiron reads two levels of backpressure: per-instance pressure from inter-token latency and throughput, and cluster-level pressure from queue size, utilization, and SLO deadlines. Its local autoscaler grows or shrinks the maximum batch size; its global autoscaler adds or retires interactive, mixed, and batch instances. In vLLM-based experiments with Llama 8B and 70B, the authors report up to 90% higher SLO attainment, up to 70% GPU savings, and up to 300% throughput gains over the Llumnix autoscaler. If correct, SLO-aware hierarchical autoscaling turns the spare capacity kept for interactive spikes into usable throughput for batch jobs.

What carries the argument

The central object is hierarchical backpressure, a pair of control signals that translate SLOs into autoscaling actions. Local backpressure is $B_{\mathrm{local}} = \max(\mathrm{ITL}/\mathrm{ITL\_SLO},\ T_{\mathrm{prev}}/T_{\mathrm{curr}})$, where ITL is the observed inter-token latency and $T$ is token throughput; when it exceeds one, Chiron halves the maximum batch size, and when it is below one the batch size is grown with an exponentially weighted moving average. Global backpressure combines interactive backpressure (the fraction of interactive-capable instances currently running interactive requests) with batch backpressure (the number of request groups whose estimated queue wait exceeds the time-to-first-token SLO). The queue wait for a request is estimated as $W_q = (\sum_{i=1}^{q-1} O_i)/\Theta$, with token-generation throughput $\Theta$ assumed constant and output token counts $O_i$ modeled as Normal by the Central Limit Theorem. Request groups are formed by clustering queued batch requests with similar TTFT SLOs, and this machinery turns SLO compliance into concrete triggers for adding or removing instances and for changing batch sizes.

What would settle it

Measure, on vLLM with Llama-70B, the actual token-generation throughput while the running batch changes size and composition, and compare it with the constant $\Theta$ assumed in the queue-wait formula. Then run Chiron on a workload where batch-request queues stay below a few hundred requests and check whether predicted waiting times match observed time-to-first-token deadlines; if the constant-throughput and Central Limit Theorem estimate is systematically off in that regime, the batch backpressure signal will add or withhold instances at the wrong times.

Watch

Extended reading notes

Core claim

Chiron's central claim is that a single autoscaler cannot satisfy mixed interactive/batch LLM workloads unless it couples local batch-size control with global instance-count control and uses SLO deadlines as the trigger for both. At the local level, the maximum batch size is adjusted online: it is halved whenever the observed inter-token latency exceeds its SLO or throughput drops, and increased with an exponentially weighted moving average toward the point where latency and throughput are balanced. At the cluster level, interactive instances are kept over-provisioned at a target ratio chosen from historical arrival bursts, and batch instances are added only when the estimated waiting time of queued request groups exceeds the time-to-first-token SLO, where waiting time is computed from token counts ahead divided by token-generation throughput. The paper reports that this combination achieves up to 90% higher SLO attainment and up to 70% better GPU efficiency than the previous Llumnix autoscaler on Llama 8B and 70B with real ShareGPT traces.

Load-bearing premise

The load-bearing premise is that a request's queue wait can be predicted from the number of output tokens ahead of it divided by a token-generation throughput that stays constant, and that those token counts follow a stable distribution; Chiron itself notes that this averaging breaks down for small queues, making the estimate conservative and possibly misleading for scaling decisions under light queue load.

Editorial extensions

If this is right

An operator can choose the interactive over-provisioning ratio from historical request-arrival spikes, and that ratio determines how much burstiness Chiron absorbs before SLO violations appear.
When batch queues are large enough for statistical averaging, Chiron adds several batch instances at once based on estimated deadlines, instead of growing capacity one instance at a time.
The local and global autoscalers each contribute roughly 30–60% throughput gains on their own, so the full reported improvement requires both levels.
Enabling prefix caching or speculative decoding changes the converged batch size; Chiron's online adaptation finds a smaller batch size that still preserves or improves end-to-end throughput.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension would be to make $\Theta$ in the waiting-time estimate depend on current batch composition and KV-cache state, which should improve batch backpressure accuracy in short queues where the paper's own goodness-of-fit data degrade.
The same interactive/mixed/batch instance split should transfer to disaggregated prefill–decode serving, with the mixed pool acting as a shared buffer between the prefill and decode phases.
Seeding the online local autoscaler with offline profile hints could reduce the observed multi-minute convergence times for large models while retaining adaptivity.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The core idea here is the combination of local batch-size autoscaling with SLO-aware global instance scaling and request queuing. That combination is genuinely new relative to Llumnix and Ray Serve, which scale on utilization and don't differentiate interactive from batch SLOs. Chiron's hierarchical backpressure concept is coherent, and the ablation shows both levels contribute 30–60% throughput gains. The evaluation against tuned and untuned Llumnix on Llama 8B/70B with ShareGPT workloads is directionally convincing and the reported gains are large.

What's soft, in order of severity. First, no code or data, and no error bars or repeated runs. The headline "up to" numbers are selected from favorable cases. That alone would make me hesitate to trust the quantitative claims. Second, the waiting-time estimator leans heavily on QLM, a self-cited prior paper, and assumes token generation throughput Θ is constant. The stress-test point is real: Chiron's own controllers change batch size and instance count, so Θ is endogenous. The paper doesn't discuss this. It could bias BBP in either direction, though the incremental scaling loop and relaxed batch SLOs give slack. This is a moderate modeling gap, not a fatal flaw; the algorithm still reacts to actual queue pressure over time. Third, several parameters (α, Θ, δ) are hand-set without sensitivity analysis beyond a few robustness plots.

I agree with the paper's own admission that small queues violate the CLT averaging assumption, making the estimator conservative in the regime where scaling decisions actually matter. That's honest but it means the central mechanism is least reliable when it's most needed.

Overall, this is a serious systems paper with a plausible design and a reasonable evaluation, but it is not independently verifiable as submitted. The right call is to send it to peer review and let a referee push for the artifact and repeated runs. The idea deserves that attention.

Referee Report

3 major / 6 minor

Summary. The paper presents Chiron, a hierarchical autoscaler for LLM serving that combines a local batch-size controller (Algorithm 1) with a global instance-level controller (Algorithm 2). The local controller adjusts the maximum batch size based on measured inter-token latency and throughput backpressure; the global controller maintains over-provisioning for interactive requests via interactive backpressure (IBP) and scales batch instances using batch backpressure (BBP), computed from queue waiting times estimated under the assumption of constant token-generation throughput and Normally distributed output-token counts (Eq. 1). The evaluation on vLLM with Llama 8B and 70B on up to 50 A100 GPUs reports up to 90% higher SLO attainment and up to 70% GPU-efficiency improvement over Llumnix baselines, with ablations showing that both hierarchical levels contribute.

Significance. If the empirical results are reproducible, Chiron advances the state of the art by demonstrating that SLO-aware queuing and dynamic batch-size control can outperform utilization-based autoscaling for mixed interactive/batch LLM workloads. The idea of using over-provisioned interactive capacity as a buffer for batch requests is timely and practical, and the paper includes useful robustness studies (varying SLO values, burstiness, prefix caching, speculative decoding) and an ablation that separates the local and global contributions, which are strengths. The contribution is, however, empirical and lacks formal analysis; the absence of released code/data and of error bars limits the strength of the claims, and the reliance on QLM's waiting-time estimator raises a correctness concern about the constant-Θ assumption that needs to be addressed before the results can be fully trusted.

major comments (3)

[Section 5.3, Eq. (1), Algorithm 2] Equation (1) models queue waiting time as Wq = Σ O_i / Θ with Θ treated as a constant token-generation throughput. In Chiron, Θ is not exogenous: Algorithm 1 changes the per-instance batch size and therefore the per-instance throughput, and Algorithm 2 changes the number of serving instances and therefore the aggregate throughput. Algorithm 2's while loop increments 'dispatch instances' and re-estimates Wg from the queue state without updating Θ for the added instances, so the BBP estimate is biased after each simulated scale-up. The paper's robustness discussion in Section 6.3 addresses CLT averaging for small queues (conservative estimates) but does not address this feedback loop. Because the number of instances added is driven directly by BBP, this bias affects the reported resource-efficiency and SLO results. Please either recompute Θ inside the simulation loop, use a closed-loop measurement of aggregate throughput, or provide a sensitivity analysis that quantifies the effect of this endogeneity on the scaling decisions and on the headline metrics.
[Section 6, Figures 9–18] All quantitative claims in Section 6 (Figures 9, 10, 14, 16–19, and the abstract's 'up to 90%' SLO attainment and 'up to 70%' GPU-efficiency numbers) are presented as point estimates with no confidence intervals, standard-deviation bars, or statement of the number of repeated runs. In a systems comparison against a per-workload-tuned baseline (Llumnix tuned), run-to-run variance can change the relative ordering, especially for SLO attainment near boundary arrival rates. The paper also does not release code, workload traces, or configuration files, which prevents independent verification. At minimum, the authors should report the distribution over multiple runs and make the artifacts available.
[Sections 4.2, 5.2, 5.3] The controller introduces several free parameters: the EWMA smoothing factor α (Algorithm 1), the over-provisioning target Θ and hysteresis margin δ (Section 5.2), and the output-token distribution moments μ_o and σ_o (Section 5.3). The evaluation fixes α = 0.5 and never reports δ or performs sensitivity analysis for α, Θ, or δ. Since the comparison in Figures 9 and 10 uses a Llumnix baseline that is tuned per workload, it is unclear whether Chiron's improvements are robust to reasonable variations in its own parameters or whether they depend on a favorable configuration. A parameter-sweep or a discussion of how these values are chosen in practice is needed to support the claim that Chiron is a practical pluggable autoscaler.

minor comments (6)

[Section 5.3, Eq. (1)] The summation 'Pq−1 i=1' is typeset incorrectly; the limits and index are malformed and should be written as Σ_{i=1}^{q−1} O_i.
[Section 6.3] The text refers to 'Figure 16' for the ITL SLO satisfaction results, but the content shown is a table, not a figure; the cross-reference should be updated (e.g., Table 1).
[Section 5.2, footnote 2] The sentence 'Θ 2 is set to 1/3' contains a stray '2' from the footnote marker; it should read 'Θ is set to 1/3.'
[Section 5.1] The definition of IBP as 'the ratio of instances running interactive requests to the total mixed and interactive instances' is ambiguous: it is unclear whether mixed instances that currently serve interactive requests count as 'running interactive requests' and whether interactive instances are always counted. Please define the ratio in terms of the three instance categories.
[Section 6.2] The statement that 'Chiron is able to handle a batch request queue of 700k requests and 80k requests' does not specify the performance target (e.g., the achieved SLO attainment level); please state the metric and threshold used for this claim.
[Section 6.3] The sentence 'We set the default level of over-provisioning as 3' is inconsistent with Section 5.2, where Θ is a ratio set to 1/3; please align the terminology (over-provisioning factor vs. target utilization ratio).

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: Chiron's headline gains are measured empirically, and the QLM-based waiting-time estimator is a control input that is restated and validated in the paper itself.

full rationale

The paper's central claims (up to 90% higher SLO attainment, up to 70% GPU savings, up to 300% throughput) are presented as experimental results from Section 6, not as consequences derived from the waiting-time model. Equation 1, Wq = sum(O_i)/Theta, is an estimator used to compute batch backpressure (BBP) in Algorithm 2; BBP in turn triggers instance scaling. This is a feedback-control input, not the quantity being predicted as a contribution. The output-token statistics (mu_o, sigma_o) are fitted from previous requests, but they are used to forecast queue waiting times, and the paper separately reports R^2 accuracy against actual queue behavior in Figure 14. The QLM citation (Patke et al., 2024) is same-author, but the paper describes the estimation approach 'for completeness' and evaluates it, so the design does not reduce to an unverified self-citation. The fixed-Theta assumption is a genuine modeling limitation: Chiron's own local and global scaling actions change Theta, so the estimate can be biased in feedback. That is a correctness/robustness concern, not circularity, because the measured SLO attainment and GPU efficiency are not defined in terms of the estimator. No equation is equivalent to its inputs by construction, no fitted parameter is renamed as a headline prediction, and no uniqueness theorem is imported from the authors' prior work.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central design rests on a small set of tuned parameters (EWMA smoothing, over-provisioning target, hysteresis margin, output-token statistics) and on domain assumptions about constant token throughput, mandatory over-provisioning, and preemption overhead. No new physical entities are introduced.

free parameters (4)

EWMA smoothing factor alpha = 0.5
Chosen by hand in Algorithm 1; controls how quickly batch size adapts. No sensitivity analysis is shown for alpha.
Over-provisioning target Theta = 1/3 over-provisioning ratio (or over-provisioning level 3 in robustness experiments)
Set from historical arrival spikes in Section 5.2: if the tail spike is 3x average arrival, Theta is set to 1/3. The default level in Figure 17 is 'over-provisioning as 3'. This directly determines when interactive/mixed instances are added or removed.
Hysteresis margin delta around Theta = unspecified
Footnote 2 states IBP is maintained in [Theta - delta, Theta + delta], but delta is never quantified. It affects churn and SLO attainment.
Output token distribution mean mu_o and std sigma_o = fitted from previous requests
Used in Eq. (1) for waiting time estimate Wq = sum(Oi)/Theta. The mean and standard deviation of output token counts are fitted from historical requests, as stated in Section 5.3.

assumptions (5)

domain assumption Token generation throughput Theta is constant throughout the generation process because of statistical averaging in continuous batching.
Underpins Eq. (1) for waiting time estimation. The paper admits small queues violate this, making estimates conservative.
domain assumption Output token counts of queued requests can be modeled by a Normal distribution for large queues via the Central Limit Theorem.
Needed to compute waiting time distributions in Eq. (1) and BBP in Eq. (2). The fitted mean and std are taken from previous requests.
domain assumption Interactive instances must be over-provisioned because model load time exceeds interactive TTFT SLO, leaving spare capacity that batch requests can exploit.
States in Section 2.3 and Appendix A.1 that over-provisioning is mandatory when load times exceed TTFT SLO. Chiron's multiplexing design depends on this spare capacity.
domain assumption Mixed instances can preempt batch requests and migrate their KV cache to CPU memory without throughput collapse.
Section 3 says interactive requests evict batch requests and fast restart is enabled by migrating KV cache to CPU memory, but no measurement of eviction overhead is provided.
domain assumption Interactive requests follow zero queuing while batch requests can tolerate queueing near their SLO deadline.
The global autoscaler separates request classes by SLO and relies on the ability to hold batch requests in a queue; this is a workload modeling assumption from production requirements.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Hierarchical Autoscaling for Large Language Model Serving with Chiron." pith.science (2026). https://pith.science/paper/SIBYRZFK

@misc{pith2026250108090,
  author       = {Pith},
  title        = {Pith review of: Hierarchical Autoscaling for Large Language Model Serving with Chiron},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SIBYRZFK}},
  note         = {Machine review of arXiv:2501.08090}
}

read the original abstract

Large language model (LLM) serving is becoming an increasingly important workload for cloud providers. Based on performance SLO requirements, LLM inference requests can be divided into (a) interactive requests that have tight SLOs in the order of seconds, and (b) batch requests that have relaxed SLO in the order of minutes to hours. These SLOs can degrade based on the arrival rates, multiplexing, and configuration parameters, thus necessitating the use of resource autoscaling on serving instances and their batch sizes. However, previous autoscalers for LLM serving do not consider request SLOs leading to unnecessary scaling and resource under-utilization. To address these limitations, we introduce Chiron, an autoscaler that uses the idea of hierarchical backpressure estimated using queue size, utilization, and SLOs. Our experiments show that Chiron achieves up to 90% higher SLO attainment and improves GPU efficiency by up to 70% compared to existing solutions.

Figures

Figures reproduced from arXiv: 2501.08090 by the authors.

**Figure 1.** Illustration comparing Chiron with previous systems. Chiron uses less instances (five for Chiron vs. three for previous systems) because of (a) global autoscaling based on queuing and request multiplexing, and (b) local autoscaling based on dynamic batch sizes. state-of-the-art serving systems such as vLLM use optimizations like continuous batching (Yu et al., 2022) and PagedAttention (Kwon et al., 2023) to impro… view at source ↗

**Figure 2.** Previously proposed LLM serving systems overestimate backpressure leading to cluster-wide underutilization. (Left) Cluster-wide utilization when serving a mix of batch and interactive requests for Llama 8B and Llama 70B. (Right) GPUs required to serve the workload across various autoscalers. “Local” and “Global” are Chiron’s autoscalers when used independently. varying SLOs when performing scaling actions. Conse… view at source ↗

**Figure 3.** Variation in inter-token latency and token throughput with increasing batch size. PagedAttention. Static allocation of the KV cache can result in significant memory waste as the KV cache grows dynamically during the decoding stage. PagedAttention (Kwon et al., 2023) introduces the idea of managing the KV cache, like OS memory, via pages and enabling dynamic allocation. Such dynamic allocation prevents fragmenta… view at source ↗

Figures from the paper (7 more)

**Figure 5.** Figure 5: Over-provisioning required for varying burstiness. 8B 70B 0.0 0.5 1.0 Normalized Metric Llumnix - Tput Request Groups - Tput Llumnix - Hysteresis Request Groups - Hysteresis [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]

**Figure 7.** Figure 7: Overview of Chiron. (i.e., horizontal scaling) via Global Autoscaler Lifecycle of a Request. All incoming requests are enqueued into a global queue and then served by the underlying LLM serving instances. Each serving instance is classified into one of three categor…

**Figure 8.** Figure 8: Token distribution in the ShareGPT dataset. Workloads. We create our experimental workloads from the requirements of a production cloud service provider except for request arrival rates due to confidentiality reasons. Request arrivals are modeled with a Poisson distri…

**Figure 9.** Figure 9: WA: Interactive workload with varying arrival rates for small model, large model, and mixed model configurations. We create our workloads using a combination of these requests as follows: [WA] Interactive-only Workload which consists of Interactive requests for small …

**Figure 11.** Figure 11: Batch size variation across different configurations for interactive requests. 8B 70B 8B+Prefix 70B+Prefix 70B+Spec 1 10 100 Convergence Time (s) Batch Interactive [PITH_FULL_IMAGE:figures/full_fig_p009_11.png]

**Figure 16.** Figure 16: ITL SLO satisfaction for varying values, with corresponding throughput and GPUs required for Llama 70B. Chiron uses a higher batch size. As batch requests have a relaxed ITL SLO, Chiron’s autoscaler sets a nearly 50× higher batch size (between 2048–4096) resulting in …

**Figure 19.** Figure 19: GPUs required over time for Chiron and Llumnix autoscalers when serving a batch and interactive workload for Llama 8B. over-provisioning and the global autoscaler reduces. However, batch size adaptation with the local autoscaler continues to remain useful. A.2 Exam…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 12 canonical work pages

[1]

APIServe: Efficient API support for large-language model inferencing

Reyna Abhyankar, Zijian He, Vikranth Srivatsa, Hao Zhang, and Yiying Zhang. APIServe: Efficient API support for large-language model inferencing. arXiv preprint arXiv:2402.01869,

arXiv
[5]

Serving DNNs like clockwork: Performance predictabil- ity from the bottom up

Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao, An- toine Kaufmann, Ymir Vigfusson, and Jonathan Mace. Serving DNNs like clockwork: Performance predictabil- ity from the bottom up. In Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2020), pages 443–462,

work page 2020
[6]

Cocktail: A multidi- mensional optimization for model serving in cloud

Jashwant Raj Gunasekaran, Cyan Subhra Mishra, Prashanth Thinakaran, Bikash Sharma, Mahmut Taylan Kandemir, and Chita R Das. Cocktail: A multidi- mensional optimization for model serving in cloud. In Proceedings of the 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 2022), pages 1041–1057,

work page 2022
[8]

Jiachen Liu, Zhiyu Wu, Jae-Won Chung, Fan Lai, Myungjin Lee, and Mosharaf Chowdhury

https://medium.com/@plienhar/ llm-inference-series-4-kv-caching-a-deeper-look-4ba9a77746c8 (Accessed on 04/10/2024). Jiachen Liu, Zhiyu Wu, Jae-Won Chung, Fan Lai, Myungjin Lee, and Mosharaf Chowdhury. An- des: Defining and enhancing quality-of-experience in LLM-based text streaming services. arXiv preprint arXiv:2404.16283,

arXiv 2024
[11]

One queue is all you need: Resolving head- of-line blocking in large language model serving

Archit Patke, Dhemath Reddy, Saurabh Jha, Hao- ran Qiu, Christian Pinto, Shengkun Cui, Chandra Narayanaswami, Zbigniew Kalbarczyk, and Ravis- hankar Iyer. One queue is all you need: Resolving head- of-line blocking in large language model serving. arXiv preprint arXiv:2407.00047,

arXiv
[12]

FIRM: An intel- ligent fine-grained resource management framework for SLO-oriented microservices

Haoran Qiu, Subho S Banerjee, Saurabh Jha, Zbigniew T Kalbarczyk, and Ravishankar K Iyer. FIRM: An intel- ligent fine-grained resource management framework for SLO-oriented microservices. In Proceedings of The 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2020),

work page 2020
[13]

INFaaS: Automated model-less in- ference serving

Francisco Romero, Qian Li, Neeraja J Yadwadkar, and Christos Kozyrakis. INFaaS: Automated model-less in- ference serving. In Proceedings of 2021 USENIX An- nual Technical Conference (ATC 2021), pages 397–411,

work page 2021
[14]

Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph E

https://huggingface.co/ datasets/anon8231489123/ShareGPT Vicuna unfiltered. Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph E. Gonzalez, and Ion Stoica. S-LoRA: Serving thousands of concur- rent LoRA adapters, 2023a. Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuo...

arXiv 2023

Show all 21 references

[15]

Accessed: 2024-04-10. tgi. Text Generation Inference. https://github.com/ huggingface/text-generation-inference,

2024
[16]

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Bap- tiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al

Accessed: 2024-04-10. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Bap- tiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971,

2024 arXiv
[17]

Yuxin Wang, Yuhan Chen, Zeyu Li, Zhenheng Tang, Rui Guo, Xin Wang, Qiang Wang, Amelie Chi Zhou, and Xi- aowen Chu

Accessed: 2024-04-10. Yuxin Wang, Yuhan Chen, Zeyu Li, Zhenheng Tang, Rui Guo, Xin Wang, Qiang Wang, Amelie Chi Zhou, and Xi- aowen Chu. Towards efficient and reliable LLM serving: A real-world workload study,

2024
[18]

Fast distributed infer- ence serving for large language models

Bingyang Wu, Yinmin Zhong, Zili Zhang, Gang Huang, Xuanzhe Liu, and Xin Jin. Fast distributed infer- ence serving for large language models. arXiv preprint arXiv:2305.05920, 2023a. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang,...

2022 arXiv
[20]

MArk: Exploiting cloud services for cost-effective, SLO-aware machine learning inference serving

Chengliang Zhang, Minchen Yu, Wei Wang, and Feng Yan. MArk: Exploiting cloud services for cost-effective, SLO-aware machine learning inference serving. In Pro- ceedings of 2019 USENIX Annual Technical Conference (ATC 2019), pages 1049–1062,

2019
[1967]

SpotServe: Serv- ing generative large language models on preemptible in- stances

Xupeng Miao, Chunan Shi, Jiangfei Duan, Xiaoli Xi, Dahua Lin, Bin Cui, and Zhihao Jia. SpotServe: Serv- ing generative large language models on preemptible in- stances. arXiv preprint arXiv:2311.15566,

arXiv
[2005]

Clipper: A low-latency online prediction serving system

Daniel Crankshaw, Xin Wang, Guilio Zhou, Michael J Franklin, Joseph E Gonzalez, and Ion Stoica. Clipper: A low-latency online prediction serving system. In Pro- ceedings of the 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 2017), pages 613–627,

2017
[2016]

Efficient memory management for large language model serving with PagedAttention

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Sys- tems Principles ...

2023
[2019]

Shepherd: Serving DNNs in the wild

Hong Zhang, Yupeng Tang, Anurag Khandelwal, and Ion Stoica. Shepherd: Serving DNNs in the wild. In Pro- ceedings of the 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 2023), pages 787–808, 2023a. Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen,...

2023
[2020]

ISBN 9781450381376

Association for Computing Machinery. ISBN 9781450381376. doi: 10.1145/3419111.3421285. Jiarui Fang, Yang Yu, Chengduo Zhao, and Jie Zhou. Tur- boTransformers: An efficient GPU serving system for transformer models. In Proceedings of the 26th ACM SIGPLAN Symposium on Principles...
[2022]

The shift from models to compound AI systems

Hierarchical Autoscaling for Large Language Model Serving Matei Zaharia, Omar Khattab, Lingjiao Chen, Jared Quincy Davis, Heather Miller, Chris Potts, James Zou, Michael Carbin, Jonathan Frankle, Naveen Rao, and Ali Ghodsi. The shift from models to compound AI systems. https:/...

2024
[2023]

TensorFlow-Serving: Flexible, high-performance ML serving

Christopher Olston, Fangwei Li, Jeremiah Harmsen, Jor- dan Soyke, Kiril Gorovoy, Li Lao, Noah Fiedel, Sukriti Ramesh, and Vinu Rajashekhar. TensorFlow-Serving: Flexible, high-performance ML serving. In Workshop on ML Systems at NIPS 2017,

2017
[2024]

On the opportunities and risks of foundation models

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Alt- man, Simran Arora, Sydney von Arx, Michael S Bern- stein, Jeannette Bohg, Antoine Bosselut, Emma Brun- skill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258,

arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

APIServe: Efficient API support for large-language model inferencing

Reyna Abhyankar, Zijian He, Vikranth Srivatsa, Hao Zhang, and Yiying Zhang. APIServe: Efficient API support for large-language model inferencing. arXiv preprint arXiv:2402.01869,

arXiv

[2] [5]

Serving DNNs like clockwork: Performance predictabil- ity from the bottom up

Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao, An- toine Kaufmann, Ymir Vigfusson, and Jonathan Mace. Serving DNNs like clockwork: Performance predictabil- ity from the bottom up. In Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2020), pages 443–462,

work page 2020

[3] [6]

Cocktail: A multidi- mensional optimization for model serving in cloud

Jashwant Raj Gunasekaran, Cyan Subhra Mishra, Prashanth Thinakaran, Bikash Sharma, Mahmut Taylan Kandemir, and Chita R Das. Cocktail: A multidi- mensional optimization for model serving in cloud. In Proceedings of the 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 2022), pages 1041–1057,

work page 2022

[4] [8]

Jiachen Liu, Zhiyu Wu, Jae-Won Chung, Fan Lai, Myungjin Lee, and Mosharaf Chowdhury

https://medium.com/@plienhar/ llm-inference-series-4-kv-caching-a-deeper-look-4ba9a77746c8 (Accessed on 04/10/2024). Jiachen Liu, Zhiyu Wu, Jae-Won Chung, Fan Lai, Myungjin Lee, and Mosharaf Chowdhury. An- des: Defining and enhancing quality-of-experience in LLM-based text streaming services. arXiv preprint arXiv:2404.16283,

arXiv 2024

[5] [11]

One queue is all you need: Resolving head- of-line blocking in large language model serving

Archit Patke, Dhemath Reddy, Saurabh Jha, Hao- ran Qiu, Christian Pinto, Shengkun Cui, Chandra Narayanaswami, Zbigniew Kalbarczyk, and Ravis- hankar Iyer. One queue is all you need: Resolving head- of-line blocking in large language model serving. arXiv preprint arXiv:2407.00047,

arXiv

[6] [12]

FIRM: An intel- ligent fine-grained resource management framework for SLO-oriented microservices

Haoran Qiu, Subho S Banerjee, Saurabh Jha, Zbigniew T Kalbarczyk, and Ravishankar K Iyer. FIRM: An intel- ligent fine-grained resource management framework for SLO-oriented microservices. In Proceedings of The 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2020),

work page 2020

[7] [13]

INFaaS: Automated model-less in- ference serving

Francisco Romero, Qian Li, Neeraja J Yadwadkar, and Christos Kozyrakis. INFaaS: Automated model-less in- ference serving. In Proceedings of 2021 USENIX An- nual Technical Conference (ATC 2021), pages 397–411,

work page 2021

[8] [14]

Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph E

https://huggingface.co/ datasets/anon8231489123/ShareGPT Vicuna unfiltered. Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph E. Gonzalez, and Ion Stoica. S-LoRA: Serving thousands of concur- rent LoRA adapters, 2023a. Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuo...

arXiv 2023