Pith. sign in

REVIEW 3 major objections 5 minor 56 references

SeaLLM: Service-Aware and Latency-Optimized Resource Sharing for Large Language Model Inference

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SeaLLM claims that a preemptive priority scheduler using profiled per-service execution times minimizes normalized latency when several LLMs share a GPU cluster, cutting normalized latency by up to 13.60x, tail latency by up to 18.69x…

desk verdict Solid systems paper with real 32-GPU gains and an honest theory-implementation gap; deserves peer review but needs code release and a proper justification for the DB scheduler. read the letter →

arxiv 2504.15720 v1 pith:GQRNY3FW submitted 2025-04-22 cs.DC

Yihao Zhao , Jiadun Chen , Peng Sun , Lei Li , Xuanzhe Liu , Xin Jin This is my paper · ORCID
classification cs.DC
keywords LLMinferenceservingresourcesharinglatency-optimizedschedulingpreemptiveprioritydoublingbudgetkey-valuecacheGPUclusternormalizedlatency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SeaLLM's central claim is that GPU clusters serving multiple large language models can be shared with much better latency if the scheduler exploits each service's profiled execution-time distribution instead of using generic first-come-first-serve or round-robin policies. The paper assigns every request a priority equal to its estimated remaining execution time multiplied by its service's average per-GPU execution time, and runs the smallest-priority request preemptively. It proves that this rule minimizes normalized latency under idealized assumptions, and adds a placement search, an adaptive replacement interval, and a unified KV cache to make the gains hold under real traffic. On real-world traces with four LLMs on a 32-GPU cluster, the authors report up to 13.60x lower normalized latency, up to 18.69x lower tail latency, and up to 3.64x higher SLO attainment against existing sharing systems. The conclusion is that multi-LLM sharing can be made latency-competitive, not just throughput-oriented.

What carries the argument

The central object is the priority rule $O_r = T'_r \hat{L}_r$ for preemptive request scheduling: a request's priority is its estimated remaining execution time multiplied by the profiled average per-GPU execution time of its LLM service, with smaller values served first. This is the c-$\mu$ rule specialized to minimizing normalized latency $L_N = \sum_{s}\sum_{r \in R_s} L_r / \hat{L}_s$, and it is carried by a doubling-budget mechanism: each request gets a budget $Q = \hat{L}_s + \mathrm{Var}_s$, and when that budget is exhausted the budget is doubled and the priority is reset. The placement controller, adaptive replacement interval, and unified KV cache are secondary machinery that keep the scheduling rule effective under dynamic traffic and heterogeneous model architectures.

What would settle it

Run SeaLLM on a trace whose output-length distribution is deliberately different from the profile used to set priorities (for example, profile on short outputs and then serve mostly long outputs), and compare normalized latency with FCFS and round-robin baselines; the central claim would be overturned if SeaLLM's advantage disappears or reverses once the output-length variance multiplier reaches the range (about 16x) where the paper's own accuracy study shows degradation.

Watch

Extended reading notes

Core claim

On the paper's own terms, SeaLLM's discovery is that the head-of-line blocking that plagues FCFS sharing of LLM services can be removed by a preemptive priority rule informed by service profiles, at the cost of preemptions only when a request exceeds its budget. The priority is $O_r = T'_r \hat{L}_r$, the product of the estimated remaining time and the profiled average per-GPU execution time, and the budget is $Q = \hat{L}_s + \mathrm{Var}_s$, doubled when exhausted. Theorem 1 states that with arbitrary preemption and identical per-service execution times this scheduling minimizes normalized latency; Theorem 2 states that the priority is optimal for expected normalized latency at every scheduling moment. The paper then shows how to keep that scheduling rule effective in practice: a two-stage placement search chooses sharing groups and parallelism, an adaptive replacement algorithm lengthens or shortens the placement interval according to the gap between estimated and achieved performance, and a unified KV cache with merged blocks lets LLMs with different block shapes share memory. The evaluation on real traces reports the normalized-latency, tail-latency, and SLO-attainment gains summarized above.

Load-bearing premise

The load-bearing premise is that the profiled per-service execution-time statistics (average and variance) predict live request behavior; if actual output lengths drift from the profile, the priority rule and budget are miscalibrated and the reported gains shrink.

Editorial extensions

If this is right

  • Under accurate profiling, the scheduling rule removes the head-of-line blocking of FCFS by letting short requests overtake long ones, with the doubling budget limiting how often long requests are preempted.
  • Bursty services no longer pay the memory and latency penalty of round-robin, because only requests that exhaust their budget are interrupted rather than rotating all services.
  • The adaptive replacement interval lets a cluster shift LLM services between GPU groups as request traffic changes, without an operator choosing a fixed interval.
  • At tested rates SeaLLM keeps SLO attainment above 90 percent while baselines drop, implying fewer GPUs are needed to meet the same service-level objectives.
  • Token-level time-to-first-token improves because prefill is not round-robinned away from bursty services; time-per-output-token stays comparable to dedicated serving.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to replace the profiled static estimate $T'_r$ with dynamic output-length predictions from the model itself (for example, from early-exit or speculative-decoding signals), which would sharpen priorities without changing the scheduling rule.
  • The same normalized-latency objective could be applied to other autoregressive workloads, such as multimodal generation, whose service-specific length distributions would feed the same priority and budget scheme.
  • The paper's identical-execution-time theorem suggests a direct lower-bound comparison: on traces with known output lengths, an offline shortest-remaining-processing-time schedule weighted by $\hat{L}_s$ should bracket SeaLLM's latency; the gap would quantify the cost of profiling error.
  • Because the benefit shrinks as output-length variance grows (the paper's own profiling-accuracy experiment), the scheduler would benefit from online re-profiling or anomaly detection that updates $\hat{L}_s$ and $\mathrm{Var}_s$ during a replacement interval.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. SeaLLM is a cluster-level system for sharing multiple LLM services on GPU clusters. The paper makes three main design contributions: a service-aware placement algorithm with adaptive replacement intervals, a preemptive 'doubling budget' (DB) request scheduler that uses profiled per-service execution characteristics, and a unified KV cache that lets different LLM architectures share GPU memory efficiently. The system is implemented on a 32-GPU testbed and evaluated against vLLM, AlpaServe, and MuxServe under real-world traces, with a simulator cross-checked against the testbed. The headline results are reductions in normalized latency by up to 13.60x, tail latency by up to 18.69x, and improvements in SLO attainment by up to 3.64x relative to baselines.

Significance. The problem addressed is important: multi-LLM serving on shared GPU clusters is increasingly common, and existing systems are either designed for classical models or optimize throughput at the expense of latency. SeaLLM's real-testbed evaluation on 32 GPUs with modern LLMs, its simulator cross-validation (Table 2 reports differences under 3%), and its ablation study separating scheduling, placement, and KV-cache effects are genuine strengths. The unified KV cache for heterogeneous architectures is a practical contribution, and the comparisons are against credible baselines. However, the paper's central theoretical claim that its scheduling algorithm minimizes normalized latency is not supported for the algorithm actually implemented: the DB scheduler differs structurally from the priority rule proven optimal, and the paper itself concedes that the distance from the optimal schedule is unknown. The empirical results may still stand, but the formal framing needs substantial revision.

major comments (3)
  1. [§4.2 and Algorithm 2 (Appendix A.2)] The implemented priority is not the priority proven optimal. Theorem 2 proves optimality of O_r = T'_r Lhat_s (Eq. 13), where T'_r is the estimated remaining execution time. Algorithm 2 instead uses O_r = Q_r Lhat_s, where Q_r is the remaining budget, initialized to Lhat_s + Var_s and decremented as the request runs. For two requests of the same service, this orders them essentially oppositely: a partially executed long request has a small Q_r and therefore higher priority, while a newly arrived short request starts with the full budget and lower priority. The paper's own statement in §4.2 that 'it is difficult to tell how far our scheduling decision is from the optimal scheduling decision' acknowledges this gap. Consequently Theorem 2 does not apply to the DB scheduler as implemented; the paper must either prove an optimality property for the DB rule or reframe the scheduler as a heuristic and adjust the abstract and Section 4.2 claims accordingly.
  2. [§4.2, Theorem 2 and Figure 13] The optimality claim is conditional on accurate knowledge of the remaining time T'_r, but at scheduling time the true output length of a running request is unknown. T'_r is therefore only a historical estimate derived from profiled per-service distributions. The paper acknowledges this limitation in Figure 13, which shows that SeaLLM's advantage shrinks as the variance of output length grows, yet the abstract and Section 4.2 present the minimization of normalized latency as an unconditional property. The formal statements and the central claims should explicitly state the profiling-accuracy condition and characterize how the guarantee degrades when the profile is wrong.
  3. [Appendix A.2, proof of Theorem 1] The proof of Theorem 1 considers only pairwise swaps of two individual requests and does not model the batched execution that the system actually performs, the memory constraints on the number of concurrently resident requests, or preemption only at iteration boundaries. Since the implemented scheduler batches same-phase requests of the same service and cannot preempt arbitrarily, the idealized model in Theorem 1 is not shown to describe the deployed system. The paper should either extend the proof to the batched, memory-constrained setting or explicitly state that Theorem 1 is an idealized bound that does not directly apply to the implementation.
minor comments (5)
  1. [Equation 1 and §6.1] Equation 1 defines normalized latency as a sum over requests, while Section 6.1 defines it as the average of end-to-end latency divided by average execution time. The two definitions differ by a factor of the number of requests; please align them and state which one is used in the experiments.
  2. [§4.2, budget definition] The budget is defined as Q = Lhat_s + Var_s, but Var_s is described as the 'standard variance' of execution time. If Var_s is a variance, the sum adds quantities with different units; if it is a standard deviation, the terminology and notation should be corrected for clarity.
  3. [Figures 6–13] The testbed and simulator results are presented as single curves without error bars, confidence intervals, or repeated-run information. Given the headline multipliers (13.60x, 18.69x), a small number of repeated runs with variance information would substantially strengthen the evaluation.
  4. [§4.3 and §6.1] The hyperparameters alpha, beta, I0, and the per-service starvation threshold are chosen as fixed values, but no sensitivity analysis is provided. Since these parameters affect placement and scheduling behavior, a brief study of their influence would improve reproducibility.
  5. [Table 2] The first column of Table 2 is labeled 'Baseline' but the rows list system names; consider renaming the column to 'System' for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the scheduling priority is derived from the external c-mu rule and the optimality proof is a self-contained pairwise exchange argument; profiled service characteristics are measured inputs rather than fitted outputs.

full rationale

The paper's load-bearing derivation is the scheduling priority in Section 4.2, Equation 7, and Theorems 1-2. Theorem 1 is proved in Appendix A.2 by a pairwise exchange argument comparing two candidate execution orders; the argument uses only the definition of normalized latency (Equation 1) and the profiled execution times, and does not assume the conclusion. Theorem 2 invokes Lemma 1, the c-mu rule, from external queueing-theory references [45, 46], with delay cost C_i(τ) = τ / Lhat_r matching the normalized-latency objective; the resulting priority O_r = T'_r Lhat_r is derived rather than imported from the authors' own prior work. The DB scheduling rule (Algorithm 2) uses profiled mean and variance as inputs, and the paper explicitly acknowledges that it may not reach the minimum normalized latency, so there is no fitted parameter being renamed as a prediction. The placement algorithm minimizes normalized latency in a simulator, but the end-to-end claims are measured on a real 32-GPU testbed against vLLM, AlpaServe, and MuxServe; the objective and metric coincide because that is the stated design goal, not a hidden circular step. Self-citations (FastServe, AlpaServe, LoongServe, dLoRA, ElasticFlow) appear as related work, baselines, or heuristics, but the central optimality argument does not rest on them. No equation in the paper reduces by construction to its own input. The gap between Theorem 2's T'_r-based priority and the implemented remaining-budget priority is a correctness or robustness concern, not a circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central scheduler rests on the c-mu rule from queueing theory (external) plus two idealized assumptions the paper acknowledges are violated in practice. The unified KV cache relies on a common hidden size across model families. Placement assumes intra-node tensor parallelism only. Hyperparameters alpha, beta, I0, and starvation thresholds are chosen by hand and not sensitivity-analyzed.

free parameters (5)
  • alpha (unserved index weight) = 0.0001
    Equation 6 weights normalized latency vs SLO violations; set manually in Section 6.1 without sensitivity analysis.
  • beta (replacement interval adjustment) = 0.1
    Equation 8 adjusts replacement interval; set manually in Section 6.1 without sensitivity analysis.
  • initial replacement interval I0 = unspecified
    Section 4.3 defines I0 as an initial hyper-parameter selected by the cluster manager; no value or tuning procedure is given.
  • profiled per-service execution time Lhat_s and variance Var_s = per-service values from traces (e.g., ShareGPT, LongBench, HumanEval)
    Section 4.2 uses these in the priority O_r = T'_r * Lhat_r and budget Q = Lhat_s + Var_s; they are estimated from historical data and their accuracy drives the scheduler's performance (Section 6.3).
  • starvation threshold per service = unspecified
    Section 4.2 says 'We set a starvation threshold for each service and serve the starved services first' but no threshold value or rule is specified.
assumptions (5)
  • domain assumption Requests can be preempted at any time (Assumption 1 of Theorem 1)
    Used to prove Theorem 1 in Appendix A.2; the paper notes it is approximate since preemption happens only at iteration boundaries.
  • domain assumption All requests of the same service have identical execution time (Assumption 2 of Theorem 1)
    Used to prove Theorem 1; the paper acknowledges this is not true in practice and relies on Theorem 2 for the general case.
  • standard math c-mu rule for priority queues (Lemma 1 from [45,46])
    Theorem 2's proof invokes the c-mu rule for the optimal policy of a single-server queue with linear delay costs.
  • domain assumption Hidden size is identical across shared LLMs (128 for Llamas and OPT >= 2.7B)
    Section 4.4 relies on identical hidden size for the merged unified KV cache; models with different hidden sizes would need padding or splitting, increasing fragmentation.
  • domain assumption Only tensor parallelism intra-node is considered; TP size is a power of two and at most the number of GPUs per node
    Section 4.1 restricts the placement search; cross-node parallelism is excluded.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SeaLLM: Service-Aware and Latency-Optimized Resource Sharing for Large Language Model Inference." pith.science (2026). https://pith.science/paper/GQRNY3FW

@misc{pith2026250415720,
  author       = {Pith},
  title        = {Pith review of: SeaLLM: Service-Aware and Latency-Optimized Resource Sharing for Large Language Model Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GQRNY3FW}},
  note         = {Machine review of arXiv:2504.15720}
}
abstract

Large language models (LLMs) with different architectures and sizes have been developed. Serving each LLM with dedicated GPUs leads to resource waste and service inefficiency due to the varying demand of LLM requests. A common practice is to share multiple LLMs. However, existing sharing systems either do not consider the autoregressive pattern of LLM services, or only focus on improving the throughput, which impairs the sharing performance, especially the serving latency. We present SeaLLM, which enables service-aware and latency-optimized LLM sharing. SeaLLM improves the overall sharing performance by (1) a latency-optimized scheduling algorithm utilizing the characteristics of LLM services, (2) a placement algorithm to determine the placement plan and an adaptive replacement algorithm to decide the replacement interval, and (3) a unified key-value cache to share GPU memory among LLM services efficiently. Our evaluation under real-world traces and LLM services demonstrates that SeaLLM improves the normalized latency by up to $13.60\times$, the tail latency by up to $18.69\times$, and the SLO attainment by up to $3.64\times$ compared to existing solutions.

Figures

Figures reproduced from arXiv: 2504.15720 by the authors.

Figure 1
Figure 1. The request traffic of four production LLM services [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Four execution strategies of serving two LLM ser [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The iteration time of the prefill phase and the decod [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: SeaLLM architecture. Placement controller. The placement controller generates the placement plan for all the LLM services in the cluster. A placement plan contains the used GPUs and the parallelism configuration for each LLM service. As the request traffic of LLM servi…
Figure 5
Figure 5. Figure 5: Comparison among (a) single LLM’s cache block, (b) split cache block, and (c) merged cache block. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Testbed experiments on MAF traces. 2 10 20 30 40 50 Avg rate (req/s) 0 400 800 Avg TTFT (s) vLLM AlpaServe MuxServe SeaLLM (a) Avg TTFT. 2 10 20 30 40 50 Avg rate (req/s) 0 0.05 0.1 0.15 Avg TPOT (s) vLLM AlpaServe MuxServe SeaLLM (b) Avg TPOT [PITH_FULL_IMAGE:figures…
Figure 7
Figure 7. Figure 7: Comparison of TTFT and TPOT. method for the KV cache. We use its open-sourced code, which is also built upon vLLM, for evaluation. 6.2 End-to-End Performance We first evaluate SeaLLM on a real cluster with 32 GPUs [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 9
Figure 9. Figure 9: Impact of the unified KV cache. 1 5 101520 30 40 50 SLO scale 0 50 100 SLO attainment (%) vLLM AlpaServe MuxServe SeaLLM [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 13
Figure 13. Figure 13: Impact of the profiling accuracy. they need clusters with more GPUs than SeaLLM to meet the SLO requirements. In other words, SeaLLM has better resource utilization and serving performance than baselines. Impact of the cluster density. To investigate the impact of the…
Figure 15
Figure 15. Figure 15: The overhead of the multi-LLM executor. there are longer requests under high variance and they can impair the performance greatly. The benefits of SeaLLM get smaller under a high variance of the output length because the inaccurate profiling information affects our sc…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 41 canonical work pages

  1. [1]

    https://github.com/NVIDIA/ FasterTransformer, 2019

    Fastertransformer. https://github.com/NVIDIA/ FasterTransformer, 2019

    work page 2019

  2. [2]

    https://grpc.io, 2021

    Grpc. https://grpc.io, 2021

    work page 2021

  3. [3]

    https://github.com/intel/ Multi-llms-Chatbot-CloudNative-LangChain , 2022

    Multi-llms chatbot on intel developer cloud. https://github.com/intel/ Multi-llms-Chatbot-CloudNative-LangChain , 2022

    work page 2022

  4. [4]

    https://docs.nvidia.com/ deploy/mps/index.html, 2022

    Multi-process service. https://docs.nvidia.com/ deploy/mps/index.html, 2022

    work page 2022

  5. [5]

    https://sharegpt.com/, 2023

    Sharegpt. https://sharegpt.com/, 2023

    work page 2023

  6. [6]

    https://github.com/NVIDIA/ TensorRT-LLM, 2023

    Tensorrt-llm. https://github.com/NVIDIA/ TensorRT-LLM, 2023

    work page 2023

  7. [7]

    https://github.com/ huggingface/text-generation-inference, 2023

    Text generation inference. https://github.com/ huggingface/text-generation-inference, 2023

    work page 2023

  8. [8]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

Show all 56 references
  1. [9]

    Sarathi: Efficient llm inference by piggy- backing decodes with chunked prefills

    Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S Gulavani, and Ramachan- dran Ramjee. Sarathi: Efficient llm inference by piggy- backing decodes with chunked prefills. arXiv preprint arXiv:2308.16369, 2023

    2023 arXiv

  2. [10]

    pfabric: Minimal near-optimal datacenter transport

    Mohammad Alizadeh, Shuang Yang, Milad Sharif, Sachin Katti, Nick McKeown, Balaji Prabhakar, and Scott Shenker. pfabric: Minimal near-optimal datacenter transport. ACM SIGCOMM Computer Communication Review, 2013

    2013

  3. [11]

    Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale

    Reza Yazdani Aminabadi, Samyam Rajbhandari, Am- mar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al. Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. In IEEE SC, 2022

    2022

  4. [12]

    Longbench: A bilingual, multi- task benchmark for long context understanding

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multi- task benchmark for long context understanding. arXiv preprint arXiv:2308.14508, 2023

    2023 arXiv

  5. [13]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NIPS, 2020

    2020

  6. [14]

    Round-robin syn- chronization: Mitigating communication bottlenecks in parameter servers

    Chen Chen, Wei Wang, and Bo Li. Round-robin syn- chronization: Mitigating communication bottlenecks in parameter servers. In IEEE INFOCOM, 2019

    2019

  7. [15]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  8. [16]

    Gonzalez, Ion Sto- ica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Sto- ica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023

    2023

  9. [17]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y . Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory- efficient exact attention with IO-awareness. In NIPS, 2022

    2022

  10. [18]

    Muxserve: Flexible spatial-temporal multiplex- ing for multiple llm serving

    Jiangfei Duan, Runyu Lu, Haojie Duanmu, Xiuhong Li, Xingcheng Zhang, Dahua Lin, Ion Stoica, and Hao Zhang. Muxserve: Flexible spatial-temporal multiplex- ing for multiple llm serving. In ICML, 2024

    2024

  11. [19]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  12. [20]

    Turbotransformers: an efficient gpu serving system for transformer models

    Jiarui Fang, Yang Yu, Chengduo Zhao, and Jie Zhou. Turbotransformers: an efficient gpu serving system for transformer models. In PPoPP, 2021

    2021

  13. [21]

    Elasticflow: An elastic server- less training platform for distributed deep learning

    Diandian Gu, Yihao Zhao, Yinmin Zhong, Yifan Xiong, Zhenhua Han, Peng Cheng, Fan Yang, Gang Huang, Xin Jin, and Xuanzhe Liu. Elasticflow: An elastic server- less training platform for distributed deep learning. In ASPLOS, 2023

    2023

  14. [22]

    Microsecond-scale preemption for concurrent gpu-accelerated dnn inferences

    Mingcong Han, Hanze Zhang, Rong Chen, and Haibo Chen. Microsecond-scale preemption for concurrent gpu-accelerated dnn inferences. In USENIX OSDI , 2022

    2022

  15. [23]

    Deepspeed-fastgen: High-throughput text generation for llms via mii and deepspeed-inference

    Connor Holmes, Masahiro Tanaka, Michael Wyatt, Am- mar Ahmad Awan, Jeff Rasley, Samyam Rajbhan- dari, Reza Yazdani Aminabadi, Heyang Qin, Arash Bakhtiari, Lev Kurilenko, et al. Deepspeed-fastgen: High-throughput text generation for llms via mii and deepspeed-inference. arXiv p...

    2024 arXiv

  16. [24]

    Flashdecod- ing++: Faster large language model inference with asyn- chronization, flat gemm optimization, and heuristics

    Ke Hong, Guohao Dai, Jiaming Xu, Qiuli Mao, Xiuhong Li, Jun Liu, Yuhan Dong, Yu Wang, et al. Flashdecod- ing++: Faster large language model inference with asyn- chronization, flat gemm optimization, and heuristics. In MLSys, 2024. 13

    2024

  17. [25]

    Inference without interfer- ence: Disaggregate llm inference for mixed downstream workloads

    Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, et al. Inference without interfer- ence: Disaggregate llm inference for mixed downstream workloads. arXiv preprint arXiv:2401.11181, 2024

    2024 arXiv

  18. [26]

    Gpipe: Effi- cient training of giant neural networks using pipeline parallelism

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Effi- cient training of giant neural networks using pipeline parallelism. In NIPS, 2019

    2019

  19. [27]

    Shinjuku: Preemptive scheduling for µsecond-scale tail latency

    Kostis Kaffes, Timothy Chong, Jack Tigar Humphries, Adam Belay, David Mazières, and Christos Kozyrakis. Shinjuku: Preemptive scheduling for µsecond-scale tail latency. In USENIX NSDI, 2019

    2019

  20. [28]

    Reducing activation recomputation in large transformer models

    Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. Reducing activation recomputation in large transformer models. In MLSys, 2023

    2023

  21. [29]

    Gonza- lez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonza- lez, Hao Zhang, and Ion Stoica. Efficient memory man- agement for large language model serving with pagedat- tention. In ACM SOSP, 2023

    2023

  22. [30]

    Sequence parallelism: Long sequence training from system perspective

    Shenggui Li, Fuzhao Xue, Chaitanya Baranwal, Yongbin Li, and Yang You. Sequence parallelism: Long sequence training from system perspective. In ACL, 2023

    2023

  23. [31]

    Competition-level code generation with alphacode

    Yujia Li, David Choi, Junyoung Chung, Nate Kush- man, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Sci- ence, 2022

    2022

  24. [32]

    Alpaserve: Sta- tistical multiplexing with model parallelism for deep learning serving

    Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E Gonzalez, et al. Alpaserve: Sta- tistical multiplexing with model parallelism for deep learning serving. In USENIX OSDI, 2023

    2023

  25. [33]

    Terapipe: Token-level pipeline parallelism for training large-scale language models

    Zhuohan Li, Siyuan Zhuang, Shiyuan Guo, Danyang Zhuo, Hao Zhang, Dawn Song, and Ion Stoica. Terapipe: Token-level pipeline parallelism for training large-scale language models. In ICML, 2021

    2021

  26. [34]

    Zico: Efficient gpu memory sharing for concurrent dnn training

    Gangmuk Lim, Jeongseob Ahn, Wencong Xiao, Youngjin Kwon, and Myeongjae Jeon. Zico: Efficient gpu memory sharing for concurrent dnn training. In USENIX ATC, 2021

    2021

  27. [35]

    Pipedream: Gen- eralized pipeline parallelism for dnn training

    Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. Pipedream: Gen- eralized pipeline parallelism for dnn training. In ACM SOSP, 2019

    2019

  28. [36]

    Splitwise: Efficient generative llm inference using phase splitting

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative llm inference using phase splitting. In IEEE ISCA, 2024

    2024

  29. [37]

    Efficiently scal- ing transformer inference

    Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scal- ing transformer inference. In MLSys, 2023

    2023

  30. [38]

    Zero: Memory optimizations toward train- ing trillion parameter models

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward train- ing trillion parameter models. In SC, 2020

    2020

  31. [39]

    Serverless in the wild: Characterizing and optimizing the serverless workload at a large cloud provider

    Mohammad Shahrad, Rodrigo Fonseca, Inigo Goiri, Go- har Chaudhry, Paul Batum, Jason Cooke, Eduardo Lau- reano, Colby Tresness, Mark Russinovich, and Ricardo Bianchini. Serverless in the wild: Characterizing and optimizing the serverless workload at a large cloud provider. In U...

    2020

  32. [40]

    Megatron-lm: Training multi-billion parameter lan- guage models using model parallelism

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter lan- guage models using model parallelism. arXiv preprint arXiv:1909.08053, 2019

    1909 arXiv

  33. [41]

    Llm-planner: Few-shot grounded planning for embodied agents with large language models

    Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. Llm-planner: Few-shot grounded planning for embodied agents with large language models. In ICCV, 2023

    2023

  34. [42]

    Dynamollm: Designing llm inference clusters for performance and energy efficiency

    Jovan Stojkovic, Chaojie Zhang, Íñigo Goiri, Josep Tor- rellas, and Esha Choukse. Dynamollm: Designing llm inference clusters for performance and energy efficiency. arXiv preprint arXiv:2408.00741, 2024

    2024

  35. [43]

    Llumnix: Dynamic scheduling for large language model serving

    Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. Llumnix: Dynamic scheduling for large language model serving. In OSDI, 2024

    2024

  36. [44]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  37. [45]

    Finding optimal policy for queueing models: New pa- rameterization

    Trang H Tran, Lam M Nguyen, and Katya Scheinberg. Finding optimal policy for queueing models: New pa- rameterization. arXiv preprint arXiv:2206.10073, 2022

    2022 arXiv

  38. [46]

    Dynamic scheduling with convex delay costs: The generalized c| mu rule

    Jan A Van Mieghem. Dynamic scheduling with convex delay costs: The generalized c| mu rule. The Annals of Applied Probability, 1995. 14

    1995

  39. [47]

    Lightseq: A high performance inference library for transformers

    Xiaohui Wang, Ying Xiong, Yang Wei, Mingxuan Wang, and Lei Li. Lightseq: A high performance inference library for transformers. arXiv preprint arXiv:2010.13887, 2020

    2010 arXiv

  40. [48]

    Towards efficient and reliable llm serving: A real-world workload study

    Yuxin Wang, Yuhan Chen, Zeyu Li, Zhenheng Tang, Rui Guo, Xin Wang, Qiang Wang, Amelie Chi Zhou, and Xiaowen Chu. Towards efficient and reliable llm serving: A real-world workload study. arXiv preprint arXiv:2401.17644, 2024

    2024 arXiv

  41. [49]

    Loongserve: Efficiently serv- ing long-context large language models with elastic se- quence parallelism

    Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, and Xin Jin. Loongserve: Efficiently serv- ing long-context large language models with elastic se- quence parallelism. arXiv preprint arXiv:2404.09526, 2024

    2024 arXiv

  42. [50]

    Fast distributed inference serving for large language models

    Bingyang Wu, Yinmin Zhong, Zili Zhang, Gang Huang, Xuanzhe Liu, and Xin Jin. Fast distributed inference serving for large language models. arXiv preprint arXiv:2305.05920, 2023

    2023 arXiv

  43. [51]

    dLoRA: Dynamically orches- trating requests and adapters for LoRA LLM serving

    Bingyang Wu, Ruidong Zhu, Zili Zhang, Peng Sun, Xu- anzhe Liu, and Xin Jin. dLoRA: Dynamically orches- trating requests and adapters for LoRA LLM serving. In USENIX OSDI, 2024

    2024

  44. [52]

    Antman: Dynamic scaling on gpu clusters for deep learning

    Wencong Xiao, Shiru Ren, Yong Li, Yang Zhang, Pengyang Hou, Zhi Li, Yihui Feng, Wei Lin, and Yangqing Jia. Antman: Dynamic scaling on gpu clusters for deep learning. In USENIX OSDI, 2020

    2020

  45. [53]

    Orca: A distributed serving system for Transformer-Based generative mod- els

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soo- jeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for Transformer-Based generative mod- els. In USENIX OSDI, 2022

    2022

  46. [54]

    Opt: Open pre-trained transformer language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022

    2022 arXiv

  47. [55]

    Multi- resource interleaving for deep learning training

    Yihao Zhao, Yuanqiang Liu, Yanghua Peng, et al. Multi- resource interleaving for deep learning training. InACM SIGCOMM, 2022

    2022

  48. [56]

    Dist- serve: Disaggregating prefill and decoding for goodput- optimized large language model serving

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. Dist- serve: Disaggregating prefill and decoding for goodput- optimized large language model serving. In USENIX OSDI, 2024. A Appendix A.1 Placement Algorithm Algorithm 1 shows the...

    2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.