Pith. sign in

REVIEW 5 cited by

MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2404.02015 v2 pith:O6TMD6NZ submitted 2024-04-02 cs.DC

Jiangfei Duan , Runyu Lu , Haojie Duanmu , Xiuhong Li , Xingcheng Zhang , Dahua Lin , Ion Stoica , Hao Zhang This is my paper · ORCID
classification cs.DC
keywords muxservellmsmultiplexingflexiblemultipleservingcolocateefficient
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Large language models (LLMs) have demonstrated remarkable performance, and organizations are racing to serve LLMs of varying sizes as endpoints for use-cases like chat, programming and search. However, efficiently serving multiple LLMs poses significant challenges for existing approaches due to varying popularity of LLMs. In the paper, we present MuxServe, a flexible spatial-temporal multiplexing system for efficient multiple LLM serving. The key insight behind is to colocate LLMs considering their popularity to multiplex memory resources, and leverage the characteristics of prefill and decoding phases to separate and flexibly colocate them to multiplex computation resources. MuxServe formally formulates the multiplexing problem, and proposes a novel placement algorithm and adaptive batch scheduling strategy to identify optimal colocations and maximize utilization. MuxServe designs a unified resource manager to enable flexible and efficient multiplexing. Evaluation results show that MuxServe can achieves up to $1.8\times$ higher throughput or processes $2.9\times$ more requests within $99\%$ SLO attainment. The code is available at: \url{https://github.com/hao-ai-lab/MuxServe}.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Elastic Gang: Per-Token Membership Change for a Hard-Barriered LLM Inference Gang Co-Scheduled with OS Processes

    cs.OS 2026-07 conditional novelty 7.0 of 10

    An ACK-latched epoch protocol lets a hard-barriered CPU LLM inference gang change membership per token bit-exactly and Pareto-dominate static core partitions on general throughput.

  2. CoCoScale: Leveraging Layer-wise Scaling to Unlock the Potential of Online LLM Serving

    cs.DC 2026-07 conditional novelty 6.5 of 10

    Layer-wise data-parallel replication of hot Transformer layers onto reclaimed idle GPUs reduces LLM serving cold-start latency 97.9–99.3% and average latency 20.7–28.1% while attaining 100% SLO on production traces.

  3. From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving

    cs.DC 2026-06 accept novelty 6.5 of 10

    KV-cache serving systems concentrate into five archetypes under a four-axis taxonomy, with ownership explaining residual distributed design variance and seven measurement gaps blocking next steps.

  4. InstantInfer: Enabling Fast LLM Cold Start with Communicating Finite Automata

    cs.DC 2026-07 conditional novelty 5.0 of 10

    InstantInfer refactors vLLM's cold start into a concurrent state-machine pipeline, speeding up startup by up to 7.2×.

  5. On Evaluating Performance of LLM Inference Serving Systems

    cs.LG 2025-07 conditional novelty 5.0 of 10

    A systematic review identifies eight anti-patterns in LLM inference evaluation and proposes a checklist, with a speculative decoding case study demonstrating how conventional metrics mislead.

Pith tools