Pith. sign in

REVIEW 4 cited by

AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2501.12162 v2 pith:MFS6ILLF submitted 2025-01-21 cs.CL cs.AIcs.DCcs.LG

Zikun Li , Zhuofu Chen , Remi Delacourt , Gabriele Oliaro , Zeyu Wang , Qinghan Chen , Shuhuai Lin , April Yang
show 6 more authors
Zhihao Zhang Zhuoming Chen Sean Lai Xinhao Cheng Xupeng Miao Zhihao Jia
This is my paper
classification cs.CLcs.AIcs.DCcs.LG
keywords servingadaservemulti-slodecodingdiverseslo-customizedslosspeculation
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Modern large language model (LLM) applications exhibit diverse service-level objectives (SLOs), from low-latency requirements in interactive coding assistants to more relaxed constraints in data wrangling tasks. Existing LLM serving systems, which rely on uniform batching and scheduling strategies, often fail to meet these heterogeneous SLOs concurrently. We present AdaServe, the first LLM serving system designed to support efficient multi-SLO serving through SLO-customized speculative decoding. AdaServe formulates multi-SLO serving as a constrained optimization problem and introduces a hardware-aware algorithm that constructs a speculation tree tailored to each request's latency target. It features a speculate-select-verify pipeline that enables fine-grained control over decoding speed while maximizing system throughput. AdaServe further adapts to workload variation by dynamically adjusting speculation parameters. Evaluations across diverse workloads show that AdaServe reduces SLO violations by up to 4.3$\times$ and improves goodput by up to 1.9$\times$ compared to the best performing baselines, highlighting its effectiveness in multi-SLO serving.

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs

    cs.DC 2026-07 conditional novelty 6.0 of 10

    DeltaServe co-serves LoRA fine-tuning with LLM inference on existing engines (vLLM, SGLang, S-LoRA) through host-agnostic hooks and an SLO-aware scheduler, achieving 2.9x the fine-tuning throughput of LLMStation at 10...

  2. MineDraft: A Framework for Batch Parallel Speculative Decoding

    cs.CL 2026-02 conditional novelty 6.0 of 10

    MineDraft speeds up speculative decoding by keeping two request batches — drafting one batch while verifying the other — gaining up to ~40% per-GPU throughput at moderate draft lengths.

  3. Utility-Driven Speculative Decoding for Mixture-of-Experts

    cs.DC 2025-06 conditional novelty 6.0 of 10

    Cascade is a utility-driven speculation manager that makes speculative decoding practical for MoE LLMs by disabling it when expert-activation cost exceeds token gain and hill-climbing to the best speculation length.

  4. Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challenges

    cs.DC 2025-07 conditional novelty 4.0 of 10

    A survey that builds a taxonomy of edge-cloud LLM-SLM collaboration for inference and training, claiming to be the first to unify both phases.

Pith tools