REVIEW 4 cited by
SLO-Aware Scheduling for Large Language Model Inferences
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models (LLMs) have revolutionized applications such as code completion, chatbots, and online classification. To elevate user experiences, service level objectives (SLOs) serve as crucial benchmarks for assessing inference services capabilities. In practice, an inference service processes multiple types of tasks, each with its own distinct SLO. To ensure satisfactory user experiences, each request's distinct SLOs should be considered in scheduling. However, existing designs lack this consideration, leading to insufficient hardware utility and suboptimal performance. This paper analyzes scenarios to process tasks with varying SLOs, and introduces a simulated annealing-based scheduler to decide request priority sequence based on a request's SLO, input lengths, and possible output lengths. As the first specialized scheduler for multi-SLO scenarios, this work improves SLO attainment by up to 5x and reduces average latency by 31.6% on Python-Code-23k-ShareGPT and ShareGPT_Vicuna_unfiltered datasets, compared to current state-of-the-art framework vLLM and a new framework LMDeploy.
Forward citations
Cited by 4 Pith papers
-
Beyond Accuracy and Cost: Latency-Aware LLM Query Routing for Dynamic Workloads
A serving-framework simulation that predicts time-to-first-token lets LLM routers jointly optimize accuracy, cost, and latency, improving accuracy-cost utility by up to 40% at matched latency.
-
MaaSO: SLO-aware Orchestration of Heterogeneous Model Instances for MaaS
MaaSO assigns different parallelism strategies and batch sizes to LLM instances and routes requests by deadline, improving simulated SLO attainment by 15 to 30 percent.
-
Rethinking Caching for LLM Serving Systems: Beyond Traditional Heuristics
SISO caches representative centroid embeddings of similar queries and adjusts its similarity threshold under load, improving semantic cache hit ratios for LLM serving by up to 1.71x with a small accuracy cost.
-
An Inquiry into Datacenter TCO for LLM Inference with FP8
Measured FP8 thin-GEMM throughput shows Intel Gaudi 2/3 outperform NVIDIA H100/H200 on LLM decode-style workloads, making workload-level benchmarks more decisive than peak TFLOPS for datacenter TCO.
Discussion (0). Continue with ORCID to comment.