REVIEW 9 cited by
DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The deployment and scaling of large language models (LLMs) have become critical as they permeate various applications, demanding high-throughput and low-latency serving systems. Existing frameworks struggle to balance these requirements, especially for workloads with long prompts. This paper introduces DeepSpeed-FastGen, a system that employs Dynamic SplitFuse, a novel prompt and generation composition strategy, to deliver up to 2.3x higher effective throughput, 2x lower latency on average, and up to 3.7x lower (token-level) tail latency, compared to state-of-the-art systems like vLLM. We leverage a synergistic combination of DeepSpeed-MII and DeepSpeed-Inference to provide an efficient and easy-to-use serving system for LLMs. DeepSpeed-FastGen's advanced implementation supports a range of models and offers both non-persistent and persistent deployment options, catering to diverse user scenarios from interactive sessions to long-running applications. We present a detailed benchmarking methodology, analyze the performance through latency-throughput curves, and investigate scalability via load balancing. Our evaluations demonstrate substantial improvements in throughput and latency across various models and hardware configurations. We discuss our roadmap for future enhancements, including broader model support and new hardware backends. The DeepSpeed-FastGen code is readily available for community engagement and contribution.
Forward citations
Cited by 9 Pith papers
-
From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving
KV-cache serving systems concentrate into five archetypes under a four-axis taxonomy, with ownership explaining residual distributed design variance and seven measurement gaps blocking next steps.
-
Beyond Storage: State as a Runtime Control Problem in Parallel and Distributed Systems
State management across streaming, serving, retrieval, and learning systems is best modeled as one coupled runtime control loop spanning access, execution, and evolution.
-
BlockServe: Block-Grained Continuous Batching for High-Throughput Diffusion LLM Serving
BlockServe raises offline diffusion-LLM batch throughput 1.9–10.6× over Fast-dLLM by block-boundary eviction, mixed-state gather-scatter execution, and token-budget admission.
-
HAP: Hybrid Adaptive Parallelism for Efficient Mixture-of-Experts Inference
HAP uses ILP over module-specific hybrid parallel choices to speed up MoE inference, reporting up to 1.77x versus tensor parallelism on tested GPUs.
-
Rethinking Caching for LLM Serving Systems: Beyond Traditional Heuristics
SISO caches representative centroid embeddings of similar queries and adjusts its similarity threshold under load, improving semantic cache hit ratios for LLM serving by up to 1.71x with a small accuracy cost.
-
Rectified Sparse Attention
ReSA keeps sparse decoding accurate by refreshing the KV cache with dense attention every f tokens, achieving near-lossless quality with a large speedup.
-
Memory Offloading for Large Language Model Inference with Latency SLO Guarantees
Select-N meets LLM latency SLOs by choosing and continuously adjusting an offloading interval that controls how much model state lives in CPU memory, increasing host memory usage and throughput versus prior offloading...
-
Chat-Ghosting: A Comparative Study of Methods for Auto-Completion in Dialog Systems
Simple tries and n-gram models beat large neural models for chat autocompletion on seen prefixes, while fine-tuned transformers and conversational context lead on unseen ones.
-
AnchorAttention: Difference-Aware Sparse Attention with Stripe Granularity
AnchorAttention uses the maximum attention score from initial and local tokens as an anchor to threshold-select important key-value positions at stripe granularity, achieving faster prefill with comparable accuracy.
Discussion (0). Continue with ORCID to comment.