REVIEW 7 cited by
TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodput
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large Language Model (LLM) serving systems batch concurrent user requests to achieve efficient serving. However, in real-world deployments, such inter-request parallelism from batching is often limited by external factors such as low request rates or memory constraints. Recent works focus on intra-request parallelism from speculative decoding as a solution to this problem. Unfortunately, benefits from intra-request parallelism are often fragile, as speculative decoding causes overhead, and speculated tokens may miss. We observe that speculative decoding may degrade LLM serving performance if added naively without tuning to the incoming requests and the speculation method. To alleviate the need for expert tuning and make speculative decoding more robust, we present TurboSpec, a speculation control system that automatically profiles the execution environment and utilizes a feedback-based algorithm to dynamically adjust the amount of intra-request parallelism in LLM serving. TurboSpec predicts "goodput" - the amount of successfully generated tokens - to evaluate and adjust intra-request parallelism amount to that with the highest goodput in runtime. We implement TurboSpec on a real-world LLM serving system vLLM and demonstrate its effectiveness across diverse workloads and hardware configurations, providing consistent performance improvements across all test scenarios.
Forward citations
Cited by 7 Pith papers
-
DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs
DeltaServe co-serves LoRA fine-tuning with LLM inference on existing engines (vLLM, SGLang, S-LoRA) through host-agnostic hooks and an SLO-aware scheduler, achieving 2.9x the fine-tuning throughput of LLMStation at 10...
-
DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation
DSpark's semi-autoregressive drafter plus load-aware confidence scheduling raises accepted draft length and shifts the production serving Pareto frontier by 60-85% higher per-user speed at matched throughput versus MTP-1.
-
MineDraft: A Framework for Batch Parallel Speculative Decoding
MineDraft speeds up speculative decoding by keeping two request batches — drafting one batch while verifying the other — gaining up to ~40% per-GPU throughput at moderate draft lengths.
-
Utility-Driven Speculative Decoding for Mixture-of-Experts
Cascade is a utility-driven speculation manager that makes speculative decoding practical for MoE LLMs by disabling it when expert-activation cost exceeds token gain and hill-climbing to the best speculation length.
-
SwiftSpec: Ultra-Low Latency LLM Decoding by Scaling Asynchronous Speculative Decoding
SwiftSpec uses asynchronous, disaggregated speculative decoding with parallel tree generation and fused kernels to speed up LLM decoding by 1.75x on average over baselines, reaching 348 tokens/s for Llama3-70B on 8 H800 GPUs.
-
What makes Reasoning Models Different? Follow the Reasoning Leader for Efficient Decoding
FoReaL-Decoding lets a strong reasoning model generate the first few tokens of each sentence and a weaker model complete the sentence, cutting theoretical FLOPs by 30-55% while retaining 86-100% of accuracy on four ma...
-
Efficient Speculative Decoding for Llama at Scale: Challenges and Solutions
Meta reports EAGLE-based speculative decoding optimizations for Llama models, achieving state-of-the-art latency (about 4 ms/token for Llama4 Maverick) and 1.4-2.0x speedups for large batches.
Discussion (0). Sign in to comment.