REVIEW 12 cited by
Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
As the usage of large language models (LLMs) grows, performing efficient inference with these models becomes increasingly important. While speculative decoding has recently emerged as a promising direction for speeding up inference, existing methods are limited in their ability to scale to larger speculation budgets, and adapt to different hyperparameters and hardware. This paper introduces Sequoia, a scalable, robust, and hardware-aware algorithm for speculative decoding. To attain better scalability, Sequoia introduces a dynamic programming algorithm to find the optimal tree structure for the speculated tokens. To achieve robust speculative performance, Sequoia uses a novel sampling and verification method that outperforms prior work across different decoding temperatures. Finally, Sequoia introduces a hardware-aware tree optimizer that maximizes speculative performance by automatically selecting the token tree size and depth for a given hardware platform. Evaluation shows that Sequoia improves the decoding speed of Llama2-7B, Llama2-13B, and Vicuna-33B on an A100 by up to $4.04\times$, $3.73\times$, and $2.27\times$. For offloading setting on L40, Sequoia achieves as low as 0.56 s/token for exact Llama2-70B inference latency, which is $9.96\times$ on our optimized offloading system (5.6 s/token), $9.7\times$ than DeepSpeed-Zero-Inference, $19.5\times$ than Huggingface Accelerate.
Forward citations
Cited by 12 Pith papers
-
Bole: Efficient Tree Speculation for Hybrid-Attention Language Models
Bole derives an exact parallel closed-form for tree-structured verification of gated delta linear attention, plus a GPU kernel and serving-runtime integration that improve hybrid-LLM decode throughput.
-
Beyond KV Reconstruction: Functional Reconstruction for MLA Draft Models in Speculative Decoding
A post-conversion training objective that reconstructs the original attention function of MLA-converted draft models improves speculative-decoding acceptance in most of 64 matched task cells.
-
Trees from Marginals: Autoregressive drafting with factorized priors
Weaver restores conditional dependencies on top-K factorized marginals to build high-acceptance draft trees, plus a fused GDN tree-verify kernel, yielding 4.37× AR speedup and 24.7% over DFlash.
-
AsymSpec: Efficient Cloud-Edge Speculative Decoding over Asymmetric Networks
AsymSpec speeds cloud-edge speculative decoding by uploading only token-probability pairs, moving correction traffic to the downlink, and overlapping independent requests instead of same-request runahead.
-
Approximate Speculative Decoding
A training-free verifier, ASD, allows a bounded number of low-regret draft-target mismatches during speculative decoding and reuses the target-greedy suffix, yielding 3 to 15 percent throughput gains with mostly small...
-
BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute
A routing system that chooses both the model and the number of samples per query to meet a quality threshold, yielding up to 60% cost savings.
-
POSS: Position Specialist Generates Better Draft for Speculative Decoding
Using position-specialized draft layers instead of one single draft model improves later-token acceptance in speculative decoding, yielding modest speedups on Llama-3-8B and Llama-2-13B.
-
CLaSp: In-Context Layer Skip for Self-Speculative Decoding
A training-free, context-adaptive layer-skipping method for self-speculative decoding that reports roughly 1.1x to 1.8x speedups on LLaMA models while preserving output distribution.
-
Ghidorah: Fast LLM Inference on Edge with Speculative Decoding and Hetero-Core Parallelism
On unified-memory edge hardware, Ghidorah partitions Medusa-style speculative decoding across CPU and GPU with all-column weight splits, sparse ARM kernels, and profile-based tuning, reporting up to 7.6x decode speedup.
-
Reinforcement Speculative Decoding for Fast Ranking
RSD uses reinforcement learning to train an agent that iteratively modifies an LLM's ranking under a fixed call budget, outperforming single-token and speculative-decoding baselines on IR and RS datasets.
-
AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding
AngelSpec + DFly pair a chat MTP drafter with a code/math block-diffusion drafter and load-aware verification pruning, reaching up to 2.4x AR throughput on Hy3-A21B.
-
SpecASR: Accelerating LLM-based Automatic Speech Recognition via Speculative Decoding
SpecASR accelerates LLM-based ASR by 3.04x-3.79x over autoregressive decoding using adaptive draft lengths, draft token recycling, and sparse token trees, but the speedups are simulated from Whisper proxy models rathe...
Discussion (0). Continue with ORCID to comment.