REVIEW 7 cited by
PEARL: Parallel Speculative Decoding with Adaptive Draft Length
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Speculative decoding (SD), where an extra draft model is employed to provide multiple draft tokens first, and then the original target model verifies these tokens in parallel, has shown great power for LLM inference acceleration. However, existing SD methods suffer from the mutual waiting problem, i.e., the target model gets stuck when the draft model is guessing tokens, and vice versa. This problem is directly incurred by the asynchronous execution of the draft model and the target model and is exacerbated due to the fixed draft length in speculative decoding. To address these challenges, we propose a conceptually simple, flexible, and general framework to boost speculative decoding, namely Parallel spEculative decoding with Adaptive dRaft Length (PEARL). Specifically, PEARL proposes pre-verify to verify the first draft token in advance during the drafting phase, and post-verify to generate more draft tokens during the verification phase. PEARL parallels the drafting phase and the verification phase via applying the two strategies, and achieves adaptive draft length for different scenarios, which effectively alleviates the mutual waiting problem. Experiments on various text generation benchmarks demonstrate the effectiveness of our PEARL, leading to a superior speed up performance up to 4.43$\times$ and 1.50$\times$, compared to auto-regressive decoding and vanilla speculative decoding, respectively. Our code is available at https://github.com/smart-lty/ParallelSpeculativeDecoding.
Forward citations
Cited by 7 Pith papers
-
AsymSpec: Efficient Cloud-Edge Speculative Decoding over Asymmetric Networks
AsymSpec speeds cloud-edge speculative decoding by uploading only token-probability pairs, moving correction traffic to the downlink, and overlapping independent requests instead of same-request runahead.
-
Unlocking Parallelism in Autoregressive Language Models via Speculative Decoding with Progressive Tree Drafting
PTD accelerates autoregressive LLM decoding up to ~2× by guiding the target model to explore multiple coherent draft paths via a progressive, pruned tree in a single forward pass.
-
Utility-Driven Speculative Decoding for Mixture-of-Experts
Cascade is a utility-driven speculation manager that makes speculative decoding practical for MoE LLMs by disabling it when expert-activation cost exceeds token gain and hill-climbing to the best speculation length.
-
SwiftSpec: Ultra-Low Latency LLM Decoding by Scaling Asynchronous Speculative Decoding
SwiftSpec uses asynchronous, disaggregated speculative decoding with parallel tree generation and fused kernels to speed up LLM decoding by 1.75x on average over baselines, reaching 348 tokens/s for Llama3-70B on 8 H800 GPUs.
-
POSS: Position Specialist Generates Better Draft for Speculative Decoding
Using position-specialized draft layers instead of one single draft model improves later-token acceptance in speculative decoding, yielding modest speedups on Llama-3-8B and Llama-2-13B.
-
AdaptiveSD A Stability-Aware, Runtime-Adaptive Speculative Decoding Framework with Multi-Policy Orchestration for CPU-Constrained LLM Inference
A runtime-adaptive speculative decoder with an 11-rule hierarchy and multi-policy engine keeps wasted draft compute under ~32% and bounds latency variance on CPU-constrained GGUF inference.
-
Advancing Decoding Strategies: Enhancements in Locally Typical Sampling for LLMs
ASTS extends locally typical sampling with semantic scoring and dynamic thresholds, reporting improved perplexity, MAUVE, and diversity on story and summarization tasks.
Discussion (0). Continue with ORCID to comment.