REVIEW 7 cited by
Multi-Candidate Speculative Decoding
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models have shown impressive capabilities across a variety of NLP tasks, yet their generating text autoregressively is time-consuming. One way to speed them up is speculative decoding, which generates candidate segments (a sequence of tokens) from a fast draft model that is then verified in parallel by the target model. However, the acceptance rate of candidate tokens receives limitations from several factors, such as the model, the dataset, and the decoding setup. This paper proposes sampling multiple candidates from a draft model and then organising them in batches for verification. We design algorithms for efficient multi-candidate verification while maintaining the distribution of the target model. Our approach shows significant improvements in acceptance rates on multiple datasets and models, consistently outperforming standard speculative decoding.
Forward citations
Cited by 7 Pith papers
-
Revisiting Lossy Verification in Speculative Decoding: Mechanisms, Trade-offs, and Failure Modes
Lossy speculative-decoding verification splits into truncation-based and collaborative methods; truncation-based methods underperform their matched baselines, and capping draft overshoot preserves quality.
-
Unlocking Parallelism in Autoregressive Language Models via Speculative Decoding with Progressive Tree Drafting
PTD accelerates autoregressive LLM decoding up to ~2× by guiding the target model to explore multiple coherent draft paths via a progressive, pruned tree in a single forward pass.
-
Mamba Drafters for Speculative Decoding
Mamba-based drafters can match self-speculation throughput with lower memory and cross-model flexibility.
-
Think Before You Accept: Semantic Reflective Verification for Faster Speculative Decoding
Reflective Verification fuses a target LLM's normal and reflection-prompted logits to accept semantically correct draft tokens, increasing accepted draft length and decoding speed by 5-15%.
-
Reinforcement Speculative Decoding for Fast Ranking
RSD uses reinforcement learning to train an agent that iteratively modifies an LLM's ranking under a fixed call budget, outperforming single-token and speculative-decoding baselines on IR and RS datasets.
-
SpecASR: Accelerating LLM-based Automatic Speech Recognition via Speculative Decoding
SpecASR accelerates LLM-based ASR by 3.04x-3.79x over autoregressive decoding using adaptive draft lengths, draft token recycling, and sparse token trees, but the speedups are simulated from Whisper proxy models rathe...
-
Consultant Decoding: Yet Another Synergistic Mechanism
Consultant Decoding speeds up LLM generation by accepting draft tokens whose negative log-likelihood under the target model falls below a fixed threshold, reaching 2-3x speedups with comparable quality.
Discussion (0). Continue with ORCID to comment.