REVIEW 9 cited by
Recurrent Drafter for Fast Speculative Decoding in Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We present Recurrent Drafter (ReDrafter), an advanced speculative decoding approach that achieves state-of-the-art speedup for large language models (LLMs) inference. The performance gains are driven by three key aspects: (1) leveraging a recurrent neural network (RNN) as the draft model conditioning on LLM's hidden states, (2) applying a dynamic tree attention algorithm over beam search results to eliminate duplicated prefixes in candidate sequences, and (3) training through knowledge distillation from the LLM. ReDrafter accelerates Vicuna inference in MT-Bench by up to 2.8x with a PyTorch implementation on Nvidia H100 GPUs. To demonstrate its practicality in real environments, we also validated its effectiveness for on-device applications by implementing the approach in MLX and benchmarking performance on Metal GPUs in Apple Silicon chips, achieving up to 2.3x speedup.
Forward citations
Cited by 9 Pith papers
-
Approximate Speculative Decoding
A training-free verifier, ASD, allows a bounded number of low-regret draft-target mismatches during speculative decoding and reuses the target-greedy suffix, yielding 3 to 15 percent throughput gains with mostly small...
-
DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation
DSpark's semi-autoregressive drafter plus load-aware confidence scheduling raises accepted draft length and shifts the production serving Pareto frontier by 60-85% higher per-user speed at matched throughput versus MTP-1.
-
MineDraft: A Framework for Batch Parallel Speculative Decoding
MineDraft speeds up speculative decoding by keeping two request batches — drafting one batch while verifying the other — gaining up to ~40% per-GPU throughput at moderate draft lengths.
-
Speculative Decoding with a Speculative Vocabulary
SpecVocab dynamically selects a per-step candidate vocabulary for the speculative-decoding draft model, beating static 32K-token vocabularies on acceptance length and throughput (up to +8.1% vs EAGLE-3).
-
Utility-Driven Speculative Decoding for Mixture-of-Experts
Cascade is a utility-driven speculation manager that makes speculative decoding practical for MoE LLMs by disabling it when expert-activation cost exceeds token gain and hill-climbing to the best speculation length.
-
Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potential
A gated LoRA and a small sampler let an autoregressive LLM draft multiple future tokens per step, and self-speculative verification converts those drafts into up to roughly 5x fewer generation steps.
-
WhisperKit: On-device Real-time ASR with Billion-Scale Transformers
WhisperKit's optimized on-device Whisper Large v3 Turbo streaming system reportedly achieves 0.46 s per-word latency and 2.2% WER, beating cloud baselines in its benchmark.
-
SpecASR: Accelerating LLM-based Automatic Speech Recognition via Speculative Decoding
SpecASR accelerates LLM-based ASR by 3.04x-3.79x over autoregressive decoding using adaptive draft lengths, draft token recycling, and sparse token trees, but the speedups are simulated from Whisper proxy models rathe...
-
Token-Operations-Oriented Inference Optimization Techniques for Large Models
The paper introduces a four-layer technical architecture for token-operations-oriented inference optimization in large models and reviews key technologies and industry status at each layer.
Discussion (0). Sign in to comment.