REVIEW 6 cited by
SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Speculative decoding (SD) has emerged as a widely used paradigm to accelerate LLM inference without compromising quality. It works by first employing a compact model to draft multiple tokens efficiently and then using the target LLM to verify them in parallel. While this technique has achieved notable speedups, most existing approaches necessitate either additional parameters or extensive training to construct effective draft models, thereby restricting their applicability across different LLMs and tasks. To address this limitation, we explore a novel plug-and-play SD solution with layer-skipping, which skips intermediate layers of the target LLM as the compact draft model. Our analysis reveals that LLMs exhibit great potential for self-acceleration through layer sparsity and the task-specific nature of this sparsity. Building on these insights, we introduce SWIFT, an on-the-fly self-speculative decoding algorithm that adaptively selects intermediate layers of LLMs to skip during inference. SWIFT does not require auxiliary models or additional training, making it a plug-and-play solution for accelerating LLM inference across diverse input data streams. Our extensive experiments across a wide range of models and downstream tasks demonstrate that SWIFT can achieve over a 1.3x-1.6x speedup while preserving the original distribution of the generated text. We release our code in https://github.com/hemingkx/SWIFT.
Forward citations
Cited by 6 Pith papers
-
Dynamic Execution Commitment of Vision-Language-Action Models
A3 reframes dynamic action chunk commitment in VLA models as self-speculative prefix verification, accepting the longest continuous sequence of actions that satisfies consensus-ordered conditional invariance and prefi...
-
SpecEyes: Accelerating Agentic Multimodal LLMs via Speculative Perception and Planning
A small tool-free MLLM plus answer-separability gating bypasses agentic tool loops for many queries, yielding 1.1–3.35× speedup with preserved or higher accuracy.
-
KnapSpec: Self-Speculative Decoding via Adaptive Layer Selection as a Knapsack Problem
By modeling layer skipping as a knapsack problem with context-dependent attention/MLP latencies, KnapSpec adaptively selects draft sub-networks that speed up self-speculative decoding by up to 1.47×.
-
HiSpec: Hierarchical Speculative Decoding for LLMs
HiSpec uses early-exit layers to verify draft tokens midway through the model, reporting 1.28×-2.01× faster decoding over baseline speculative decoding, but the accuracy claim is not empirically tested.
-
CLaSp: In-Context Layer Skip for Self-Speculative Decoding
A training-free, context-adaptive layer-skipping method for self-speculative decoding that reports roughly 1.1x to 1.8x speedups on LLaMA models while preserving output distribution.
-
MixReasoning: Switching Modes to Think
A reasoning model can cut token usage by roughly a third to a half without losing accuracy if it switches to brief output on low-uncertainty steps and expands only at high-uncertainty forks.
Discussion (0). Sign in to comment.