REVIEW 11 cited by
SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inference
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Autoregressive large language models (LLMs) have made remarkable progress in various natural language generation tasks. However, they incur high computation cost and latency resulting from the autoregressive token-by-token generation. To address this issue, several approaches have been proposed to reduce computational cost using early-exit strategies. These strategies enable faster text generation using reduced computation without applying the full computation graph to each token. While existing token-level early exit methods show promising results for online inference, they cannot be readily applied for batch inferencing and Key-Value caching. This is because they have to wait until the last token in a batch exits before they can stop computing. This severely limits the practical application of such techniques. In this paper, we propose a simple and effective token-level early exit method, SkipDecode, designed to work seamlessly with batch inferencing and KV caching. It overcomes prior constraints by setting up a singular exit point for every token in a batch at each sequence position. It also guarantees a monotonic decrease in exit points, thereby eliminating the need to recompute KV Caches for preceding tokens. Rather than terminating computation prematurely as in prior works, our approach bypasses lower to middle layers, devoting most of the computational resources to upper layers, allowing later tokens to benefit from the compute expenditure by earlier tokens. Our experimental results show that SkipDecode can obtain 2x to 5x inference speedups with negligible regression across a variety of tasks. This is achieved using OPT models of 1.3 billion and 6.7 billion parameters, all the while being directly compatible with batching and KV caching optimization techniques.
Forward citations
Cited by 11 Pith papers
-
Per-Token Fixed-Point Convergence in Depth-Recurrent Transformers
In a depth-recurrent transformer, each token converges to a fixed point at its own rate; a parameter-free early-exit rule reads this and matches depth-8 quality at 4.94 average loops, beating a learned router.
-
CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsification
CogVLA pairs instruction-conditioned visual-token aggregation (EFA-Routing) with transformer-layer pruning (LFP-Routing) and bidirectional action decoding (CAtten), reporting LIBERO 97.4%, real-world 70.0%, 2.5x less ...
-
OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference
OrthoRank selects tokens for computation by their orthogonality to the sink token in normalized hidden states, yielding better perplexity and accuracy than layer pruning at matched sparsity without additional training.
-
DIVE into MoE: Diversity-Enhanced Reconstruction of Large Language Models from Dense into Mixture-of-Experts
DIVE builds diverse MoE experts by pruning the dense FFN with different domain-specific calibration sets, and reports better perplexity and accuracy than LLM-Pruner, FLAP, and LLaMA-MoE at equal active parameters on T...
-
AdaDecode: Accelerating LLM Decoding with Adaptive Layer Parallelism
AdaDecode speeds up LLM generation by predicting tokens at early layers when confidence is high, running the skipped layers in parallel, and verifying the output exactly matches standard decoding.
-
CLaSp: In-Context Layer Skip for Self-Speculative Decoding
A training-free, context-adaptive layer-skipping method for self-speculative decoding that reports roughly 1.1x to 1.8x speedups on LLaMA models while preserving output distribution.
-
System-1.5 Reasoning: Traversal in Language and Latent Spaces with Dynamic Shortcuts
System-1.5 Reasoning lets LLMs reason in latent space with early exits and step-skipping, matching chain-of-thought accuracy at over 20x speedup on GSM8K and StrategyQA.
-
QTALE: Quantization-Robust Token-Adaptive Layer Execution for LLMs
QTALE's entropy-regularized routing plus an inference-time threshold adjustment recovers most of the accuracy lost when quantizing token-adaptive layer-skipping LLMs.
-
DASH: Input-Aware Dynamic Layer Skipping for Efficient LLM Inference with Markov Decision Policies
DASH trains a token-level policy to skip, quantize, or fully run each Transformer layer, reporting 1.33x-2.0x compute reduction with smaller accuracy loss than prior skipping methods.
-
Constraint-Driven Model Optimization: An Industry Framework for Selecting Compression and Acceleration Techniques in Modern Machine Learning Systems
A decision framework mapping five production constraints to established model-optimization techniques, synthesized from 25+ cited papers, with four worked industry pipelines.
-
SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decoupling
SkipGPT learns per-token, per-module routers that dynamically skip attention and MLP blocks, then uses LoRA to recover most of the original model's quality.
Discussion (0). Continue with ORCID to comment.