Pith. sign in

REVIEW 20 cited by

Break the Sequential Dependency of LLM Inference Using Lookahead Decoding

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.02057 v1 pith:B4LVT5T6 submitted 2024-02-03 cs.LG cs.CL

Yichao Fu , Peter Bailis , Ion Stoica , Hao Zhang This is my paper · ORCID
classification cs.LGcs.CL
keywords decodinglookaheadacceleratorsautoregressivecodemodelsmodernmultiple
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Autoregressive decoding of large language models (LLMs) is memory bandwidth bounded, resulting in high latency and significant wastes of the parallel processing power of modern accelerators. Existing methods for accelerating LLM decoding often require a draft model (e.g., speculative decoding), which is nontrivial to obtain and unable to generalize. In this paper, we introduce Lookahead decoding, an exact, parallel decoding algorithm that accelerates LLM decoding without needing auxiliary models or data stores. It allows trading per-step log(FLOPs) to reduce the number of total decoding steps, is more parallelizable on single or multiple modern accelerators, and is compatible with concurrent memory-efficient attention (e.g., FlashAttention). Our implementation of Lookahead decoding can speed up autoregressive decoding by up to 1.8x on MT-bench and 4x with strong scaling on multiple GPUs in code completion tasks. Our code is avialable at https://github.com/hao-ai-lab/LookaheadDecoding

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 20 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Beyond KV Reconstruction: Functional Reconstruction for MLA Draft Models in Speculative Decoding

    cs.LG 2026-07 conditional novelty 7.0 of 10

    A post-conversion training objective that reconstructs the original attention function of MLA-converted draft models improves speculative-decoding acceptance in most of 64 matched task cells.

  2. From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving

    cs.DC 2026-06 accept novelty 6.5 of 10

    KV-cache serving systems concentrate into five archetypes under a four-axis taxonomy, with ownership explaining residual distributed design variance and seven measurement gaps blocking next steps.

  3. PaDoc: Layout-Grounded Parallel Decoding for Document Parsing

    cs.AI 2026-08 conditional novelty 6.0 of 10

    PaDoc generates document layout regions and their contents in parallel branches from a shared page image, cutting decoding depth and speeding up end-to-end parsing on a single GPU.

  4. Streaming Communication in Multi-Agent Reasoning

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    StreamMA introduces streaming communication in multi-agent reasoning to reduce latency via pipelining and improve effectiveness by leveraging reliable early steps, with closed-form analysis and a step-level scaling law.

  5. HiSpec: Hierarchical Speculative Decoding for LLMs

    cs.CL 2025-10 conditional novelty 6.0 of 10

    HiSpec uses early-exit layers to verify draft tokens midway through the model, reporting 1.28×-2.01× faster decoding over baseline speculative decoding, but the accuracy claim is not empirically tested.

  6. Structuring The Future: Diffusion LLM Speculative Decoding via Calibrated Draft Graphs

    cs.LG 2025-09 conditional novelty 6.0 of 10

    Spiffy speeds up diffusion LLM inference up to about 3x (and up to 7.9x with parallel decoding) by verifying multiple candidate unmasked states in one batched model call, while preserving greedy output.

  7. Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling

    cs.CL 2025-08 conditional novelty 6.0 of 10

    N-gram based speculative decoding methods, especially SAM and hybrid SAM[EAGLE-3], achieve strong speedups in LLM test-time scaling by exploiting repetitive reasoning patterns.

  8. Mamba Drafters for Speculative Decoding

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Mamba-based drafters can match self-speculation throughput with lower memory and cross-model flexibility.

  9. Speculative Decoding Reimagined for Multimodal Large Language Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    MSD decouples text and visual tokens in the draft model and trains it first on text, then on a gradually increasing mix of visual data, yielding roughly 2x lossless speedups on LLaVA benchmarks.

  10. ParVL: Parallel Scaling and Expandable Compute Allocation for Multimodal LLMs

    cs.CV 2026-08 conditional novelty 5.0 of 10

    ParVL scales MLLM computation by running multiple prefix-conditioned ViT and LLM branches over a shared backbone, improving average benchmark scores by 0.3 to 0.9 points and showing task-dependent vision-language allocation.

  11. KAP: Bridging the Knowledge Selection-Runtime Consumption Gap in LLM Systems

    cs.LG 2026-07 conditional novelty 5.0 of 10

    KAP compiles structured knowledge priors into runtime access plans that cut proposal-time KV access to 5.5% at 128K while matching full-context QA quality and yielding up to 1.19× decode throughput.

  12. WAR: Workload-Aware Rollouts for Synchronous Agentic Reinforcement Learning

    cs.LG 2026-07 conditional novelty 5.0 of 10

    A workload-aware rollout system combining suffix-based speculative decoding (low load) and cache-aware scheduling (high load) speeds synchronous agentic RL rollout by 1.4-1.6x.

  13. Parallel Decoder Transformer: Planner-Conditioned Latent Coordination for Model-Intrinsic Parallel Generation

    cs.AI 2025-12 reject novelty 5.0 of 10

    A Parallel Decoder Transformer architecture for multi-lane parallel text generation is proposed, but the paper's own evidence is incomplete and internally contradictory.

  14. XSpecMesh: Quality-Preserving Auto-Regressive Mesh Generation Acceleration via Multi-Head Speculative Decoding

    cs.GR 2025-07 conditional novelty 5.0 of 10

    XSpecMesh speeds up auto-regressive mesh generation by about 1.7x using multi-head speculative decoding with cross-attention heads and a probability threshold verification, while keeping output quality close to the ba...

  15. CoDec: Prefix-Shared Decoding Kernel for LLMs

    cs.LG 2025-05 conditional novelty 5.0 of 10

    CoDec combines KV-cache reads across requests that share a prefix, yielding average 1.9x decode-attention speedup and 120.9x less global memory traffic versus FlashDecoding.

  16. S$^4$C: Speculative Sampling with Syntactic and Semantic Coherence for Efficient Inference of Large Language Models

    cs.CL 2025-06 conditional novelty 4.0 of 10

    S4C accelerates LLM generation by combining multi-head autoregressive draft heads with a continuous verification tree, measuring 2.26x to 2.60x speedups on Spec-bench.

  17. Consultant Decoding: Yet Another Synergistic Mechanism

    cs.CL 2025-06 conditional novelty 4.0 of 10

    Consultant Decoding speeds up LLM generation by accepting draft tokens whose negative log-likelihood under the target model falls below a fixed threshold, reaching 2-3x speedups with comparable quality.

  18. AutoChemSchematic AI: Agentic Physics-Aware Automation for Chemical Manufacturing Scale-Up

    cs.LG 2025-05 reject novelty 4.0 of 10

    The framework trains small models on synthetic AI-generated data to produce PFD/PID text, then validates two examples by manual DWSIM setup, leaving the industrial-viability claim unproven.

  19. CopySpec: Accelerating LLMs with Speculative Copy-and-Paste Without Compromising Quality

    cs.CL 2025-02 conditional novelty 4.0 of 10

    CopySpec accelerates LLM inference by copying repeated token sequences from context and verifying them with the target model, achieving up to 3.08x speedups without altering outputs.

  20. The Hitchhiker's Guide to Agentic AI: From Foundations to Systems

    cs.AI 2026-06 unverdicted novelty 2.0 of 10

    A survey-style reference book mapping the full agentic-AI stack from transformer internals to production deployment, with no new research result.

Pith tools