Pith. sign in

REVIEW 5 cited by

Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2203.16487 v6 pith:RBBEHN7D submitted 2022-03-30 cs.CL cs.LG

Heming Xia , Tao Ge , Peiyi Wang , Si-Qing Chen , Furu Wei , Zhifang Sui This is my paper · ORCID
classification cs.CLcs.LG
keywords decodingspeculativespecdecspeeduptimesacceleratingexecutionexploiting
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

We propose Speculative Decoding (SpecDec), for the first time ever, to formally study exploiting the idea of speculative execution to accelerate autoregressive (AR) decoding. Speculative Decoding has two innovations: Spec-Drafter -- an independent model specially optimized for efficient and accurate drafting -- and Spec-Verification -- a reliable method for verifying the drafted tokens efficiently in the decoding paradigm. Experimental results on various seq2seq tasks including machine translation and abstractive summarization show our approach can achieve around $5\times$ speedup for the popular Transformer architectures with comparable generation quality to beam search decoding, refreshing the impression that the draft-then-verify paradigm introduces only $1.4\times$$\sim$$2\times$ speedup. In addition to the remarkable speedup, we also demonstrate 3 additional advantages of SpecDec, revealing its practical value for accelerating generative models in real-world applications. Our models and codes are available at https://github.com/hemingkx/SpecDec.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. EdgeXpert: An Edge Device for Memory-Efficient LLM Inference with Mixture-of-Experts and Speculative Decoding

    cs.AR 2026-08 conditional novelty 6.0 of 10

    A co-designed edge accelerator that avoids loading redundant expert weights when speculative decoding meets mixture-of-experts, cutting latency by up to 56.3% and energy by up to 44.1% in simulation.

  2. IAM: Efficient Inference through Attention Mapping between Different-scale LLMs

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Attention matrices of same-series small and large LLMs are similar enough that replacing up to 50% of a large model's attention layers with the small model's matrices preserves most performance while reducing KV cache...

  3. POSS: Position Specialist Generates Better Draft for Speculative Decoding

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Using position-specialized draft layers instead of one single draft model improves later-token acceptance in speculative decoding, yielding modest speedups on Llama-3-8B and Llama-2-13B.

  4. Hardware-Efficient Attention for Fast Decoding

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Grouped-Tied Attention and Grouped Latent Attention reduce KV-cache memory and speed up LLM decoding by up to 2x while matching the quality of GQA and MLA at up to 1.47B parameters.

  5. XSpecMesh: Quality-Preserving Auto-Regressive Mesh Generation Acceleration via Multi-Head Speculative Decoding

    cs.GR 2025-07 conditional novelty 5.0 of 10

    XSpecMesh speeds up auto-regressive mesh generation by about 1.7x using multi-head speculative decoding with cross-attention heads and a probability threshold verification, while keeping output quality close to the ba...

Pith tools