Pith. sign in

REVIEW 4 cited by

REST: Retrieval-Based Speculative Decoding

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2311.08252 v2 pith:PBA2UZ46 submitted 2023-11-14 cs.CL cs.AIcs.IRcs.LG

Zhenyu He , Zexuan Zhong , Tianle Cai , Jason D. Lee , Di He This is my paper · ORCID
classification cs.CLcs.AIcs.IRcs.LG
keywords restlanguagedecodinggenerationspeculativecodedraftmodel
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We introduce Retrieval-Based Speculative Decoding (REST), a novel algorithm designed to speed up language model generation. The key insight driving the development of REST is the observation that the process of text generation often includes certain common phases and patterns. Unlike previous methods that rely on a draft language model for speculative decoding, REST harnesses the power of retrieval to generate draft tokens. This method draws from the reservoir of existing knowledge, retrieving and employing relevant tokens based on the current context. Its plug-and-play nature allows for seamless integration and acceleration of any language models, all without necessitating additional training. When benchmarked on 7B and 13B language models in a single-batch setting, REST achieves a significant speedup of 1.62X to 2.36X on code or text generation. The code of REST is available at https://github.com/FasterDecoding/REST.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Revisiting Lossy Verification in Speculative Decoding: Mechanisms, Trade-offs, and Failure Modes

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Lossy speculative-decoding verification splits into truncation-based and collaborative methods; truncation-based methods underperform their matched baselines, and capping draft overshoot preserves quality.

  2. Mamba Drafters for Speculative Decoding

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Mamba-based drafters can match self-speculation throughput with lower memory and cross-model flexibility.

  3. Speculate, then Collaborate: Fusing Knowledge of Language Models during Decoding

    cs.CL 2025-02 conditional novelty 5.0 of 10

    CoSD fuses two LLMs at decoding time: it keeps the draft model's tokens unless a confidence rule or a decision tree over token probabilities says the assistant model's token is better.

  4. S$^4$C: Speculative Sampling with Syntactic and Semantic Coherence for Efficient Inference of Large Language Models

    cs.CL 2025-06 conditional novelty 4.0 of 10

    S4C accelerates LLM generation by combining multi-head autoregressive draft heads with a continuous verification tree, measuring 2.26x to 2.60x speedups on Spec-bench.

Pith tools