Pith. sign in

REVIEW 7 cited by

Online Speculative Decoding

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.07177 v4 pith:PFL7JURC submitted 2023-10-11 cs.AI cs.CLcs.LG

Xiaoxuan Liu , Lanxiang Hu , Peter Bailis , Alvin Cheung , Zhijie Deng , Ion Stoica , Hao Zhang This is my paper · ORCID
classification cs.AIcs.CLcs.LG
keywords modeldraftdecodingqueryspeculativedistributiononlinetarget
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Speculative decoding is a pivotal technique to accelerate the inference of large language models (LLMs) by employing a smaller draft model to predict the target model's outputs. However, its efficacy can be limited due to the low predictive accuracy of the draft model, particularly when faced with diverse text inputs and a significant capability gap between the draft and target models. We introduce online speculative decoding to address this challenge. The main idea is to continuously update the (multiple) draft model(s) on observed user query data. Adapting to query distribution mitigates the shifts between the training distribution of the draft model and the query distribution, enabling the draft model to more accurately predict the target model's outputs. We develop a prototype of online speculative decoding based on knowledge distillation and evaluate it using both synthetic and real query data. The results show a substantial increase in the token acceptance rate by 0.1 to 0.65, bringing 1.42x to 2.17x latency reduction. Our code is available at https://github.com/LiuXiaoxuanPKU/OSD.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A training-free self-speculative decoding system with a recallable sparse KV cache and entropy-guided adaptive speculation achieves up to 2.79× speedup while preserving the target distribution.

  2. Adversarial Prompts for Acceptance Collapse in Speculative Decoding

    cs.CR 2026-07 conditional novelty 6.0 of 10

    ADSD shows that a short adversarial suffix appended to a prompt can collapse the token-acceptance rate in speculative decoding, increasing latency by 62.3% on GSM8K while preserving answer accuracy.

  3. AdaDecode: Accelerating LLM Decoding with Adaptive Layer Parallelism

    cs.CL 2025-06 conditional novelty 6.0 of 10

    AdaDecode speeds up LLM generation by predicting tokens at early layers when confidence is high, running the skipped layers in parallel, and verifying the output exactly matches standard decoding.

  4. AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding

    cs.CL 2026-07 conditional novelty 5.0 of 10

    AngelSpec + DFly pair a chat MTP drafter with a code/math block-diffusion drafter and load-aware verification pruning, reaching up to 2.4x AR throughput on Hy3-A21B.

  5. Thinking While Speaking: Inference-Time Knowledge Transfer for Responsive and Intelligent Conversational Voice Agents

    cs.CL 2025-11 reject novelty 5.0 of 10

    A 360M talker model trained on synthetic data answers immediately while integrating streamed knowledge chunks from a large backend model, trading accuracy (46-52% vs 69-80%) for low latency.

  6. Advancing Decoding Strategies: Enhancements in Locally Typical Sampling for LLMs

    cs.CL 2025-06 reject novelty 5.0 of 10

    ASTS extends locally typical sampling with semantic scoring and dynamic thresholds, reporting improved perplexity, MAUVE, and diversity on story and summarization tasks.

  7. SpecASR: Accelerating LLM-based Automatic Speech Recognition via Speculative Decoding

    eess.AS 2025-07 reject novelty 4.0 of 10

    SpecASR accelerates LLM-based ASR by 3.04x-3.79x over autoregressive decoding using adaptive draft lengths, draft token recycling, and sparse token trees, but the speedups are simulated from Whisper proxy models rathe...

Pith tools