Pith. sign in

REVIEW 5 cited by

OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.17276 v4 pith:BIH56DPC submitted 2024-06-25 cs.CL

Jikai Wang , Yi Su , Juntao Li , Qingrong Xia , Zi Ye , Xinyu Duan , Zhefeng Wang , Min Zhang This is my paper · ORCID
classification cs.CL
keywords draftdecodingopt-treestepacceptanceadaptiveautoregressivebecome
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Autoregressive language models demonstrate excellent performance in various scenarios. However, the inference efficiency is limited by its one-step-one-word generation mode, which has become a pressing problem recently as the models become increasingly larger. Speculative decoding employs a "draft and then verify" mechanism to allow multiple tokens to be generated in one step, realizing lossless acceleration. Existing methods mainly adopt fixed heuristic draft structures, which fail to adapt to different situations to maximize the acceptance length during verification. To alleviate this dilemma, we proposed OPT-Tree, an algorithm to construct adaptive and scalable draft trees. It searches the optimal tree structure that maximizes the mathematical expectation of the acceptance length in each decoding step. Experimental results reveal that OPT-Tree outperforms the existing draft structures and achieves a speed-up ratio of up to 3.2 compared with autoregressive decoding. If the draft model is powerful enough and the node budget is sufficient, it can generate more than ten tokens in a single step. Our code is available at https://github.com/Jikai0Wang/OPT-Tree.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding

    cs.CL 2026-07 conditional novelty 5.0 of 10

    AngelSpec + DFly pair a chat MTP drafter with a code/math block-diffusion drafter and load-aware verification pruning, reaching up to 2.4x AR throughput on Hy3-A21B.

  2. Advancing Decoding Strategies: Enhancements in Locally Typical Sampling for LLMs

    cs.CL 2025-06 reject novelty 5.0 of 10

    ASTS extends locally typical sampling with semantic scoring and dynamic thresholds, reporting improved perplexity, MAUVE, and diversity on story and summarization tasks.

  3. AdaEAGLE: Optimizing Speculative Decoding via Explicit Modeling of Adaptive Draft Structures

    cs.AI 2024-12 conditional novelty 5.0 of 10

    AdaEAGLE learns to predict the number of accepted draft tokens from the last hidden state and uses that prediction as the adaptive draft length in EAGLE-style speculative decoding.

  4. Consultant Decoding: Yet Another Synergistic Mechanism

    cs.CL 2025-06 conditional novelty 4.0 of 10

    Consultant Decoding speeds up LLM generation by accepting draft tokens whose negative log-likelihood under the target model falls below a fixed threshold, reaching 2-3x speedups with comparable quality.

  5. Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding

    cs.CL 2024-11 conditional novelty 2.0 of 10

    A survey that categorizes speculative decoding methods into draft-centric and model-centric families and discusses deployment challenges.

Pith tools