Pith. sign in

REVIEW 1 cited by

A Practical Investigation of Training-free Relaxed Speculative Decoding

T0 review · reviewed 2026-07-10 · glm-5.2

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Relaxed Speculative Decoding Fails With Lightweight Drafters

desk verdict Solid empirical study with one load-bearing generalization gap read the letter →

arxiv 2607.08690 v1 pith:ZCBPPTWA submitted 2026-07-09 cs.LG cs.AI

Guoxuan Xia , Luka Ribar , Paul Balanca This is my paper · ORCID
classification cs.LGcs.AI
keywords decodingspeculativerelaxedapproachescapabilitylosslessmodelsampling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Speculative decoding speeds up large language model inference by having a smaller, faster model draft tokens that the large model then verifies in parallel. Standard speculative decoding is lossless: the rejection and resampling steps exactly preserve the large model's output distribution. A recent line of research proposes relaxing that lossless guarantee, accepting slightly more draft tokens in exchange for either more speed, a controllable trade-off between speed and quality, or even improved capability. This paper unifies six training-free relaxed speculative decoding methods into a single framework, benchmarks them on modern reasoning tasks (math olympiad, graduate-level science, competitive programming) with contemporary drafter-verifier pairs, and finds a fundamental mismatch between what relaxed methods need and what modern drafters provide. The core finding is that most relaxed methods implicitly assume the drafter is a competent standalone language model whose distribution can be trusted when it diverges from the verifier. Lightweight dedicated multi-token-prediction (MTP) drafters, which the field is moving toward, are specialized for fast drafting under strict rejection and are not good language models in their own right. When relaxation tries to rely on their distribution, the result is degraded capability, longer rambling responses, and sometimes outright slowdowns even though raw token throughput increases. Only methods that tightly bound how far the relaxed distribution can drift from the verifier (CACTUS and mentored-dec) produce useful trade-offs with such drafters. A strong standalone drafter, by contrast, can yield approximately lossless speed-ups from relaxation, but at the cost of much higher drafter overhead that limits real-world gains. The paper also shows that optimizing draft length under strict speculative decoding has comparable impact on speed-up to relaxation itself, without requiring any capability re-evaluation, and that relaxation hyperparameters do not transfer across tasks.

What carries the argument

The paper introduces a unified framework (Algorithm 2) that replaces three distributions in standard speculative decoding (rejection target, residual sampling distribution, and bonus token distribution) with relaxed targets parameterized by a relaxation parameter alpha. Six methods (CACTUS, mentored-dec, r-fuzzy, spec-casc-opt, ensemble, spec-cont-dec) are expressed as different choices for these relaxed distributions. Speed-up is modeled with a proxy that accounts for both token acceptance rates and changes in average response length, evaluated across a grid of draft lengths and relative drafter costs.

What would settle it

If a future MTP drafter architecture were designed to be both lightweight and a competent standalone language model, or if serving frameworks optimized the drafter path to reduce c_rel dramatically for standalone drafters, the conflict between relaxation and lightweight drafters would diminish and the trade-offs could shift in favor of relaxed methods.

Watch

Extended reading notes

Core claim

The central discovery is a conflict between the design assumptions of relaxed speculative decoding and the direction the inference community is heading. Relaxed methods that trade losslessness for speed implicitly require the drafter to be a reliable language model, because they accept draft tokens based on the drafter's own distribution or confidence. Dedicated MTP drafters are built to be fast and to maintain short-range acceptance under strict rejection, not to generate coherent long-form text. When these lightweight drafters are used with relaxation, the model produces longer, lower-quality responses, sometimes entering repetition loops that inflate generation length and erase any speed.

Load-bearing premise

The speed-up proxy model assumes that the relative cost of drafting versus verification (c_rel) and the average accepted draft length are sufficient to estimate real-world speed-up, but real wall-clock performance depends on hardware-software specifics that could systematically favor or penalize certain methods differently than the proxy predicts.

Editorial extensions

If this is right

  • Practitioners adopting bundled MTP drafters should prioritize optimizing draft length under strict speculative decoding before considering any relaxation, as the former yields comparable speed-up gains without capability re-evaluation.
  • Methods that tightly control deviation from the verifier distribution are the only viable relaxation path for lightweight drafters; methods that trust the drafter's distribution require a strong standalone language model as drafter.
  • Relaxation hyperparameters need per-task calibration, making relaxed speculative decoding a deployment-specific optimization rather than a drop-in replacement for strict speculative decoding.
  • Speculative contrastive decoding (spec-cont-dec) offers a distinct trade-off direction, sacrificing speed for potential capability gains when the drafter is weak, which could be useful in capability-constrained rather than speed-constrained deployments.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

0 major / 0 minor

Summary. This paper presents a practical investigation of training-free relaxed speculative decoding methods, which relax the strict distribution-preservation guarantee of standard speculative decoding in exchange for potential speed-ups or capability-speed trade-offs. The authors unify six existing approaches (CACTUS, mentored-dec, r-fuzzy, spec-casc-opt, ens, spec-cont-dec) within a shared algorithmic framework (Algorithm 2), benchmark them across three drafter-verifier pairs (dedicated MTP, weak LM, strong LM) and three reasoning benchmarks (AIME24, GPQA, LCB), and distill eight key findings for practitioners. The central empirical finding is that many relaxed approaches rely on the drafter being a good standalone language model, making them unsuited for lightweight dedicated multi-token-prediction (MTP) drafters—the direction the community is moving toward. The paper also provides a primer on strict speculative decoding, a speed-up model (Eqs. 2 and 4), and a proof that the reducible variant of fuzzy speculative decoding (r-fuzzy) weakly improves draft acceptance over the standard fuzzy variant while preserving the same output distribution.

Significance. The paper addresses a timely and practically important question. As speculative decoding sees increasing adoption in production LLMs (DeepSeek, Qwen, etc.) and inference frameworks (vLLM, SGLang), and as the community shifts toward bundled MTP drafters, understanding whether relaxed speculative decoding methods actually work in these settings is of clear value to practitioners. The unification of disparate methods under a single framework (Algorithm 2, Table 2) is a useful contribution that makes the landscape more navigable. The finding that MTP drafters are largely unsuited for relaxed methods (except CACTUS/mentored-dec) is a non-trivial and potentially impactful result that challenges the applicability of a body of recent literature. The authors provide reproducible code and a response-length-aware speed-up metric (Eq. 4) that corrects a common omission in prior work. The r-fuzzy proof (Appendix C) is a clean, self-contained result. The experimental design—spanning MTP/weak/strong drafters, multiple benchmarks, draft lengths, and relaxation parameters with standard error bars—is thorough relative to the existing literature, which the authors correctly note suffers from narrow,

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical investigation with externally-sourced formulas and no self-citation chain

full rationale

This paper is an empirical benchmarking study, not a derivation paper. The speed-up proxy model (Eq. 2) is explicitly attributed to Leviathan et al. (2023) and used as a tool, not presented as a novel derivation. The acceptance probability formula (Eq. 3) is cited from Yin et al. (2024). The r-fuzzy proof (Appendix C) relies on the standard speculative-correction fact from Yin et al. (2024), an independent external result. The taxonomy (Table 2) organizes methods from Hao & Mou (2026), Tran-Thien (2023), Holsman et al. (2025), Narasimhan et al. (2025), Wang et al. (2026), and Yuan et al. (2024) — none of which are self-citations by the current authors (Xia, Ribar, Balanca). The one self-citation (Xia & Bouganis 2023) appears only as background context for early-exit cascades in Section 3 and is not load-bearing for any claim. The key findings (e.g., Key Finding 4 about MTP drafters) are directly measured empirical observations (capability scores on AIME24/GPQA/LCB), not predictions derived from a model that could reduce to its inputs. No step in the paper's argument chain reduces by construction to a fitted constant or a self-citation.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities, particles, or postulated objects. It reformulates existing methods under a shared framework (Algorithm 2) and benchmarks them. The free parameters are experimental hyperparameters (c_rel, N_draft, alpha) swept over grids, not constants fitted to make a derivation work. The axioms are standard domain assumptions from the speculative decoding literature.

free parameters (3)
  • c_rel = {0.05, 0.2, 0.5}
    Relative drafter cost is swept over three values as a proxy for real deployment scenarios rather than measured end-to-end for each configuration.
  • N_draft = {3, 5, 10, 20}
    Draft length is swept over four values; not a fitted parameter per se but an inference-time hyperparameter grid.
  • alpha (per method) = varies by method, e.g. CACTUS {0.1, 0.25, 1.0, 10.0}
    Relaxation parameter grids are method-specific and chosen by the authors; the paper shows these do not generalize across tasks.
assumptions (4)
  • domain assumption C_verify ≈ C_AR (verification cost approximates single AR step cost)
    Stated in §2.1 as a common assumption enabling the simplified speed-up model (Eq. 2). Holds when AR decoding is memory-bound but may break at high batch sizes.
  • standard math Standard speculative correction fact: accepting v~q with prob min{1, rho(v)/q(v)} and resampling from norm((rho-q)+) emits tokens with marginal distribution rho
    Used in the r-fuzzy proof (Appendix C), attributed to Yin et al. (2024). This is a well-established result in the spec-dec literature.
  • domain assumption Top-p/top-k softmax truncation is standard practice for LLM generation
    Used to define the plausibility set M_top-p/k for spec-cont-dec (Eq. 14) and as support truncation across all experiments. Cited to Noarov et al. (2025).
  • domain assumption Jensen-Shannon Divergence is the appropriate divergence criterion for r-fuzzy
    Recommended in Holsman et al. (2025) to avoid divide-by-zero issues with KL-divergence on non-overlapping support from top-p/k masking.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Practical Investigation of Training-free Relaxed Speculative Decoding." pith.science (2026). https://pith.science/paper/ZCBPPTWA

@misc{pith2026260708690,
  author       = {Pith},
  title        = {Pith review of: A Practical Investigation of Training-free Relaxed Speculative Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZCBPPTWA}},
  note         = {Machine review of arXiv:2607.08690}
}
read the original abstract

Speculative decoding accelerates sampling from an autoregressive LLM by using a faster auxiliary model to draft tokens which are then verified in parallel by the LLM. Standard speculative decoding is lossless: its rejection and resampling steps exactly preserve the LLM's sampling distribution. Recent work argues that relaxing this strict guarantee can yield further speed-ups, controlled capability-speed trade-offs, or even capability gains. We practically investigate training-free relaxed speculative decoding techniques, unify existing approaches within a shared framework, benchmark them on contemporary settings, and distil takeaways and empirical findings for practitioners. Important takeaways include: relaxation can require considerable capability evaluation unlike lossless speculative decoding, and many relaxed approaches rely on a drafter that is a good language model, making them unsuited for lightweight dedicated multi-token-prediction drafters.

Figures

Figures reproduced from arXiv: 2607.08690 by the authors.

Figure 1
Figure 1. Left: In strict speculative decoding tokens are rapidly drafted and then verified in parallel. Draft tokens are stochastically rejected; the first rejected token is resampled such that overall sampling is from verifier target distribution p. Right: Relaxed speculative decoding can replace p with relaxed target distribution π at different points. 2. We build a taxonomy over the literature of relaxed speculative decod… view at source ↗
Figure 2
Figure 2. Left: The speed-up model in Eq. (2). Drafter relative cost crel has a strong impact on speed-up, the best choice of Ndraft, and the speed-up gain from increased average accepted length ¯laccept. Right: strict spec-dec gives lossless speed-up, whilst relaxed spec-dec trades capability for even more speed. If deployment is capability constrained, the former is lossless so it doesn’t require the burden of additional ca… view at source ↗
Figure 3
Figure 3. Comparison between CACTUS and mentored-dec. Left: a visualisation of how the relaxed probability of the draft token x is boosted, right: empirical trade-off comparisons. Their algorithmic and narrative similarities (Sec. 4) result in similar empirical behaviour. We omit mentored-dec in favour of CACTUS in the main paper body for brevity. . drafter module, a weak language model drafter and a strong language model dra… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Capability-speed trade-offs on AIME24 for relaxed spec-dec methods across drafter+verifier pairs, relative drafter costs crel and draft lengths Ndraft. Standard error lines indicate measurement noise from stochastic sampling. (A) Ndraft has a comparable effect on speed…
Figure 5
Figure 5. Figure 5: Left: (spec-casc-opt) drafters that are weak language models improve token throughput much more than length-adjusted speed-up. Right: relaxation into an MTP drafter increases generation length with rambling responses. models, and so can’t be relied on for relaxation, e…
Figure 6
Figure 6. Figure 6: Relaxation hyperparameters do not reliably generalise between evaluation tasks. Comparing [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Capability-speed trade-offs on AIME24 for relaxed spec-dec methods across drafter+verifier pairs, relative drafter costs crel and draft lengths Ndraft. Each block shows one drafter+verifier pair, with method columns and crel rows. The x-axis is length-adjusted speed-up…
Figure 8
Figure 8. Figure 8: Capability-speed trade-offs on GPQA for relaxed spec-dec methods across drafter+verifier pairs, relative drafter costs crel and draft lengths Ndraft. Each block shows one drafter+verifier pair, with method columns and crel rows. The x-axis is length-adjusted speed-up (…
Figure 9
Figure 9. Figure 9: Capability-speed trade-offs on LCB for relaxed spec-dec methods across drafter+verifier pairs, relative drafter costs crel and draft lengths Ndraft. Each block shows one drafter+verifier pair, with method columns and crel rows. The x-axis is length-adjusted speed-up (l…
Figure 10
Figure 10. Figure 10: Capability-speed trade-offs on GPQA for relaxed spec-dec methods for Llama3, relative drafter costs crel and draft lengths Ndraft. The x-axis is length-adjusted speed-up (log scale) and the y-axis is task capability. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_10.png]
Figure 11
Figure 11. Figure 11: Modelled memory cost for native-MTP Qwen3.5 models as batch size increases. The decomposition [PITH_FULL_IMAGE:figures/full_fig_p029_11.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Oilbird: Training-Free Speculative Decoding with Keys the Verifier Already Computes

    cs.AI 2026-08 conditional novelty 8.0 of 10

    A training-free speculative decoder using the verifier's hidden states as a semantic key retrieves correct continuations unreachable by exact suffix matching, lifting accepted length 24-29%.

Reference graph

Works this paper leans on

39 extracted references · 39 canonical work pages · cited by 1 Pith paper

  1. [1]

    1 Gregor Bachmann, Sotiris Anagnostidis, Albert Pumarola, Markos Georgopoulos, Artsiom Sanakoyeu, Yuming Du, Edgar Schönfeld, Ali Thabet, and Jonas K Kohler

    Accessed: 2026-05-29. 1 Gregor Bachmann, Sotiris Anagnostidis, Albert Pumarola, Markos Georgopoulos, Artsiom Sanakoyeu, Yuming Du, Edgar Schönfeld, Ali Thabet, and Jonas K Kohler. Judge decoding: Faster speculative sampling requires going beyond model alignment. InThe Thirteenth International Conference on Learning Representations,

    work page 2026

  2. [2]

    URL https://openreview.net/forum?id=mtSSFiqW6y. 1, 5 Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris H...

    work page 1901

  3. [3]

    1 Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D

    URL https://proceedings.neurips.cc/paper_files/paper/ 2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf. 1 Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nu...

    work page 2020

  4. [4]

    5 Haw-Shiuan Chang, Nanyun Peng, Mohit Bansal, Anil Ramakrishna, and Tagyoung Chung

    URLhttps://proceedings.mlr.press/v235/cai24b.html. 5 Haw-Shiuan Chang, Nanyun Peng, Mohit Bansal, Anil Ramakrishna, and Tagyoung Chung. Explaining and improving contrastive decoding by extrapolating the probabilities of a huge and hypothetical LM. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.),Proceedings of the 2024 Conference on Empirical M...

    work page 2024

  5. [5]

    doi: 10.18653/v1/2024.emnlp-main.484

    Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.484. URLhttps://aclanthology.org/2024.emnlp-main.484/. 8 Jian Chen, Yesheng Liang, and Zhijian Liu. Dflash: Block diffusion for flash speculative decoding,

    work page doi:10.18653/v1/2024.emnlp-main.484 2024

  6. [6]

    DFlash: Block Diffusion for Flash Speculative Decoding

    URL https://arxiv.org/abs/2602.06036. 3, 5 Zhuoming Chen, Avner May, Ruslan Svirschevski, Yuhsun Huang, Max Ryabinin, Zhihao Jia, and Beidi Chen. Sequoia: scalable and robust speculative decoding. InProceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY , USA,

    work page Pith review arXiv

  7. [7]

    DeepSeek-V3 Technical Report

    URLhttps://arxiv.org/abs/2412.19437. 1, 6 Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 889–898, Melbourne, Australia, July

    work page Pith review arXiv

  8. [8]

    doi: 10.18653/v1/P18-1082

    Association for Computational Linguistics. doi: 10.18653/v1/P18-1082. URL https://aclanthology.org/P18-1082/. 8 13 Jiale Fu, Yuchu Jiang, Junkai Chen, Jiaming Fan, Xin Geng, and Xu Yang. Fast large language model collabo- rative decoding via speculation. InForty-second International Conference on Machine Learning,

    work page doi:10.18653/v1/p18-1082

Show all 39 references
  1. [10]

    6 Yongchang Hao and Lili Mou

    URL https://arxiv.org/abs/2602.15763. 6 Yongchang Hao and Lili Mou. Cactus: Accelerating auto-regressive decoding with constrained acceptance speculative sampling. InThe Fourteenth International Conference on Learning Representations,

    arXiv

  2. [11]

    4, 6, 7, 9, 23 Zhenyu He, Zexuan Zhong, Tianle Cai, Jason Lee, and Di He

    URL https://openreview.net/forum?id=lpUIkCAy9p. 4, 6, 7, 9, 23 Zhenyu He, Zexuan Zhong, Tianle Cai, Jason Lee, and Di He. REST: Retrieval-based speculative decoding. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.),Proceedings of the 2024 Conference of the North American ...

    2024

  3. [12]

    doi: 10.18653/v1/2024.naacl-long.88

    Association for Computational Linguistics. doi: 10.18653/v1/2024.naacl-long.88. URLhttps://aclanthology.org/2024.naacl-long.88/. 5 Maximilian Holsman, Yukun Huang, and Bhuwan Dhingra. Fuzzy speculative decoding for a tunable accuracy-runtime tradeoff. In Wanxiang Che, Joyce Na...

    2024 doi

  4. [13]

    ISBN 979-8-89176-256-5

    Association for Computational Linguistics. ISBN 979-8-89176-256-5. doi: 10.18653/v1/2025.findings-acl.1346. URLhttps://aclanthology.org/2025.findings-acl.1346/. 1, 4, 7, 20, 23 Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degen...

    2025 doi

  5. [14]

    Accessed: 2026-05-29

    URL https://blog.google/innovation-and-ai/technology/ developers-tools/multi-token-prediction-gemma-4/. Accessed: 2026-05-29. 6 14 Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative de- coding. In Andreas Krause, Emma Brunskill, K...

    2026

  6. [15]

    doi: 10.18653/v1/2023.acl-long.687

    Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.687. URL https://aclanthology.org/2023.acl-long.687/. 8, 12 Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: speculative sampling requires rethinking feature uncertainty. InProceedings of th...

    2023 doi

  7. [16]

    3, 4 LiveCodeBench

    URLhttps://arxiv.org/abs/2601.11580. 3, 4 LiveCodeBench. LiveCodeBench code generation lite, release_v6,

  8. [17]

    Versioned Hugging Face dataset configuration used for LiveCodeBench Lite v6

    URL https://huggingface.co/datasets/ livecodebench/code_generation_lite. Versioned Hugging Face dataset configuration used for LiveCodeBench Lite v6. 22 Llama Team. The llama 3 herd of models.CoRR, abs/2407.21783,

    arXiv

  9. [18]

    URL https://doi.org/10.48550/arXiv.2407.21783

    doi: 10.48550/arXiv.2407.21783. URL https://doi.org/10.48550/arXiv.2407.21783. 9, 23 Xiaoyu Ma and David Patterson. Challenges and research directions for large language model inference hardware. Computer, 59(5):55–64,

    doi arXiv

  10. [19]

    doi: 10.1109/MC.2026.3652916. 2 Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. Specinfer: Accelerating larg...

    2026 doi

  11. [20]

    ISBN 9798400703867

    Association for Computing Machinery. ISBN 9798400703867. doi: 10.1145/3620666.3651335. URLhttps://doi.org/10.1145/3620666.3651335. 5 Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Seungyeon Kim, Neha Gupta, Aditya Krishna Menon, and Sanjiv Kumar. Faster cascad...

    doi

  12. [21]

    4, 6, 7, 8, 11, 20, 23 Shashi Narayan, Shay B

    URL https://openreview.net/forum?id=vo9t20wsmd. 4, 6, 7, 8, 11, 20, 23 Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Ellen Riloff, David Chiang, Julia Hock...

    2018

  13. [22]

    doi: 10.18653/v1/D18-1206

    Association for Computational Linguistics. doi: 10.18653/v1/D18-1206. URLhttps://aclanthology.org/D18-1206/. 8 15 Georgy Noarov, Soham Mallick, Tao Wang, Sunay Joshi, Yan Sun, Yangxinyu Xie, Mengxin Yu, and Edgar Dobriban. Foundations of top-k decoding for language models. InA...

    doi

  14. [23]

    9 NVIDIA

    URL https://proceedings.neurips.cc/paper_files/paper/2025/hash/ a96d4fda3017f1773b261a52a3efc8dd-Abstract-Conference.html. 9 NVIDIA. Nemotron 3 ultra: Open, efficient mixture-of-experts hybrid mamba-transformer model for agentic reasoning, June

    2025

  15. [24]

    8, 12 OpenAI

    URL https://openreview.net/forum?id=SzV37yefM4. 8, 12 OpenAI. Why SWE-bench verified no longer measures frontier coding capabilities. https://openai.com/index/ why-we-no-longer-evaluate-swe-bench-verified/, February 2026a. Accessed: 2026-06-06. 5 OpenAI. Introducing gpt-5.3-co...

    2026

  16. [25]

    URLhttps://openreview.net/forum?id=Ti67584b98. 9, 22 Sha Sajadieh, Loredana Fattorini, Raymond Perrault, Yolanda Gil, Vanessa Parli, Lapo Santarlasci, Juan Pava, Nestor Maslej, Russ Altman, Erik Brynjolfsson, Carla Brodley, Jack Clark, Virginia Dignum, Vipin Kumar, James Landa...

    2026

  17. [26]

    1 Alexander Samarin, Sergei Krutikov, Anton Shevtsov, Sergei Skvortsov, Filipp Fisin, and Alexander Golubev

    URL https://hai.stanford.edu/ai-index/2026-ai-index-report. 1 Alexander Samarin, Sergei Krutikov, Anton Shevtsov, Sergei Skvortsov, Filipp Fisin, and Alexander Golubev. LK losses: Direct acceptance rate optimization for speculative decoding,

    2026

  18. [27]

    URL https://arxiv.org/abs/2602.23881. ICML

    arXiv

  19. [28]

    Accessed: 2026-05-29

    URL https://vllm.ai/blog/2025-12-13-speculators-v030. Accessed: 2026-05-29. 1, 6 Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generator networks. InProceedings of the 55th Annual Meeting of the Association for Computationa...

    2025

  20. [29]

    doi: 10.18653/v1/P17-1099

    Association for Computational Linguistics. doi: 10.18653/v1/P17-1099. URLhttps://aclanthology.org/P17-1099/. 8 Sonali Singh, Karthik Sangaiah, Shenrun Zhang, Ryan Swann, and Ganesh Dasika. Accelerating LLM inference: Up to 3x speedup on MI300X with speculative decoding. ROCm B...

    doi

  21. [30]

    com/artificial-intelligence/spec_decode_mi300x/README.html

    URL https://rocm.blogs.amd. com/artificial-intelligence/spec_decode_mi300x/README.html. Accessed: 2026-05-30. 3 Ziteng Sun, Uri Mendlovic, Yaniv Leviathan, Asaf Aharoni, Jae Hun Ro, Ahmad Beirami, and Ananda Theertha Suresh. Block verification accelerates speculative decoding....

    2026

  22. [31]

    1 Ziyi Wang, Siva Rajesh Kasa, Ankith M S, Santhosh Kumar Kasa, Jiaru Zou, Sumit Negi, Ruqi Zhang, Nan Jiang, and Qifan Song

    URL https://proceedings.neurips.cc/paper_files/paper/2017/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf. 1 Ziyi Wang, Siva Rajesh Kasa, Ankith M S, Santhosh Kumar Kasa, Jiaru Zou, Sumit Negi, Ruqi Zhang, Nan Jiang, and Qifan Song. DIVERSED: Relaxed speculative decoding via ...

    2017

  23. [32]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    7 An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, J...

    arXiv

  24. [33]

    Accessed: 2026-05-29

    URL https://epoch.ai/data-insights/ai-chip-production . Accessed: 2026-05-29. 1 Hongyi Yuan, Keming Lu, Fei Huang, Zheng Yuan, and Chang Zhou. Speculative contrastive decoding. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.),Proceedings of the 62nd Annual Meeting of th...

    2026

  25. [34]

    doi: 10.18653/v1/2024.acl-short.5

    Association for Compu- tational Linguistics. doi: 10.18653/v1/2024.acl-short.5. URL https://aclanthology.org/2024.acl-short.5/. 1, 6, 7, 8, 23 Yifan Zhang and Math-AI Team. American invitational mathematics examination (aime) 2024,

    2024 doi

  26. [35]

    Notation Meaning q,pDrafter and verifier models;pis the LLM being accelerated q(·),p(·)Drafter/verifier next-token distributions qt,pt,qt+i,pt+i Next-token distributions at positionstort+i x<t Autoregressive context before positiont xt,xt+i Tokens at positionstandt+i x,v,VDraf...

    2025

  27. [36]

    The exam is part of the U.S

    evaluates olympiad-style contest mathematics with a three-digit integer answer. The exam is part of the U.S. mathematical olympiad qualification pathway, which gives context for the level of contest problem solving being evaluated. We use the 30-question AIME 2024 set distribu...

    2024

  28. [37]

    The questions are written for domain experts in biology, physics and chemistry, so the task is meant to probe expert-level scientific knowledge rather than general STEM familiarity

    evaluates graduate-level STEM scientific reasoning in a multiple-choice format. The questions are written for domain experts in biology, physics and chemistry, so the task is meant to probe expert-level scientific knowledge rather than general STEM familiarity. We use the 198-...

    2025

  29. [38]

    ### Format: Read the inputs from stdin solve the problem and write the answer to stdout (do not directly test on the sample inputs)

    Print the sum, counting repeated table entries separately. ### Format: Read the inputs from stdin solve the problem and write the answer to stdout (do not directly test on the sample inputs). Enclose your code within delimiters as follows. Ensure that when the python program r...

    2025

  30. [39]

    The standard thinking budget is 32,768 tokens and the maximum generation length is 36,864 tokens

    for thinking mode, temperature=0.6, top- p=0.95 and top-k=20, omitting additional logit-penalty parameters for simplicity. The standard thinking budget is 32,768 tokens and the maximum generation length is 36,864 tokens. The Qwen3.5-27B native-MTP setting uses a 65,536-token t...

    2023

  31. [40]

    Overall, broader results across Figs

    These figures use the same capability-speed grid as the main body, without the highlighted annotations, so the broader behaviour across tasks can be inspected directly. Overall, broader results across Figs. 7 to 10 match the takeaways and findings in the main body of the paper...

Pith tools

Reviewed July 10, 2026 · model on record in the stance chip above.