Pith. sign in

REVIEW 3 major objections 5 minor 41 references

Consultant Decoding: Yet Another Synergistic Mechanism

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Consultant Decoding replaces speculative decoding's likelihood-ratio check with a fixed threshold of about 2.0 on the target model's token-level negative log-likelihood, claiming up to 2.5x speedup at near-full quality.

desk verdict Simple, well-tested threshold-based verification rule for speculative decoding; the experiments are stronger than the theory, and the 'around 100% quality' headline needs caveats. read the letter →

arxiv 2506.02391 v1 pith:ZUMX3O6N submitted 2025-06-03 cs.CL cs.AI

Chuanghao Ding , Jiaping Wang , Ziqing Yang , Xiaoliang Wang , Dahua Lin , Cam-Tu Nguyen , Fei Tan This is my paper
classification cs.CLcs.AI
keywords speculativedecodinginferenceaccelerationtokenverificationnegativelog-likelihoodChinchillascalinglawnucleussamplinglargelanguagemodelsdraft-verify
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Consultant Decoding asks whether a draft token is acceptable by checking the large target model's own negative log-likelihood for that token, and accepts it when that value falls below a fixed threshold of about 2.0. The paper argues this small change does what speculative decoding's likelihood-ratio verification fails to do: it admits far more draft tokens without admitting low-quality ones, because a token the target model finds as predictable as a typical training token is likely a good continuation. Across math, code, chat, and instruction-following benchmarks, the paper reports up to a 2.5-fold wall-clock speedup over greedy target decoding, output quality at roughly 100 percent of the target model, and large-model call ratios below 10 percent. The claim matters because it would turn speculative decoding into a training-free, parameter-free, drop-in acceleration where the draft model can be orders of magnitude smaller than the target.

What carries the argument

The load-bearing object is the acceptance rule $V_{\mathrm{CD}}(x_i)=\varepsilon - (-\log p_i(x_i))$ together with its EMA-smoothed version $V(x_i)=\varepsilon-\mathrm{EMA}_\beta(r_i)$, where $r_i=-\log p_i(x_i)$ and $\beta=0.2$ in the generic setting. This is a single-token perplexity filter: the draft token is treated as pseudo ground truth, and it is accepted only if the target model assigns it a negative log-likelihood at or below the training-time convergence loss. The paper links the rule to nucleus sampling by observing that when $\beta\to 0$, acceptance is equivalent to $p_i(x_i) > e^{-\varepsilon}$, so an accepted draft token lies inside the target model's high-probability nucleus of mass $1-e^{-\varepsilon}$; this analogy is what is supposed to explain why quality is preserved with a fixed threshold.

What would settle it

Run CD with the generic $\varepsilon=2.0$ on a broad held-out suite using a model whose Chinchilla-estimated convergence loss is far from 2.0, then compare generation quality and acceptance rate against the target model's greedy decoding; if quality falls noticeably below the target baseline while acceptance stays high, the generic-threshold claim is refuted. The paper's own speed-optimal tables already show best thresholds from 4.0 to 7.5, so an even sharper test is to check whether CD with $\varepsilon=2.0$ underperforms on exactly those tasks or on a new task with a different optimal threshold.

Watch

Extended reading notes

Core claim

The central claim is that token correctness for speculative decoding can be verified by a target-model-only likelihood test rather than by a ratio between draft and target distributions. Formally, a draft token $x_i$ is accepted when $V_{\mathrm{CD}}(x_i)=\varepsilon - (-\log p_i(x_i)) > 0$, with $\varepsilon\approx 2.0$ estimated from the Chinchilla convergence loss and smoothed by an exponential moving average over recent tokens. The paper argues that this aligns the acceptance criterion with the target model's own training objective, preserves the target model's output distribution approximately in the sense of Top-P sampling, and on some tasks even lets the draft-guided path outperform the target model's greedy decoding. The empirical claim is that this yields speedups of roughly 2x to 3x with quality near 100 percent of the target model and large-model calls below 10 percent, including on a 0.5B/72B draft-target pair.

Load-bearing premise

The whole method rests on a single number: that a fixed threshold of about 2.0 on the target model's negative log-likelihood marks the boundary between acceptable and unacceptable draft tokens across models, tasks, and contexts.

Editorial extensions

If this is right

  • Speculative decoding can be accelerated without retraining, without per-task tuning, and without a separate acceptance threshold per model pair, because the generic setting uses one value $\varepsilon=2.0$.
  • Draft models much smaller than the target become practical: a 0.5B draft paired with a 72B target still yields speedups of roughly 2x to 3x, so the method scales to model pairs with two orders of magnitude in parameter count.
  • Longer draft windows become efficient: the speedup lost when draft length grows from 6 to 20 is much smaller for CD than for speculative decoding or Mentored Decoding, reducing the need to tune draft length.
  • Large-model call frequency drops below 10 percent on reasoning and coding tasks, which directly increases the throughput available for parallel serving.
  • The verification module can be swapped into self-drafting frameworks such as EAGLE-2 and still add speed, so the mechanism is not tied to a separate small draft model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • This suggests that acceptance in speculative decoding may be better predicted by the target model's calibration than by the agreement between draft and target distributions; if so, per-model thresholds could be learned from a few held-out loss statistics instead of searched per task.
  • The reported cases where CD outperforms the greedy target model point to a voting or ensemble effect between the draft and target models, which could be tested directly by using several small draft models simultaneously and counting votes.
  • Since the rule is just a threshold on the target's own loss, it should compose naturally with block-wise, tree-based, and multi-candidate drafts, potentially pushing large-model call ratios even lower without extra training.
  • A practical extension would be to expose $\varepsilon$ as a latency-quality dial: the paper's own tables show optimal thresholds between 4.0 and 7.5 for speed-optimal settings, and a learned per-request epsilon could let a server trade accuracy for throughput continuously.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Consultant Decoding (CD), a modification of the verification step in speculative decoding. Instead of accepting a draft token with probability min(1, p/q) as in speculative sampling, CD accepts token x_i when the EMA-smoothed negative log-likelihood -log p_i(x_i) of the target model is below a fixed threshold ε, set to ε=2.0 in the 'Generic' configuration using the Chinchilla scaling-law estimate of the convergence loss. On rejection, CD resamples from the target distribution at the rejection position. The authors claim speedups up to 2.5x in the generic setting (and >3x in a speed-optimal configuration) with quality around 100% of the target model and large-model call ratios below 10%, and they present experiments on GSM8K, HumanEval, MT-Bench, AlpacaEval, a business dataset, on Qwen2.5 0.5B/1.5B/3B/7B/72B and Llama-3.1-8B/70B pairs, and on EAGLE-2. Section 3.4 argues that CD is approximately equivalent to Top-P sampling with nucleus size 1-e^{-ε}.

Significance. If the empirical results hold, CD is a simple, training-free, drop-in alternative to speculative sampling: no distribution matching is required, and the verification rule uses only target-model likelihoods. The paper's strengths are the breadth of the benchmark coverage (four public tasks plus a business benchmark; two model families; independent and self-drafting settings), the inclusion of sampling-decoding results, and a clear algorithm. The method is easy to reproduce from Algorithm 1. The principal caveat is that the paper's theoretical rationale for the fixed threshold is approximate, and the headline quality claim is stronger than some of the reported numbers (especially Table 8). The empirical generic-setting results are nevertheless substantial and would be useful to the speculative-decoding community if the claims are appropriately qualified.

major comments (3)
  1. [Section 3.4] The claimed equivalence to Top-P sampling is not mathematically established. The derivation requires β→0, but Algorithm 1 and the Generic setting use β=0.2, so the EMA smoothing is not negligible. More importantly, the acceptance criterion p(x_i) > e^{-ε} is a per-token probability threshold, while a Top-P nucleus is defined by cumulative probability mass; these sets do not coincide. Finally, upon rejection CD samples x'_{n+1} from the full target distribution p_{n+1}(·), so the final output distribution includes tokens outside any fixed nucleus. Consequently the sentence 'CD shares similar property of Top-P sampling' and the Conclusion's 'theoretical basis' overstate the support. Please either provide a formal statement (e.g., a divergence bound or a distributional identity) or explicitly label Section 3.4 as intuition.
  2. [Section 3.3 and Table 15] The fixed threshold ε=2.0 is load-bearing for the generic-setting quality claim. The derivation from the Chinchilla convergence loss gives only a training-set average; it does not imply that per-token NLL below 2.0 separates correct from incorrect draft tokens. Table 15 reports speed-optimal thresholds of 4.0 to 7.5 for the very same model pairs and tasks, and the Limitations section concedes that the optimal threshold depends on the question and model combination. The paper should add an analysis of the NLL distributions of accepted/rejected tokens under ε=2.0 (or an ablation across model families and tasks) to substantiate the generic threshold; without it, the generic-setting quality result remains an empirical finding limited to Qwen2.5 and Llama-3.1.
  3. [Abstract, Section 4.1.1, and Table 8] The headline claim of 'comparable generation quality (around 100% of the target model's performance)' is not consistent with Appendix A.1. In the Speed-Optimal setting on the business dataset, CD achieves 83.5% of the target score (Table 8), a 16.5% drop, while Section 4.1.1 says the speed-optimal setting allows a performance decrease 'within 5%' of the target model. The abstract and Section 4.1.1 need to be qualified (e.g., restrict the 'around 100%' claim to the Generic setting), and the speed-optimal parameter search should either enforce the stated 5% bound or report the exception transparently.
minor comments (5)
  1. [Section 3.2] In the paragraph after Eq. (6), 'modls' should be 'models'; the sentence 'otherwise we sample a new token from the distribution given by the target model' would read more clearly as 'otherwise we sample a new token from the distribution given by the target model at the rejection position.'
  2. [Figure 2] The figure contains the placeholder text 'Re-Sampling from ??' in the diagram; this needs to be cleaned up, and the label should state explicitly that the resampling is from the full target distribution p_{n+1}(·), since that is precisely the point where CD differs from Top-P sampling.
  3. [Algorithm 1] Line 17 computes r_{n+1} and returns it, but no subsequent iteration uses this value as shown; if it is intended as the EMA state for the next drafting block, say so explicitly, otherwise remove it.
  4. [Section 4] The normalization description says metrics are normalized to the target model's greedy decoding unless otherwise specified, but Table 9 reports sampling-decoding results; it should be stated explicitly whether those normalized percentages are relative to greedy target output or to a sampling baseline, as this affects interpretation.
  5. [Conclusion] The Conclusion says the paper provides a 'theoretical analysis' and a 'theoretical basis' for the threshold; given the approximations acknowledged in Section 3.4, it would be more accurate to describe the support as an empirical study plus an approximate analogy.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the epsilon threshold is externally anchored by the Chinchilla scaling law, quality is empirically benchmarked, and the tuned speed-optimal results are explicitly labeled as such.

full rationale

Consultant Decoding's derivation is not circular. The acceptance rule V_CD = epsilon - (-log p_i(x_i)) (Eq. 6) depends on the target model's own token likelihoods and on epsilon = 2.0, which is imported from the Chinchilla scaling law (Eq. 9), an external fit by Hoffmann et al., not from the paper's own benchmark outputs or from a quantity equivalent to the claimed result. The quality-preservation claim is then validated empirically against GSM8K, HumanEval, MT-Bench, and AlpacaEval relative to target greedy decoding, so it is not forced by construction. The Top-P consistency argument in Section 3.4 is explicitly approximate (beta approaches 0 and a nucleus of size 1 - e^{-epsilon}); even if the analogy is imperfect, an approximate or unsound justification is a correctness risk, not circularity. The Speed-Optimal tables (Table 1 and Appendix C, Table 15) are explicitly obtained by grid search over threshold and draft length and are presented as such, while the paper separately reports the untuned Generic setting (epsilon = 2.0, beta = 0.2), so the tuned numbers are not disguised predictions. Self-citations appear in the reference list (e.g., Gu et al. 2024, Lu et al. 2023), but none is load-bearing for the CD derivation. The paper's own Limitations section acknowledges that epsilon = 2.0 is not optimal and depends on the task and model combination, which weakens the generality claim but does not make the derivation circular.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central mechanism rests on one free threshold (epsilon), one smoothing constant (beta), and tuned draft lengths for optimal-speed results. The main axioms are heuristic: the Chinchilla convergence loss estimates the right threshold, and threshold acceptance approximates Top-P sampling. No new entities are introduced.

free parameters (4)
  • epsilon (acceptance threshold) = 2.0 (generic); 4.0-7.5 (speed-optimal per task/model)
    Defines V_CD in Eq. 6. Generic value is estimated from Chinchilla convergence loss, but speed-optimal results grid-search it per dataset and model (Table 15).
  • EMA decay beta = 0.2
    Chosen by hand in the Generic setting; no sweep is reported, yet it controls the smoothing in Eq. 7-8 and affects speed and quality.
  • draft length gamma = 6 (generic), 8-25 (speed-optimal)
    Tuned per task and model in Table 15 for speed-optimal results; affects acceptance length and LLM-call ratio.
  • temperature (sampling experiments) = 0.7
    Fixed for sampling decoding experiments in Table 9; not central to the main claim but a setting for those results.
assumptions (5)
  • domain assumption Chinchilla scaling law L(N,D)=E+A/N^alpha+B/D^beta with E=1.69, A=406.4, B=410.7, alpha=0.34, beta=0.28 correctly estimates the convergence loss of Qwen2.5 and Llama-3.1 target models.
    Used in Section 3.3 to justify epsilon=2.0 as the generic threshold; the law was fit to a different model family, and the paper calls it a rough estimate.
  • ad hoc to paper A draft token is acceptable iff its target-model NLL is below the training convergence loss.
    Core acceptance criterion in Eq. 6-7; no theorem proves that this preserves generation quality; the paper offers an informal convergence-loss and Top-P analogy in Section 3.4.
  • ad hoc to paper Threshold acceptance on p_i(x_i) is approximately equivalent to Top-P sampling with nucleus size 1-e^{-epsilon}.
    Used in Section 3.4 to argue accuracy preservation; mathematically the two criteria differ because Top-P is cumulative over sorted probabilities, not a per-token threshold.
  • domain assumption Walltime speedup measured under the authors' inference setup transfers to other engines and hardware.
    No code or engine details are provided; speedup ratios depend on batching, kernel efficiency, and draft model speed.
  • domain assumption Benchmark metrics (GSM8K accuracy, HumanEval pass@1, MT-Bench score, AlpacaEval win rate) are adequate proxies for generation quality.
    Standard practice in the field; used throughout Section 4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Consultant Decoding: Yet Another Synergistic Mechanism." pith.science (2026). https://pith.science/paper/ZUMX3O6N

@misc{pith2026250602391,
  author       = {Pith},
  title        = {Pith review of: Consultant Decoding: Yet Another Synergistic Mechanism},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZUMX3O6N}},
  note         = {Machine review of arXiv:2506.02391}
}
read the original abstract

The synergistic mechanism based on Speculative Decoding (SD) has garnered considerable attention as a simple yet effective approach for accelerating the inference of large language models (LLMs). Nonetheless, the high rejection rates require repeated LLMs calls to validate draft tokens, undermining the overall efficiency gain of SD. In this work, we revisit existing verification mechanisms and propose a novel synergetic mechanism Consultant Decoding (CD). Unlike SD, which relies on a metric derived from importance sampling for verification, CD verifies candidate drafts using token-level likelihoods computed solely by the LLM. CD achieves up to a 2.5-fold increase in inference speed compared to the target model, while maintaining comparable generation quality (around 100% of the target model's performance). Interestingly, this is achieved by combining models whose parameter sizes differ by two orders of magnitude. In addition, CD reduces the call frequency of the large target model to below 10%, particularly in more demanding tasks. CD's performance was even found to surpass that of the large target model, which theoretically represents the upper bound for speculative decoding.

Figures

Figures reproduced from arXiv: 2506.02391 by the authors.

Figure 1
Figure 1. (a) A visual description of the sub-optimal [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The comparison between Consultant Decod [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. A comparison of the decrease in speedup ratios among SD, MD, and CD verification methods as the draft length increases from 6 to 20, using greedy decoding. Here target and draft models are 72B and 0.5B models, respectively. on GSM8K and HumanEval datasets, using the Qwen2.5-0.5B as the draft model and the Qwen2.5- 72B as the target model, CD achieves the lowest LLM-calls ratio of 9.1%. On HumanEval dataset, with the… view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: The figure shows the scalability of average [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

41 extracted references · 11 canonical work pages

  1. [1]

    Oscar Brown, Zhengjie Wang, Andrea Do, Nikhil Mathew, and Cheng Yu. 2024. Dynamic depth decoding: Faster speculative decoding for llms. arXiv preprint arXiv:2409.00142

    arXiv 2024

  2. [2]

    Medusa: Simple llm inference acceleration framework with multiple decoding heads, 2024

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple llm inference acceleration framework with multiple decoding heads, 2024. URL https://arxiv. org/abs/2401.10774

    arXiv 2024

  3. [3]

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318

    arXiv 2023

  4. [4]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    arXiv 2021

  5. [5]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    arXiv 2021

  6. [6]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

  7. [7]

    Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. 2024. Break the sequential dependency of llm inference using lookahead decoding. arXiv preprint arXiv:2402.02057

    arXiv 2024

  8. [8]

    Jiawei Gu, Zacc Yang, Chuanghao Ding, Rui Zhao, and Fei Tan. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.903 CMR scaling law: Predicting critical mixture ratios for continual pre-training of language models . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16143--16162

    doi:10.18653/v1/2024.emnlp-main.903 2024

Show all 41 references
  1. [9]

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556

    2022 arXiv

  2. [10]

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751

    2019 arXiv

  3. [11]

    Accelerated speculative sampling based on tree monte carlo

    Zhengmian Hu and Heng Huang. Accelerated speculative sampling based on tree monte carlo. In Forty-first International Conference on Machine Learning

  4. [12]

    Kaixuan Huang, Xudong Guo, and Mengdi Wang. 2024. Specdec++: Boosting speculative decoding via adaptive candidate lengths. arXiv preprint arXiv:2405.19715

    2024 arXiv

  5. [13]

    Sehoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W Mahoney, Amir Gholami, and Kurt Keutzer. 2024. Speculative decoding with big little decoder. Advances in Neural Information Processing Systems, 36

    2024

  6. [14]

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274--19286. PMLR

    2023

  7. [15]

    Dedong Li, Rui Zhao, and Fei Tan. 2023 a . Cwseg: An efficient and general approach to chinese word segmentation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), pages 1--10

    2023

  8. [16]

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023 b . Alpacaeval: An automatic evaluator of instruction-following models

    2023

  9. [17]

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024 a . Eagle-2: Faster inference of language models with dynamic draft trees. arXiv preprint arXiv:2406.16858

    2024 arXiv

  10. [18]

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024 b . Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint arXiv:2401.15077

    2024 arXiv

  11. [19]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

    2024 arXiv

  12. [20]

    Chengbo Liu and Yong Zhu. 2024. Sdsat: Accelerating llm inference through speculative decoding with semantic adaptive tokens. arXiv preprint arXiv:2403.18647

    2024 arXiv

  13. [21]

    Jinghui Lu, Dongsheng Zhu, Weidong Han, Rui Zhao, Brian Mac Namee, and Fei Tan. 2023. What makes pre-trained language models better zero-shot learners? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2288--2303

    2023

  14. [22]

    Jonathan Mamou, Oren Pereg, Daniel Korat, Moshe Berchansky, Nadav Timor, Moshe Wasserblat, and Roy Schwartz. 2024. Dynamic speculation lookahead accelerates speculative decoding of large language models. arXiv preprint arXiv:2405.04304

    2024 arXiv

  15. [23]

    Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, et al. 2024. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. In Proceedings...

    2024

  16. [24]

    Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Seungyeon Kim, Neha Gupta, Aditya Krishna Menon, and Sanjiv Kumar. 2024. Faster cascades via speculative decoding. arXiv preprint arXiv:2405.19261

    2024 arXiv

  17. [25]

    Ziteng Sun, Uri Mendlovic, Yaniv Leviathan, Asaf Aharoni, Ahmad Beirami, Jae Hun Ro, and Ananda Theertha Suresh. 2024. Block verification accelerates speculative decoding. In Workshop on Efficient Systems for Foundation Models II@ ICML2024

    2024

  18. [26]

    Vivien Tran-Thien. 2023. https://vivien000.github.io/blog/journal/a-provably-optimal-lossy-variant-of-speculative-decoding.html#references An optimal lossy variant of speculative decoding

    2023

  19. [27]

    Jikai Wang, Yi Su, Juntao Li, Qingrong Xia, Zi Ye, Xinyu Duan, Zhefeng Wang, and Min Zhang. 2024 a . Opt-tree: Speculative decoding with adaptive draft tree structure. arXiv preprint arXiv:2406.17276

    2024 arXiv

  20. [28]

    Shiqi Wang, Zhengze Zhang, Rui Zhao, Fei Tan, and Nguyen Cam-Tu. 2024 b . https://doi.org/10.18653/v1/2024.findings-emnlp.115 Reward difference optimization for sample reweighting in offline RLHF . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages...

    2024 doi

  21. [29]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  22. [30]

    Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. 2023. Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3909--3925

    2023

  23. [31]

    Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. 2024. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. arXiv preprint arXiv:2401.07851

    2024 arXiv

  24. [32]

    Bin Xiao, Lujun Gui, Lei Su, and Weipeng Chen. 2024 a . Clover-2: Accurate inference for regressive lightweight speculative decoding. arXiv preprint arXiv:2408.00264

    2024 arXiv

  25. [33]

    Bin Xiao, Chunan Shi, Xiaonan Nie, Fan Yang, Xiangwei Deng, Lei Su, Weipeng Chen, and Bin Cui. 2024 b . Clover: Regressive lightweight speculative decoding with sequential knowledge. arXiv preprint arXiv:2405.00263

    2024 arXiv

  26. [34]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024 a . Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  27. [35]

    Sen Yang, Shujian Huang, Xinyu Dai, and Jiajun Chen. 2024 b . Multi-candidate speculative decoding. arXiv preprint arXiv:2401.06706

    2024 arXiv

  28. [36]

    Ofir Zafrir, Igor Margulis, Dorin Shteyman, and Guy Boudoukh. 2024. Fastdraft: How to train your draft. arXiv preprint arXiv:2411.11055

    2024 arXiv

  29. [37]

    Hengyuan Zhang, Yanru Wu, Dawei Li, Sak Yang, Rui Zhao, Yong Jiang, and Fei Tan. 2024. Balancing speciality and versatility: a coarse to fine framework for supervised fine-tuning large language model. In Findings of the Association for Computational Linguistics ACL 2024, pages...

    2024

  30. [38]

    Weilin Zhao, Yuxiang Huang, Xu Han, Wang Xu, Chaojun Xiao, Xinrong Zhang, Yewei Fang, Kaihuo Zhang, Zhiyuan Liu, and Maosong Sun. 2024. Ouroboros: Generating longer drafts phrase by phrase for faster speculative decoding. In Proceedings of the 2024 Conference on Empirical Meth...

    2024

  31. [39]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595--46623

    2023

  32. [40]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  33. [41]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.