Pith. sign in

REVIEW 2 major objections 5 minor 2 cited by

AdaEAGLE: Optimizing Speculative Decoding via Explicit Modeling of Adaptive Draft Structures

T0 review · 2 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read AdaEAGLE predicts the optimal draft length at each decoding step with a lightweight three-layer MLP, reaching a 1.62x speedup over vanilla autoregressive decoding and beating the best fixed-length EAGLE baseline on six benchmarks while…

desk verdict Plausible, modest extension of EAGLE; the 'optimality' claim exceeds the evidence, but the method is simple, reproducible, and deserves peer review. read the letter →

arxiv 2412.18910 v1 pith:LAJVBKQJ submitted 2024-12-25 cs.AI cs.CL

Situo Zhang , Hankun Wang , Da Ma , Zichen Zhu , Lu Chen , Kunyao Lan , Kai Yu This is my paper · ORCID
classification cs.AIcs.CL
keywords speculativedecodingadaptivedraftlengthpredictionEAGLELLMinferenceaccelerationhiddenstatelosslessgreedy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Speculative decoding, in which a small model drafts tokens that a large model verifies in one parallel pass, usually fixes the draft length in advance. The paper argues that this is wasteful because the number of draft tokens the target model will accept fluctuates sharply from step to step. AdaEAGLE therefore adds a Lightweight Draft Length Predictor (LDLP), a three-layer MLP that reads the last verified token's embedding and hidden state and predicts how many tokens to draft before drafting begins. Under greedy decoding, LDLP is trained to match the acceptance length, and the paper proves, under a monotonicity assumption, that drafting exactly the acceptance length minimizes target-model forward passes. On six benchmarks with Vicuna-7B, AdaEAGLE reaches 64.44 tokens/s versus 62.88 tokens/s for the best fixed-length EAGLE, and combining it with the threshold-based method DDD reaches 65.20 tokens/s, a 1.62x speedup over vanilla autoregressive decoding with unchanged output quality.

What carries the argument

The load-bearing component is the Lightweight Draft Length Predictor (LDLP), a three-layer MLP with residual connections. It takes as input the embedding $e_j$ and the final-layer-normalized hidden state $f_j$ of the last accepted token, computes a scalar score, and outputs the next draft length as $\bar{k}_{r+1} = \mathrm{Round}(\mathrm{MLP}(e_j, f_j))$, clamped to $[0, k_{max}]$. Training labels are produced offline by comparing a target-generated continuation with a $k_{max}$-token EAGLE draft and taking the longest common prefix; the loss is $\lambda|\bar{k} - k^\circ|$ when the prediction is too short and $|\bar{k} - k^\circ|$ otherwise, with $\lambda > 1$. The mechanism carries the argument by replacing a manually chosen fixed draft length with a context-dependent prediction derived from the hidden state's plan for future tokens.

What would settle it

Run greedy EAGLE decoding on a fixed prompt set and measure $p_{accept}(\hat{t}_m | j)$ for drafts conditioned on true verified histories of length $j$ and on shorter draft-influenced histories $j' < j$; if any longer history yields lower acceptance probability, Assumption 1 is violated. Then compare total target forward passes for drafts stopped exactly at the first rejection versus one token earlier: if the shorter draft ever reaches the same formal output with fewer target forward passes, Lemma 1's optimality claim collapses.

Watch

Extended reading notes

Core claim

The central claim is that the optimal draft length for EAGLE-style speculative decoding is the acceptance length: the largest prefix of the draft that the target model accepts, provided that longer official token histories never make the draft less accurate (Assumption 1 in the appendix). Lemma 1 shows that under that assumption, stopping the draft exactly at the first rejection minimizes target forward passes. AdaEAGLE turns this length into a predicted quantity: the LDLP module takes the embedding $e_j$ and final-layer-normalized hidden state $f_j$ of the last accepted token and outputs a clamped, rounded scalar draft length, trained with a penalized L1 loss that over-penalizes predictions shorter than the true acceptance length because a target forward pass costs about 20 times a draft step. The paper reports that this explicit length control outperforms every fixed-length EAGLE setting and matches or exceeds the threshold-based DDD baseline across six benchmarks, with AdaEAGLE at 64.44 tokens/s and AdaEAGLE-DDD at 65.20 tokens/s.

Load-bearing premise

The proof that drafting exactly the acceptance length is optimal assumes that a longer verified history never makes the draft model's guesses worse, and that monotonicity is treated as reasonable rather than tested.

Editorial extensions

If this is right

  • Fixed draft length disappears as a hyperparameter: one AdaEAGLE model adapts across tasks, drafting longer on predictable code and shorter on more dynamic question-answering, without per-dataset tuning.
  • Because the target model's verification step is unchanged, the acceleration is lossless and the output distribution remains identical to vanilla autoregressive decoding.
  • Explicit length prediction and threshold-based stopping are complementary: AdaEAGLE-DDD achieves the highest throughput on four of six benchmarks, reaching 65.20 tokens/s.
  • The measured gap to the EAGLE-Oracle topline, 83.46 versus 66.35 tokens/s on MT-Bench, shows the remaining headroom for adaptive draft-length modeling.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if Assumption 1 ever fails, meaning a longer verified history can make the draft worse, then the LDLP training label is no longer guaranteed to minimize latency, and stopping one token before the apparent acceptance boundary could occasionally be faster.
  • Editorial inference: the penalized L1 surrogate could be replaced by a latency-aware objective, such as minimizing expected target forward passes per token directly, which might close more of the gap to the EAGLE-Oracle topline.
  • Editorial inference: extending this explicit-modeling idea to tree-based and non-greedy decoding would require predicting width and depth of subtrees, not just a single length; the LDLP design suggests the same lightweight predictor family could be reused there.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes AdaEAGLE, an extension of the EAGLE speculative decoding framework in which a lightweight MLP (LDLP) predicts, at each decoding iteration, the number of draft tokens to be generated. The predictor takes the embedding and final-layer-normalized hidden state of the last accepted token and is trained with an asymmetric L1 loss against the length of the longest common prefix between draft and target outputs. Experiments on Vicuna-7B across six benchmarks report average throughput of 64.44 Tok/s for AdaEAGLE and 65.20 for AdaEAGLE-DDD, versus 62.88 for the best fixed-length EAGLE and 64.43 for DDD. An oracle-based analysis in Table 2 motivates the adaptive approach by showing a large potential gain, and ablations support the loss penalty. The paper claims this is the first explicit model of adaptive draft structures and that LDLP predicts the optimal draft length.

Significance. The contribution is practically relevant: the base variant removes manual thresholds, the predictor is cheap, and the combination with DDD improves throughput on four of six benchmarks relative to the compared baselines. The oracle analysis in §2.4 is a useful way to quantify headroom, and the limitations section is candid about the sequential and greedy scope. The strengths are the clean architecture, the ablation showing the benefit of the loss penalty, and the explicit modeling idea that can be transferred to other speculative decoding frameworks. However, the 'optimality' framing is not fully supported, and the throughput advantage over the best fixed-length baseline is small enough that variance reporting is needed before the headline claim can be taken at face value.

major comments (2)
  1. [Appendix A; §2.3; §3.3] The optimality claim for the predicted draft length is load-bearing and depends on an untested monotonicity assumption. In Appendix A, the paper explicitly notes that a shorter draft can change the hidden-state trajectory and produce an accepted token that a longer draft would miss, and it then introduces Assumption 1 (paccept(ˆtm|j) ≥ paccept(ˆtm|j′) for j′<j) to prove Lemma 1 that the acceptance length equals the latency-optimal draft length. Section 3.3 uses exactly this quantity, the longest common prefix k_j, as the LDLP training label, and Table 2's EAGLE-Oracle is presented as an upper bound on achievable throughput. If Assumption 1 fails on real data, the training labels are not latency-optimal and the oracle need not bound the achievable speedup. Please validate the monotonicity empirically on the evaluation benchmarks, or replace the acceptance-length oracle with a true latency-minimizing oracle, or revise the abstract and Section 2.3 to say 'predicted acceptance length' rather than 'optimal draft length'.
  2. [Table 3; §4.2] The headline empirical gain is not supported by variance estimates. The average throughput difference between AdaEAGLE (64.44 Tok/s) and the best fixed-length EAGLE (62.88) is about 2.5%, and AdaEAGLE is not uniformly better per benchmark (for example, HumanEval: EAGLE-5 77.95 vs AdaEAGLE 77.55; Natural Questions: EAGLE-3 57.44 vs AdaEAGLE 57.41). No error bars, repeated runs, or multiple seeds are reported for a single-GPU batch-1 throughput measurement that is known to be sensitive to hardware noise. Please report at least 3-5 runs with standard deviations, or provide a paired comparison on identical hardware, to establish that the average gain is not within noise.
minor comments (5)
  1. [§4.1; Eq. (3); Eq. (4)] Please specify the numerical values of λ in Eq. (4) and kmax in Eq. (3) in the experimental setup; these are needed to reproduce the reported results.
  2. [Abstract; §6] The abstract's '1.62x speedup' and the conclusion's '1.61'/'1.61%' do not obviously match Table 3 (65.20/40.56 ≈ 1.61, 64.44/40.56 ≈ 1.59); state the aggregation rule used for the reported speedup ratio.
  3. [§3.1; Table 3 caption; §6] There are several typos: 'a a Lightweight' in §3.1, 'accpeted' in the Table 3 caption, and 'avxchieves' and '1.61%' in §6.
  4. [Table 4; §4.3.2] The classification baseline ('w/ Cls') is not described in enough detail; specify the class set (for example, binned lengths) and the loss used.
  5. [Appendix A] The statement of Assumption 1 would be clearer if the positions j and j′ were explicitly tied to the iteration index in which the draft begins; consider adding a short notational remark.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: LDLP is trained to predict acceptance length and its speedup is measured empirically; the 'optimal' claim rests on an explicit, untested assumption, not on a definitional or fitted circularity.

full rationale

The paper's central derivation chain is self-contained and not circular. LDLP is a three-layer MLP trained in Section 3.3 with a penalized L1 loss on labels obtained by comparing draft outputs with target outputs: 'compare T_out to T_out and calculate the length kj of the longest common prefix between them.' The reported throughput numbers in Tables 2 and 3 come from running the trained predictor on held-out benchmarks; no measured Tok/s value is produced by fitting constants to those benchmarks. The only load-bearing conceptual link is the identification of the latency-optimal draft length with the acceptance length, and that link is explicitly conditional: Section 2.3 states 'the optimal draft length is exactly the acceptance length (need an additional assumption, see the proof in Appendix A)', and Appendix A provides Assumption 1 and Lemma 1 for this purpose. The paper even concedes that without the assumption, 'generating it greedily is no longer guaranteed optimal,' so the 'optimal' vocabulary is supported by an explicit, testable assumption rather than smuggled in by definition or self-citation. No load-bearing self-citations are present; references to EAGLE, DDD, and prior hidden-state work are external baselines. The Limitations section candidly scopes the method to sequential greedy decoding. The untested status of Assumption 1 is a correctness and robustness concern, not circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

No independent physical or structural entities are postulated. LDLP is a trainable module whose predictions are directly evaluable against draft and target behavior, so it does not belong in the invented-entities category. The free parameters are the hand-set loss penalty, the draft length cap, and the combined-method threshold; the MLP weights themselves are learned from data and are not listed as ad hoc constants.

free parameters (3)
  • lambda (loss penalty) = not reported (stated only as lambda > 1)
    Eq. (4) scales the L1 loss when LDLP underpredicts; chosen by hand, and the ablation in Table 4 shows it affects throughput (66.35 vs 64.62).
  • kmax (maximum draft length) = not reported
    Eq. (3) sets the upper bound on predicted draft length; the paper says kmax can differ between training and inference but does not give the value used.
  • DDD threshold in AdaEAGLE-DDD = -0.6 (log probability)
    Table 5 tunes this threshold on MT-Bench; the authors report small throughput differences across -0.2 to -1.0 and select -0.6 for the main results.
assumptions (3)
  • domain assumption Assumption 1: paccept(ˆtm|j) >= paccept(ˆtm|j′) whenever j′ < j, i.e., longer official token history cannot hurt draft accuracy.
    Appendix A, Eq. (5); used to prove Lemma 1 that the optimal draft length equals the acceptance length, which makes the LDLP training label and the EAGLE-Oracle upper bound latency-optimal. It is plausible but not tested.
  • domain assumption The last token's hidden state and embedding contain enough information to predict how many draft tokens will be accepted.
    Section 3.2, based on cited prior work on future-token planning; this is the core modeling assumption behind LDLP and is not independently verified in this paper.
  • domain assumption An LDLP trained on ShareGPT transfers to the six evaluation benchmarks.
    Section 4.1 trains only on ShareGPT and evaluates on MT-Bench, Alpaca, HumanEval, GSM8K, CNN/Daily Mail, and Natural Questions without reporting calibration or domain adaptation analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AdaEAGLE: Optimizing Speculative Decoding via Explicit Modeling of Adaptive Draft Structures." pith.science (2026). https://pith.science/paper/LAJVBKQJ

@misc{pith2026241218910,
  author       = {Pith},
  title        = {Pith review of: AdaEAGLE: Optimizing Speculative Decoding via Explicit Modeling of Adaptive Draft Structures},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LAJVBKQJ}},
  note         = {Machine review of arXiv:2412.18910}
}
abstract

Speculative Decoding (SD) is a popular lossless technique for accelerating the inference of Large Language Models (LLMs). We show that the decoding speed of SD frameworks with static draft structures can be significantly improved by incorporating context-aware adaptive draft structures. However, current studies on adaptive draft structures are limited by their performance, modeling approaches, and applicability. In this paper, we introduce AdaEAGLE, the first SD framework that explicitly models adaptive draft structures. AdaEAGLE leverages the Lightweight Draft Length Predictor (LDLP) module to explicitly predict the optimal number of draft tokens during inference to guide the draft model. It achieves comparable speedup results without manual thresholds and allows for deeper, more specialized optimizations. Moreover, together with threshold-based strategies, AdaEAGLE achieves a $1.62\times$ speedup over the vanilla AR decoding and outperforms fixed-length SotA baseline while maintaining output quality.

Figures

Figures reproduced from arXiv: 2412.18910 by the authors.

Figure 1
Figure 1. Motivation: finding the optimal draft stop [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Maximum draft lengths accepted at different [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Illustration of AdaEAGLE framework. (a) The architecture of AdaEAGLE. The draft length are predicted [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Comparison of draft length distributions with [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Draft Model Knows When to Stop: Self-Verification Speculative Decoding for Long-Form Generation

    cs.CL 2024-11 conditional novelty 5.0 of 10

    Speculative decoding can be accelerated by letting the draft model decide when to stop based on its prediction entropy, yielding up to 17-22% speedups in long-form generation.

  2. CURE: Local Uncertainty Repair for Block-Parallel Speculative Decoding

    cs.AI 2026-08 conditional novelty 4.0 of 10

    Adding confidence-gated repair branches to block-parallel drafts increases accepted token count slightly but, in the reported setup, reduces end-to-end speedup relative to the repair-free parallel baseline.

Reference graph

Works this paper leans on

37 extracted references · 5 canonical work pages · cited by 2 Pith papers

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Oscar Brown, Zhengjie Wang, Andrea Do, Nikhil Mathew, and Cheng Yu. 2024. https://arxiv.org/abs/2409.00142 Dynamic depth decoding: Faster speculative decoding for llms . Preprint, arXiv:2409.00142

    arXiv 2024

  4. [4]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    2020

  5. [5]

    Lee, Deming Chen, and Tri Dao

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. 2024. https://proceedings.mlr.press/v235/cai24b.html Medusa: Simple LLM inference acceleration framework with multiple decoding heads . In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, ...

    work page 2024

  6. [6]

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. https://arxiv.org/abs/2302.01318 Accelerating large language model decoding with speculative sampling . Preprint, arXiv:2302.01318

    arXiv 2023

  7. [7]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    arXiv 2021

  8. [8]

    Zhuoming Chen, Avner May, Ruslan Svirschevski, Yuhsun Huang, Max Ryabinin, Zhihao Jia, and Beidi Chen. 2024. https://arxiv.org/abs/2402.12374 Sequoia: Scalable, robust, and hardware-aware speculative decoding . Preprint, arXiv:2402.12374

    arXiv 2024

Show all 37 references
  1. [9]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. https://lmsys.org/blog/2023-03-30-vicuna/ Vicuna: An open-source chatbot impressing gpt-4 with 90\

    2023

  2. [10]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . ...

    2021 arXiv

  3. [11]

    Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. 2024. https://openreview.net/forum?id=w7LU2s14kE Linearity of relation decoding in transformer language models . In The Twelfth International Conference...

    2024

  4. [12]

    Kaixuan Huang, Xudong Guo, and Mengdi Wang. 2024. https://arxiv.org/abs/2405.19715 Specdec++: Boosting speculative decoding via adaptive candidate lengths . Preprint, arXiv:2405.19715

    2024 arXiv

  5. [13]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav...

    2019 doi

  6. [14]

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. https://proceedings.mlr.press/v202/leviathan23a.html Fast inference from transformers via speculative decoding . In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine...

    2023

  7. [15]

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024 a . https://arxiv.org/abs/2406.16858 Eagle-2: Faster inference of language models with dynamic draft trees . Preprint, arXiv:2406.16858

    2024 arXiv

  8. [16]

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024 b . https://proceedings.mlr.press/v235/li24bt.html EAGLE : Speculative sampling requires rethinking feature uncertainty . In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedi...

    2024

  9. [17]

    Fangcheng Liu, Yehui Tang, Zhenhua Liu, Yunsheng Ni, Kai Han, and Yunhe Wang. 2024 a . https://arxiv.org/abs/2404.18911 Kangaroo: Lossless self-speculative decoding via double early exiting . Preprint, arXiv:2404.18911

    2024 arXiv

  10. [18]

    Tianyu Liu, Yun Li, Qitan Lv, Kai Liu, Jianchen Zhu, and Winston Hu. 2024 b . https://arxiv.org/abs/2408.11850 Parallel speculative decoding with adaptive draft length . Preprint, arXiv:2408.11850

    2024 arXiv

  11. [19]

    Jonathan Mamou, Oren Pereg, Daniel Korat, Moshe Berchansky, Nadav Timor, Moshe Wasserblat, and Roy Schwartz. 2024. https://arxiv.org/abs/2405.04304 Dynamic speculation lookahead accelerates speculative decoding of large language models . Preprint, arXiv:2405.04304

    2024 arXiv

  12. [20]

    Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, C a g lar Gul c ehre, and Bing Xiang. 2016. https://doi.org/10.18653/v1/K16-1028 Abstractive text summarization using sequence-to-sequence RNN s and beyond . In Proceedings of the 20th SIGNLL Conference on Computational Natural ...

    2016 doi

  13. [21]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2024

  14. [22]

    Koyena Pal, Jiuding Sun, Andrew Yuan, Byron Wallace, and David Bau. 2023. https://doi.org/10.18653/v1/2023.conll-1.37 Future lens: Anticipating subsequent tokens from a single hidden state . In Proceedings of the 27th Conference on Computational Natural Language Learning (CoNL...

    2023 doi

  15. [23]

    Andrea Santilli, Silvio Severino, Emilian Postolache, Valentino Maiorca, Michele Mancusi, Riccardo Marin, and Emanuele Rodola. 2023. https://doi.org/10.18653/v1/2023.acl-long.689 Accelerating transformer inference for translation via parallel decoding . In Proceedings of the 6...

    2023 doi

  16. [24]

    Benjamin Spector and Chris Re. 2023. https://arxiv.org/abs/2308.04623 Accelerating llm inference with staged speculative decoding . Preprint, arXiv:2308.04623

    2023 arXiv

  17. [25]

    Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. 2018. https://proceedings.neurips.cc/paper/2018/hash/c4127b9194fe8562c64dc0f5bf2c93bc-Abstract.html Blockwise parallel decoding for deep autoregressive models . In Advances in Neural Information Processing Systems 31: Annual C...

    2018

  18. [26]

    Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix X. Yu. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/6034a661584af6c28fd97a6f23e56c0a-Abstract-Conference.html Spectr: Fast speculative decoding via optimal transport . In Advanc...

    2023

  19. [27]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2024. https://crfm.stanford.edu/2023/03/13/alpaca.html Stanford CRFM

    2024

  20. [28]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023 a . https://arxiv.org/abs/2302.13971 Lla...

    2023 arXiv

  21. [29]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  22. [30]

    Jikai Wang, Yi Su, Juntao Li, Qingrong Xia, Zi Ye, Xinyu Duan, Zhefeng Wang, and Min Zhang. 2024. https://arxiv.org/abs/2406.17276 Opt-tree: Speculative decoding with adaptive draft tree structure . Preprint, arXiv:2406.17276

    2024 arXiv

  23. [31]

    Morris, and Lionel Levine

    Wilson Wu, John X. Morris, and Lionel Levine. 2024. https://arxiv.org/abs/2404.00859 Do language models plan ahead for future tokens? Preprint, arXiv:2404.00859

    2024 arXiv

  24. [32]

    Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.257 Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation . In Findings of the Association for Computational Ling...

    2023 doi

  25. [33]

    Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. 2024. https://doi.org/10.18653/v1/2024.findings-acl.456 Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding . In Findin...

    2024 doi

  26. [34]

    Seongjun Yang, Gibbeum Lee, Jaewoong Cho, Dimitris Papailiopoulos, and Kangwook Lee. 2023. https://openreview.net/forum?id=xK9FnwDMZp Predictive pipelined decoding: A compute-latency trade-off for exact LLM decoding . In Workshop on Efficient Systems for Foundation Models @ ICML2023

    2023

  27. [35]

    Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. 2024. https://doi.org/10.18653/v1/2024.acl-long.607 Draft & verify: Lossless large language model acceleration via self-speculative decoding . In Proceedings of the 62nd Annual Meeting of the As...

    2024 doi

  28. [36]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2024. Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the 37th Internat...

    2024

  29. [37]

    Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, Shengen Yan, Guohao Dai, Xiao-Ping Zhang, Yuhan Dong, and Yu Wang. 2024. https://arxiv.org/abs/2404.14294 A survey on efficient inference for large language ...

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.