Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Think Before You Accept: Semantic Reflective Verification for Faster Speculative Decoding

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Reflective Verification claims that a training-free semantic check, obtained by prompting the target model to reflect on a copy of the draft in the same forward pass, raises draft acceptance length and adds 5–15% end-to-end decoding speed…

desk verdict Clever logit-fusion trick for speculative decoding that probably works as a speedup, but the semantic-correctness story is under-evidenced and needs tighter empirical work. read the letter →

arxiv 2505.18629 v1 pith:DMAYHV3J submitted 2025-05-24 cs.LG

Yixuan Wang , Yijun Liu , Shiyu ji , Yuzhuang Xu , Yang Xu , Qingfu Zhu , Wanxiang Che This is my paper
classification cs.LG
keywords speculativedecodingself-reflectionlogitfusionsemanticverificationinferenceaccelerationtraining-freeacceptancelengthreflectiveprompting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reflective Verification tries to establish that speculative decoding can be accelerated by checking semantic correctness rather than only distributional agreement between draft and target models. The method is training-free: during verification, the target model processes the original draft, a reflection prompt, a position marker, and a second copy of the draft in one forward pass, yielding both an ordinary prediction and a reflective prediction over the same tokens. The two sets of logits are blended as $\mathrm{Prob}_{\mathrm{mix}}[i] = \mathrm{Softmax}((1-\alpha)\,\mathrm{Logits}[i] + \alpha\,\mathrm{Logits}[i+\mathrm{shift\_len}])$ with $\alpha=0.3$, and the blended distribution is handed to any existing statistical verifier. The paper reports that this raises the number of accepted draft tokens by roughly one per forward pass and adds 5–15% end-to-end decoding speed on top of speculative sampling and typical sampling, without consistent task degradation on MT-Bench, GSM8K, and HumanEval. The significance is that semantic-level verification would be available for free from the target model's own reflective abilities.

What carries the argument

The load-bearing object is the reflective draft template $\mathrm{Draft}_{\mathrm{final}} = \mathrm{Concat}(\mathrm{Draft}_{\mathrm{ori}} \parallel \mathrm{Prompt}_{\mathrm{reflection}} \parallel \mathrm{Prefix}_{\mathrm{position}} \parallel \mathrm{Draft}_{\mathrm{ori}})$ together with the logit-fusion identity $\mathrm{Prob}_{\mathrm{mix}}[i] = \mathrm{Softmax}((1-\alpha)\,\mathrm{Logits}[i] + \alpha\,\mathrm{Logits}[i+\mathrm{shift\_len}])$. Unidirectional attention ensures the first copy's logits are unaffected by the later prompt and duplicate, while the second copy, reading the reflection prompt plus the draft, produces logits that encode semantic judgement. The weighted sum balances consistency with the original target distribution against correctness as judged by reflection. This pair of constructions extracts a semantic verification signal in a single forward pass, which is what makes the method training-free and compatible with existing verifiers.

What would settle it

Replace the reflection prompt in Equation 1 with a semantically empty fixed string such as 'abcdef' while keeping every other setting identical; if the accepted-token length and downstream task scores remain at the elevated level, the semantic content of the prompt is not load-bearing, and if they collapse to baseline, the reflective signal is what drives the result.

Watch

Extended reading notes

Core claim

The central discovery is that a large language model's self-reflection, triggered by a prompt probe, can serve as a semantic correctness signal during speculative decoding. When the draft tokens are duplicated after a reflection prompt, the target model's logits over the second copy encode an evaluation of the draft: they give higher probability to continuations that are semantically correct even when those continuations differ from the draft's own distribution. Fusing the original and reflective logits produces a verification distribution that accepts semantically valid drafts that exact-match or statistical verification would reject. Because the fusion only changes the target distribution before the verification step, it composes with statistical verification schemes such as speculative sampling and typical sampling. The experiments claim acceptance-length gains near one token and 5–15% throughput improvement while maintaining or slightly improving task performance.

Load-bearing premise

The method assumes that the logits produced over the second copy of the draft, after the reflection prompt, genuinely encode semantic correctness rather than a spurious distributional shift, and that a fixed blend weight $\alpha=0.3$ keeps the fused distribution close enough to the true target distribution.

Editorial extensions

If this is right

  • Existing statistical verifiers can be upgraded without retraining by inserting the reflective template and the logit fusion, so the reported speedup composes with speculative sampling and typical sampling.
  • The acceptance-length ceiling of speculative decoding is no longer set only by draft-target distributional agreement, because semantically correct but distributionally different drafts can be accepted.
  • Gains should grow with target-model scale, since stronger reflection abilities produce more informative reflective logits, a trend the paper observes when the target grows from 8B to 70B parameters.
  • Lossy verification methods suffer less task degradation when reflective fusion is added, because the semantic signal helps reject incorrect drafts that distributional statistics would accept.
  • The extra cost is limited to one forward pass over a duplicated draft plus a short prompt, with KV-cache entries beyond the first copy pruned after each step.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper fixes $\alpha=0.3$ globally; an adaptive per-position blend weight, based on agreement or entropy between original and reflective logits, could improve the trade-off further.
  • The reflective logits could serve as a training-free quality score for selecting among multiple draft paths in tree-based speculative decoding, not just for accepting a single chain.
  • A direct test of the semantic-load-bearing assumption is to replace the reflection prompt with a random token sequence; if acceptance length and task quality stay elevated, the gains are not caused by semantic reflection.
  • The method would be strengthened by a calibration check: does the fused distribution assign higher probability to human-verified correct continuations than the original distribution does, especially on open-domain text outside the three benchmarks?
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Reflective Verification, a training-free modification to the verification stage of speculative decoding. The method concatenates the draft tokens, a hand-written reflection prompt, the current prefix, and a second copy of the draft tokens, then obtains the target model's logits at both the original and the reflected draft positions in one forward pass. These logits are fused as Prob_mix[i] = Softmax((1-alpha)*Logits[i] + alpha*Logits[i+shift_len]) with alpha=0.3 (Eq. 2), and the fused distribution is plugged into existing verification schemes such as speculative sampling, exact-match speculative decoding, and typical sampling. Experiments on MT-Bench, GSM8K, and HumanEval with Llama3 1B/8B and 8B/70B report increased mean accepted tokens (#MAT) and 5-15% end-to-end throughput improvements over the corresponding baselines without, the authors claim, significant task performance degradation.

Significance. If the mechanism works as described, this is a useful and simple contribution: it is training-free, requires no auxiliary models, and is presented as orthogonal to common statistical verification methods. The paper is generally clearly written, gives explicit pseudocode, and tests multiple draft-model scales and verification strategies; the speedup and #MAT gains are positive in every main configuration in Table 1. The main empirical risks are that the semantic-correction mechanism is not isolated from generic distributional effects, alpha is tuned on the evaluation benchmarks, and no uncertainty estimates are reported. The conceptual claim that the method is merely an auxiliary 'verification' layer is also weakened by the fact that the fused distribution replaces the target distribution. These issues are addressable within the manuscript's scope, so the central idea should be given the opportunity to be strengthened.

major comments (4)
  1. [Section 3.2, Eq. (2); Section 3.3, Algorithm 1] The fused distribution Prob_mix is not the target model's distribution, so plugging it into speculative sampling (Algorithm 1) and exact-match verification changes the output distribution of the overall decoder. In particular, the unbiasedness guarantee of speculative sampling no longer holds, and 'Spec Decoding + Reflect Verify' is not lossless in the standard sense. The text in Section 3.2 that the method 'merely produces an output distribution with a higher acceptance rate' and is 'fully orthogonal' to statistical verification understates this: it is not an auxiliary verification rule on top of an unchanged target distribution, but a modification of the target-side proposal distribution. The paper should explicitly frame the method as a lossy accelerator and quantify distributional shift (for example, KL divergence between the fused and original target distributions, or task metrics with confidence intervals).
  2. [Section 5.1 and Table 1] The hyperparameter alpha=0.3 is selected on the same benchmarks that produce the main results, and no error bars or significance tests are reported even though the checklist states that results are averaged over three runs. Several individual cells in Table 1 show task degradation (e.g., MT-Bench 7.37 vs. 7.44 under Spec Decoding in the 1B&8B block; GSM8K 84.91 vs. 85.52 under Typical Sampling in the 8B&70B block). The claim 'without significant task performance degradation' is therefore not established. Report per-cell standard deviations or confidence intervals, and ideally select alpha on a validation split rather than on the test benchmarks.
  3. [Section 5.3, Table 2] The only quasi-control in the paper is the last row of Table 2, where an empty reflective prompt is used: duplication alone raises #MAT from 6.39 to 7.45 while lowering Pass@1 from 65.85 to 64.63. This leaves open the possibility that part or all of the acceleration with full prompts comes from a generic distributional shift caused by duplicating draft tokens and re-feeding the prefix, rather than from semantic reflection. Add matched-length neutral controls (for example, an irrelevant filler prompt or a non-reflective instruction of comparable length) and report variance across runs; without such controls, the mechanistic claim that reflective logits 'encode semantic correctness' is not directly supported.
  4. [Section 5.4, Table 3] The comparison with tree-based verification is not on equal footing: the chain and MCSD configurations use budgets of 5 and 60, while 'Ours' uses a budget of 174.92. Comparing #MAT under substantially different input budgets does not support a claim of superiority per unit of compute or latency. In addition, the notation 'Ours {5+3+4+5}' and the origin of 174.92 are not explained. Report a matched-budget comparison or normalized throughput, or remove the table from the main text.
minor comments (4)
  1. [Section 3.1, Eq. (1)] There is a typo in Eq. (1): 'Promptrefection' should be 'Promptreflection'; the same identifier is also written inconsistently elsewhere in the text.
  2. [Algorithms 1-3] The pseudocode contains broken identifiers such as 'ref lective_draf t' and notation suggesting multiple forward passes ('o1(x),...,o_{gamma+1}(x), om(x),...,om+gamma(x) <- Mp(prefix), ..., Mp(prefix+reflective_draft)'). Clarify that a single forward pass over the concatenated sequence produces the logits at all positions and that the line is a notational shorthand.
  3. [Section 3.1] The statement that the additional reflective tokens 'do not significantly increase forward latency' is asserted without profiling data. Since the reflective sequence roughly doubles the number of verification-time tokens, a latency or FLOPs breakdown would strengthen the throughput claim.
  4. [Section 5.4] The configuration notation 'Ours {5+3+4+5}' should be defined in the text; the relationship between these four component token counts and the reported budget 174.92 is unclear to the reader.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's central claim is an empirical speedup measurement, and the reflective-logit construction is an inference-time intervention rather than a quantity forced by its own definitions or by load-bearing self-citation.

full rationale

The derivation chain in this paper is empirical rather than formal. Equation (1) constructs a prompt that duplicates the draft with a reflective probe, and Equation (2) fuses the resulting logits with a tuned weight alpha. The claim that the reflective logits 'encode semantic correctness' is an empirical premise, not a definitional equivalence: no equation defines correctness as the reflective output, and no fitted quantity is renamed as a prediction. The headline speedups and acceptance lengths are measured end-to-end on external benchmarks, and alpha=0.3 is a hyperparameter chosen from an ablation, not a constant whose value directly determines the reported throughput. The paper does cite prior work by overlapping authors, but only as background related work for drafting and verification methods; the mechanism of reflective verification is not justified by those citations, and no uniqueness theorem or forcing argument is imported from the authors' own prior work. The absence of a matched prompt-length control is a validity concern about whether the reflection prompt specifically drives the effect, but that is an experimental-control issue, not a circularity of the kind where the result reduces by construction to its input. Accordingly, no circular step is identified.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central speed claim depends on alpha, draft length, temperature, and prefix length, all hand-selected per benchmark. The reflective signal is an assumption about the target model's own behavior, and the 'semantic correctness' of that signal is not independently verified. No new physical entities are introduced; reflective logits are just a second set of logits from the same model.

free parameters (4)
  • alpha = 0.3
    Fusion weight in Eq. 2; selected from the ablation in Section 5.1 and fixed across all experiments (Appendix B). There is no theoretical basis for the value.
  • draft length K = 5/8/8 for 1B&8B; 8/10/10 for 8B&70B
    Chosen per benchmark as 'optimal' in Appendix B; acceptance rate and measured speedup depend on it.
  • temperature = 0.8 for MT-Bench; 0.2 for GSM8K and HumanEval
    Set per dataset in Appendix B; affects draft generation and task performance.
  • prefix length = 4
    Length of Prefixposition in Eq. 1; set in Appendix B and used to locate the regeneration position.
assumptions (6)
  • domain assumption Unidirectional attention in decoder-only transformers allows the second copy of draft tokens to condition only on the prefix plus reflection prompt, yielding reflective logits for the same positions.
    Invoked in Section 3.1 around Eq. 1; no empirical verification across model families.
  • domain assumption Appending a reflection prompt (e.g., 'Oh! I made a mistake! The correct answer is:') makes the target LLM's output distribution encode semantic correctness.
    Core premise of the method, motivated by examples in Section 2.2 and Figure 1; not formalized or validated against an external semantic metric.
  • domain assumption With alpha=0.3, the fused distribution Softmax((1-alpha)Logits[i] + alpha Logits[i+shift_len]) is close enough to the true target distribution that downstream task performance is preserved.
    Underlies the 'no degradation' claim; only checked empirically on three benchmarks in Table 1.
  • standard math Rejection sampling remains valid when the fused distribution p_mix is treated as the target distribution.
    Algorithm 1 uses the standard r_i <= p_i/q_i test; mathematically valid for any p, but the resulting samples are from p_mix, not from the original target model.
  • domain assumption Draft and target models share a tokenizer and vocabulary, so draft tokens can be concatenated into the target prompt.
    Required by Eq. 1; holds for the Llama-family pairs used but not guaranteed in general.
  • domain assumption KV-cache entries for the reflective segment can be pruned after each forward pass without changing future decoding.
    Stated in Section 3.1; an implementation assumption about inference engines.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Think Before You Accept: Semantic Reflective Verification for Faster Speculative Decoding." pith.science (2026). https://pith.science/paper/DMAYHV3J

@misc{pith2026250518629,
  author       = {Pith},
  title        = {Pith review of: Think Before You Accept: Semantic Reflective Verification for Faster Speculative Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DMAYHV3J}},
  note         = {Machine review of arXiv:2505.18629}
}
abstract

Large language models (LLMs) suffer from high inference latency due to the auto-regressive decoding process. Speculative decoding accelerates inference by generating multiple draft tokens using a lightweight model and verifying them in parallel. However, existing verification methods rely heavily on distributional consistency while overlooking semantic correctness, thereby limiting the potential speedup of speculative decoding. While some methods employ additional models for relaxed verification of draft tokens, they often fail to generalize effectively to more diverse or open-domain settings. In this work, we propose Reflective Verification, a training-free and semantics-aware approach that achieves a better trade-off between correctness and efficiency. Specifically, we leverage the inherent reflective capacity of LLMs to semantically assess the correctness of draft tokens in parallel during verification. Using prompt-based probing, we obtain both the original and reflective distributions of draft tokens in a single forward pass. The fusion of these distributions enables semantic-level verification of draft tokens that incorporates both consistency and correctness. Experiments across multiple domain benchmarks and model scales demonstrate that our method significantly increases the acceptance length of draft tokens without compromising model performance. Furthermore, we find that the proposed Reflective Verification is orthogonal to existing statistical verification methods, and their combination yields additional 5$\sim$15\% improvements in decoding speed.

Figures

Figures reproduced from arXiv: 2505.18629 by the authors.

Figure 1
Figure 1. An illustration of draft tokens rejected by standard speculative decoding. Self-reflection [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overall structural diagram of Reflective Verification. Compared to vanilla speculative [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Effect of α on task and acceleration performance. 1 2 3 4 5 6 7 8 9 10 Draft Token Length 2 3 4 5 6 7 #MAT of 8B of 1B The Effect of Draft Token Length on #MAT w. Reflect Verify(8B) w.o. Reflect Verify(8B) w. Reflect Verify(1B) w.o. Reflect Verify(1B) [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: An illustration of reflective verification on MT-Bench. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: An illustration of reflective verification on GSM8K. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: An illustration of reflective verification on HumanEval. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Parallel Decoder Transformer: Planner-Conditioned Latent Coordination for Model-Intrinsic Parallel Generation

    cs.AI 2025-12 reject novelty 5.0 of 10

    A Parallel Decoder Transformer architecture for multi-lane parallel text generation is proposed, but the paper's own evidence is incomplete and internally contradictory.

Reference graph

Works this paper leans on

35 extracted references · 12 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. ArXiv preprint, abs/2303.08774, 2023. URL https://arxiv.org/abs/2303.08774

    arXiv 2023

  2. [2]

    Judge decoding: Faster speculative sampling requires going beyond model alignment

    Gregor Bachmann, Sotiris Anagnostidis, Albert Pumarola, Markos Georgopoulos, Artsiom Sanakoyeu, Yuming Du, Edgar Sch \"o nfeld, Ali Thabet, and Jonas Kohler. Judge decoding: Faster speculative sampling requires going beyond model alignment. ArXiv preprint, abs/2501.19309, 2025. URL https://arxiv.org/abs/2501.19309

    arXiv 2025

  3. [3]

    Fast and robust early-exiting framework for autoregressive language models with synchronized parallel decoding

    Sangmin Bae, Jongwoo Ko, Hwanjun Song, and Se-Young Yun. Fast and robust early-exiting framework for autoregressive language models with synchronized parallel decoding. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5910--5924, Singapore, 2023. Associatio...

    doi:10.18653/v1/2023.emnlp-main.362 2023

  4. [4]

    Lee, Deming Chen, and Tri Dao

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024. URL https://openreview.net/forum?id=PEpbUobfJv

    work page 2024

  5. [5]

    Accelerating large language model decoding with speculative sampling

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. ArXiv preprint, abs/2302.01318, 2023. URL https://arxiv.org/abs/2302.01318

    arXiv 2023

  6. [6]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. ArXiv preprint, abs/2107.03374, 2021. URL https://arxiv.org/abs/2107.03374

    arXiv 2021

  7. [7]

    Towards reasoning era: A survey of long chain-of-thought for reasoning large language models

    Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. Towards reasoning era: A survey of long chain-of-thought for reasoning large language models. ArXiv preprint, abs/2503.09567, 2025. URL https://arxiv.org/abs/2503.09567

    arXiv 2025

  8. [8]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. ArXiv preprint, abs/2110.14168, 2021. URL https://arxiv.org/abs/2110.14168

    arXiv

Show all 35 references
  1. [9]

    Break the sequential dependency of LLM inference using lookahead decoding

    Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of LLM inference using lookahead decoding. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024. URL https://openrevi...

    2024

  2. [10]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. ArXiv preprint, abs/2407.21783, 2024. URL https://arxiv.org/abs/2407.21783

    2024 arXiv

  3. [11]

    REST : Retrieval-based speculative decoding

    Zhenyu He, Zexuan Zhong, Tianle Cai, Jason Lee, and Di He. REST : Retrieval-based speculative decoding. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: H...

    2024

  4. [12]

    Mahoney, Amir Gholami, and Kurt Keutzer

    Sehoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W. Mahoney, Amir Gholami, and Kurt Keutzer. Speculative decoding with big little decoder. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neur...

    2023

  5. [13]

    Cllms: Consistency large language models

    Siqi Kou, Lanxiang Hu, Zhezhi He, Zhijie Deng, and Hao Zhang. Cllms: Consistency large language models. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024. URL https://openreview.net/forum?id=8uzBOVmh8H

    2024

  6. [14]

    Fast inference from transformers via speculative decoding

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 20...

    2023

  7. [15]

    EAGLE: speculative sampling requires rethinking feature uncertainty

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE: speculative sampling requires rethinking feature uncertainty. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024. URL https://openreview....

    2024

  8. [16]

    Reward-guided speculative decoding for efficient llm reasoning

    Baohao Liao, Yuhui Xu, Hanze Dong, Junnan Li, Christof Monz, Silvio Savarese, Doyen Sahoo, and Caiming Xiong. Reward-guided speculative decoding for efficient llm reasoning. ArXiv preprint, abs/2501.19324, 2025. URL https://arxiv.org/abs/2501.19324

    2025 arXiv

  9. [17]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. ArXiv preprint, abs/2412.19437, 2024 a . URL https://arxiv.org/abs/2412.19437

    2024 arXiv

  10. [18]

    Online speculative decoding

    Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Alvin Cheung, Zhijie Deng, Ion Stoica, and Hao Zhang. Online speculative decoding. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024 b . URL https://openrevi...

    2024

  11. [19]

    Turning trash into treasure: Accelerating inference of large language models with token recycling

    Xianzhen Luo, Yixuan Wang, Qingfu Zhu, Zhiming Zhang, Xuanyu Zhang, Qing Yang, Dongliang Xu, and Wanxiang Che. Turning trash into treasure: Accelerating inference of large language models with token recycling. ArXiv preprint, abs/2408.08696, 2024. URL https://arxiv.org/abs/2408.08696

    2024 arXiv

  12. [20]

    Self-refine: Iterative refinement with self-feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterativ...

    2023

  13. [21]

    Specinfer: Accelerating generative llm serving with speculative inference and token tree verification

    Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Rae Ying Yee Wong, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. Specinfer: Accelerating generative llm serving with speculative inference and token tree verification. ArXiv preprint, abs/23...

    2023 arXiv

  14. [22]

    Jie Ou, Yueming Chen, and Prof. Tian. Lossless acceleration of large language model via adaptive n-gram parallel decoding. In Yi Yang, Aida Davani, Avi Sil, and Anoop Kumar, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Comput...

    2024

  15. [23]

    Optimized multi-token joint decoding with auxiliary model for llm inference

    Zongyue Qin, Ziniu Hu, Zifan He, Neha Prakriya, Jason Cong, and Yizhou Sun. Optimized multi-token joint decoding with auxiliary model for llm inference. ArXiv preprint, abs/2407.09722, 2024. URL https://arxiv.org/abs/2407.09722

    2024 arXiv

  16. [24]

    Accelerating transformer inference for translation via parallel decoding

    Andrea Santilli, Silvio Severino, Emilian Postolache, Valentino Maiorca, Michele Mancusi, Riccardo Marin, and Emanuele Rodola. Accelerating transformer inference for translation via parallel decoding. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings...

    2023 doi

  17. [25]

    Blockwise parallel decoding for deep autoregressive models

    Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. Blockwise parallel decoding for deep autoregressive models. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicol \` o Cesa - Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing S...

    2018

  18. [26]

    Make some noise: Unlocking language model parallel inference capability through noisy training

    Yixuan Wang, Xianzhen Luo, Fuxuan Wei, Yijun Liu, Qingfu Zhu, Xuanyu Zhang, Qing Yang, Dongliang Xu, and Wanxiang Che. Make some noise: Unlocking language model parallel inference capability through noisy training. ArXiv preprint, abs/2406.17404, 2024. URL https://arxiv.org/ab...

    2024 arXiv

  19. [27]

    Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation

    Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics...

    2023 doi

  20. [28]

    Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding

    Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. ArXiv preprint, abs/2401.07851, 2024. URL https://arxiv.org/abs/2401.07851

    2024 arXiv

  21. [29]

    Densing law of llms

    Chaojun Xiao, Jie Cai, Weilin Zhao, Guoyang Zeng, Biyuan Lin, Jie Zhou, Zhi Zheng, Xu Han, Zhiyuan Liu, and Maosong Sun. Densing law of llms. ArXiv preprint, abs/2412.04315, 2024. URL https://arxiv.org/abs/2412.04315

    2024 arXiv

  22. [30]

    Inference with reference: Lossless acceleration of large language models

    Nan Yang, Tao Ge, Liang Wang, Binxing Jiao, Daxin Jiang, Linjun Yang, Rangan Majumder, and Furu Wei. Inference with reference: Lossless acceleration of large language models. ArXiv preprint, abs/2304.04487, 2023. URL https://arxiv.org/abs/2304.04487

    2023 arXiv

  23. [31]

    Multi-candidate speculative decoding

    Sen Yang, Shujian Huang, Xinyu Dai, and Jiajun Chen. Multi-candidate speculative decoding. ArXiv preprint, abs/2401.06706, 2024. URL https://arxiv.org/abs/2401.06706

    2024 arXiv

  24. [32]

    Physics of language models: Part 2.2, how to learn from mistakes on grade-school math problems

    Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of language models: Part 2.2, how to learn from mistakes on grade-school math problems. ArXiv preprint, abs/2408.16293, 2024. URL https://arxiv.org/abs/2408.16293

    2024 arXiv

  25. [33]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei - Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In Alice Oh, Tristan Naumann, Amir Glob...

    2023

  26. [34]

    Distillspec: Improving speculative decoding via knowledge distillation

    Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadeh, Sanjiv Kumar, Jean - Fran c ois Kagy, and Rishabh Agarwal. Distillspec: Improving speculative decoding via knowledge distillation. In The Twelfth International Conference on Learning Repr...

    2024

  27. [35]

    A survey on efficient inference for large language models

    Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, et al. A survey on efficient inference for large language models. ArXiv preprint, abs/2404.14294, 2024 b . URL https://arxiv.org/abs/2404.14294

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.