Pith. sign in

REVIEW 5 major objections 5 minor 14 references

Out-of-Vocabulary Sampling Boosts Speculative Decoding

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a pruned drafter can regain lost acceptance rates by redistributing its probability mass through token-affinity priors, and that a first-order Taylor approximation achieves the same effect in linear time.

desk verdict Novel idea with plausible algorithm and clear prose, but the main theorems do not hold as stated; the linear-time approximation is unproven in exactly the regime the paper targets. read the letter →

arxiv 2506.03206 v1 pith:HZCBW22J submitted 2025-06-02 cs.LG cs.CL

Nadav Timor , Jonathan Mamou , Oren Pereg , Hongyang Zhang , David Harel This is my paper
classification cs.LGcs.CL
keywords speculativedecodingout-of-vocabularysamplingvocabularypruningacceptanceratedrafterkerneltoken-affinitypriorslinear-timeTaylorapproximationlargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Speculative decoding runs faster when the draft model has a small vocabulary, but a small-vocabulary drafter cannot propose tokens outside its vocabulary, and that limitation pushes acceptance rates down. This paper introduces Redistributing Drafter Kernels (RDK), an out-of-vocabulary sampler that first applies Token-Level Intersection and then reallocates the drafter's excess probability mass toward target tokens using a row-stochastic token-affinity matrix. Because acceptance rate equals one minus the $\ell^1$ norm of the drafter kernel $k(x)=\max(0,q(x)-p(x))$, moving mass away from over-assigned tokens can only help. The paper proves worst-case bounds showing RDK is never worse than TLI, proves a two-token model where RDK beats TLI with high probability, and derives a first-order Taylor approximation that reduces redistribution cost from $O(N^2)$ to $O(N)$. In synthetic and off-the-shelf experiments, linear-time RDK keeps acceptance nearly constant even after removing more than 75% of the drafter's vocabulary, while the baselines collapse.

What carries the argument

The central object is the drafter kernel $k(x)=\max(0,q(x)-p(x))$, whose $\ell^1$ norm is tied to acceptance rate by $\alpha=1-\|k\|_1$, so shrinking the kernel is the same as raising acceptance. The redistribution is carried by a row-stochastic token-affinity matrix $M_{ij}=\exp(\Omega_{ij}/\tau)/\sum_k \exp(\Omega_{ik}/\tau)$, built from the target's covariance matrix and applied as $p'=M^\top q'$; its non-expansiveness in $\ell^1$ is what makes the worst-case bound go through. The efficiency mechanism is the Taylor expansion of $p'$ around small $\theta$, yielding $\tilde p_i=(Nq'_i+\theta p_i)/(N+p_i)$ followed by normalization, which turns the quadratic matrix application into a linear-time logit adjustment. Together, the kernel identity, the affinity matrix, and the Taylor approximation convert vocabulary pruning from a lossy shortcut into a distributional repair problem.

What would settle it

Measure the $\ell^1$ gap between the exact $M^\top q'$ update and the Taylor approximator on a real model with a 128k vocabulary pruned to its 500 most frequent tokens; if the gap scales like $N\theta$ rather than $\theta+1/N^2$ across contexts, the theorem's uniformity assumption fails in exactly the extreme-pruning regime the paper advertises.

Watch

Extended reading notes

Core claim

The central discovery is that out-of-vocabulary sampling through a token-affinity prior can restore most of the acceptance-rate loss caused by vocabulary pruning. The authors define acceptance rate as $\alpha=\sum_x \min\{p(x),q(x)\}$ and show $\alpha=1-\|k\|_1$, where $k$ is the drafter kernel of excess mass; this makes mass redistribution a principled route to higher acceptance. RDK first normalizes the drafter onto the vocabulary intersection (TLI) and then applies $p' = M^\top q'$, where $M$ is a softmax-normalized co-occurrence matrix, so the drafter can propose tokens it never had in its vocabulary. The paper proves that this redistribution error is bounded by the TLI error plus a term controlled by how well $M$ preserves the target, and in a two-token Gaussian setting that RDK beats TLI with probability at least 99.46%. The key computational result is that the full $N\times N$ update is approximated to first order in $\theta=p^\top q'$ by a closed-form linear-time formula, so no matrix needs to be formed or stored; experiments show this approximation maintains an almost constant acceptance rate of 26.7% after pruning a 200k vocabulary down to 500 tokens.

Load-bearing premise

The proof that the linear-time formula matches the full RDK matrix assumes the pruned drafter's renormalized distribution is nearly uniform over the whole target vocabulary and that token affinities are dominated by a single component aligned with the target, both of which are violated when pruning leaves only a tiny intersection of tokens.

Editorial extensions

If this is right

  • Drafter vocabularies can be cut to a few thousand tokens or even hundreds without paying the full acceptance-rate penalty, which directly lowers the drafter's forward-pass cost for large-vocabulary language models.
  • The linear-time approximation removes the need to store or multiply an $N\times N$ co-occurrence matrix, so RDK can run as a small logit-space adjustment at inference time.
  • Because RDK is training-free and does not require matching tokenizers, it can pair a target model with any reasonably fast drafter, including one from a different model family.
  • The acceptance-rate improvement transfers to end-to-end speculative-decoding speedup, since the token-overlap identity in Lemma 2.3 links acceptance rate directly to the $\ell^1$ error between drafter and target distributions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the authors leave implicit is that the linear-time approximation's proof carries a hidden near-uniformity assumption: the post-TLI drafter distribution is treated as spread evenly over the whole target vocabulary, which is exactly what extreme pruning destroys.
  • If the approximation degrades gracefully in practice, the same redistribution idea could be composed with other reject-sampling accelerators, such as tree-based or block-verification drafters, without retraining, because it only reshapes the drafter's logits.
  • The practical ceiling of RDK is set by how well a static, data-estimated co-occurrence prior captures dynamic, context-dependent token affinities; a context-dependent prior is a natural next test, since the current experiments use a fixed affinity matrix derived from aggregate statistics.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes Redistributing Drafter Kernels (RDK), a sampling method intended to improve speculative decoding when the drafter has a pruned or smaller vocabulary than the target. RDK first applies token-level intersection (TLI) and then multiplies the resulting distribution by a row-stochastic token-affinity matrix M, with the goal of recovering probability mass for out-of-vocabulary tokens. The authors claim a first-order Taylor approximation of the matrix-vector product that runs in O(N) time, prove theorems about acceptance-rate improvements and approximation accuracy, and report synthetic and off-the-shelf experiments showing RDK maintains acceptance under extreme pruning.

Significance. The core idea—redistributing the drafter's probability mass using token-affinity priors to compensate for vocabulary pruning—is timely and could be practically useful if the theoretical and empirical claims held. The paper correctly identifies that drafter kernel mass is redundant from the acceptance-rate perspective (Lemma 2.3), and the O(N) approximation is an appealing design goal. However, the central technical results are not established. Theorem 5.4's proof relies on a false uniformity claim and on an unstated rank-1 covariance assumption, Theorem 5.3 applies only to a two-token special case while misusing M as both a redistribution matrix and a covariance matrix, and the main synthetic experiments construct the affinity prior from the exact target distribution being decoded. As a result, the claimed acceptance-rate recovery and the linear-time approximation are unsupported in the regime that motivates the paper. The contribution is interesting but would require substantial revision to be credible.

major comments (5)
  1. [Appendix B.2 / Theorem 5.4] The proof asserts 'Since Σ_i q'_i = 1 and θ = p^T q', one shows N q'_j = 1 + O(θ).' This step is false for arbitrary q': θ is a weighted average of p under q' and carries no per-coordinate information about q'_j. Under the paper's own extreme-pruning regime, q' is concentrated on the small intersection I, so N q'_j is either 0 (for j outside I) or approximately N/|I| (for j inside I), not 1 + O(θ). Consequently the O(θ + 1/N^2) bound and the claimed linear-time Taylor approximation are unproven exactly in the setting where RDK is supposed to provide a benefit.
  2. [Appendix B.2, Eqs. (5)-(6)] The proof of Theorem 5.4 silently introduces a rank-1 covariance alignment assumption: Ω_ij/τ ≈ θ p_j + O(θ^2) and s_i ≈ θ + O(θ^2). This assumption is not stated in Theorem 5.4's hypotheses, which instead quantify over a general co-occurrence matrix M from Eq. (2). The theorem as stated cannot be derived from its assumptions; the proof validates a different, unstated result.
  3. [Theorem 5.3 and its proof (Appendix B.4)] The theorem simultaneously treats M as the row-stochastic redistribution matrix of Algorithm 1 and as the covariance matrix of the Gaussian logits in Eq. (4). The proof states 'Since z ∼ N(µ, M)' and uses M11 and M22 as variances, but a row-stochastic matrix is not generally a covariance matrix. In addition, the theorem is restricted to m = 2 and q' = [1,0]^T, so even if a corrected proof existed it would not establish the paper's broad claim that RDK outperforms vanilla and state-of-the-art samplers for general vocabularies and drafter distributions.
  4. [Section 6.2 and Appendix C.3] The synthetic acceptance experiments construct the co-occurrence matrix M from the same t-distributed target distribution that is being decoded (Appendix C.3 explicitly describes 'the co-occurrence matrix M for our t-distributed synthetic target'). This makes the affinity prior a function of the test distribution, so the reported 26.7% acceptance rate partly reflects oracle knowledge rather than the method's general behavior. The off-the-shelf experiment in Section C.4 is a step in the right direction, but the main empirical evidence for extreme-pruning performance is circular.
  5. [Theorem 5.1 / Section 5.1] The bound ∥p' − p∥_1 ≤ ∥q0 − p∥_1 + ε is a direct consequence of the triangle inequality and does not by itself prove that RDK increases acceptance rates over TLI. The actual comparison depends on the magnitude of ε = ∥M^T p − p∥_1 relative to the improvement from redistribution; for a general row-stochastic M this quantity can be large. Thus the abstract's claim that RDK 'can achieve higher acceptance rates than vanilla and state-of-the-art samplers' is not supported by Theorem 5.1, and the only supporting theorem (5.3) is both special-case and flawed.
minor comments (5)
  1. [Section 6.2] The text says 'pruning to 0.00025% of the vocabulary (500 tokens from an originally 200k)', but 500/200,000 equals 0.25%, not 0.00025%; the percentage and the count are inconsistent.
  2. [Theorem 5.2] The statement defines p := softmax(z) and q := softmax(z+η) twice; the duplicate definition should be removed.
  3. [Equation (3)] The notation p̃_i and the subsequent definition pTaylor = p̃ / Σ p̃ is introduced abruptly; please state explicitly that p̃ is unnormalized and pTaylor is its normalization.
  4. [Theorem 5.3 proof] The proof's 'Case 2' uses row-stochasticity to assert M11 + M12 = 1, while the same proof treats M11 and M22 as variances; the two interpretations of M are inconsistent and the reader cannot tell which is intended.
  5. [Figure 7] The claim that RDK maintains 'an almost constant acceptance rate of 26.7%' is presented without error bars or the number of independent trials; please specify the experimental variability.

Circularity Check

3 steps flagged · score 7.0 of 10

The linear-time RDK used in the headline experiments is defined using the target distribution p itself, so its high acceptance rates are substantially by construction; the synthetic affinity prior is also fit to the test distribution.

  1. self definitional [§4, Eq. (3); §6.2]
    "Let θ=p^T q′. Define p̃_i = (N q′_i + θ p_i)/(N+p_i), pTaylor = p̃/Σ_j p̃_j. ... we sample synthetic target distributions from a t distribution (Fig. 5) and measure acceptance rates under three schemes: ... (c) our first-order Taylor approximations of RDK."

    The linear-time drafter pTaylor is constructed from the target probability vector p itself: θ is the inner product p^T q′, and p_i appears additively in the numerator, so pTaylor is a shrinkage of q′ toward p. The acceptance-rate experiments then evaluate pTaylor against that same p. High acceptance is therefore partly definitional: the proposal distribution is a function of the answer it is supposed to predict. In real speculative decoding, p is not known at drafting time, so this is an oracle-aided quantity rather than an independent drafter. The headline claim that linear-time RDK significantly boosts acceptance after extreme pruning rests on this by-construction overlap.

  2. fitted input called prediction [§6.2, Fig. 11 caption]
    "A 200×200 block of the co-occurrence matrix M for our t-distributed synthetic target. Most entries are near zero, indicating that token affinities are highly sparse."

    The token-affinity prior M, which is the mechanism by which RDK redistributes drafter mass, is explicitly built for the same t-distributed synthetic target used to define the test distribution and measure acceptance. Thus the empirical acceptance recovery is an in-sample fit: M encodes the covariance structure of the very distribution against which acceptance is computed. No independent test distribution is used to validate that M transfers across contexts or datasets, so the reported 26.7% acceptance at extreme pruning is at least partially forced by the construction of the experimental setup.

1 more flagged steps
  1. other [Theorem 5.3, Appendix B.4]
    "Let m=2. In Eq.(4), denote by μ=... and M=... Assume that μ1 + 3√M11 ≤ μ2 −3√M22 ... Then with probability 99.46%, under Assumption 4.1, we have αTLI ≤ αRDK. ... Since z∼N(μ,M), ... p′=Mq′=[M11, M12]^T."

    The theorem reuses the same symbol M for the row-stochastic affinity matrix of Algorithm 1 and for the covariance matrix of the target logit distribution in Eq. (4). The proof computes p′ = M q′ using the target's covariance as the redistribution matrix. The guarantee is therefore conditional on RDK being handed the target distribution's second-order statistics as its prior. This is oracle knowledge, not an independent affinity prior; it does not establish that a prior estimated from independent data improves acceptance. The conclusion is built into the assumption that the sampler has access to the target's covariance.

full rationale

The paper has a self-contained conditional framework: Theorem 5.1 is a genuine L1 bound given a row-stochastic M, and the non-expansiveness lemma is not circular. However, the central practical claim is substantially circular. The O(N) approximator pTaylor is defined in Eq. (3) using the target vector p itself (through θ=p^T q′ and the p_i term), and the acceptance experiments in §6.2 evaluate that same pTaylor against the same p; high acceptance is therefore partly engineered by construction. In addition, the synthetic co-occurrence matrix M is fit to the same t-distributed target used for evaluation, making the affinity prior an oracle for the test distribution. Theorem 5.3 similarly assumes the redistribution matrix is the target covariance, so its guarantee is conditional on knowledge of the target's distribution. Beyond these circularity issues, the proof of Theorem 5.4 in Appendix B.2 contains a load-bearing assertion that is not justified by the theorem's hypotheses: 'Since Σ_i q′_i = 1 and θ=p^T q′, one shows N q′_j = 1+O(θ).' This is false for the pruned, concentrated q′ used in the extreme-pruning regime, and the proof silently imposes Ω_ij/τ ≈ θ p_j and s_i ≈ θ. Those are correctness gaps rather than definitional circularities, but they leave the linear-time approximation unproven exactly where the paper claims it works. On balance, the headline 'linear-time RDK significantly boosts acceptance rates after extreme pruning' reduces to an oracle-aided construction, so the circularity score is 7.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The central claims rest on row-stochasticity of M, on the Gaussian logit model, and on two unstated assumptions introduced inside proofs: rank-1 covariance aligned with p, and near-uniform post-TLI draft distributions. The synthetic evaluation further uses an M estimated from the exact target being decoded, which inflates measured acceptance.

free parameters (3)
  • temperature tau in co-occurrence softmax = not specified
    Controls the sharpness of the redistribution in Eq (2). No value or selection procedure is given, and the synthetic experiments do not report tau.
  • co-occurrence matrix M (or covariance Omega) = estimated from data
    Input prior fitted to target data. In the synthetic evaluation, M is computed from the same t-distributed target being decoded, making the empirical success partly oracle-based.
  • Student t degrees of freedom in synthetic targets = 5 (MLE)
    The theoretical framework assumes Gaussian logits (Eq 4), but the experiments use t-distributed synthetic targets with df=5. The mismatch is not reconciled.
assumptions (6)
  • domain assumption M is row-stochastic (Assumption 4.1)
    Required for mass preservation in Algorithm 1. Standard for a transition kernel, but restrictive when M is also treated as a covariance matrix.
  • domain assumption Target logits follow a multivariate Gaussian (Eq 4)
    Used in Theorem 5.3. Contradicted by the paper's own Q-Q analysis showing Student t fits logits better, and by the choice of t-distributed synthetic targets in experiments.
  • ad hoc to paper Covariance matrix of logits equals the row-stochastic redistribution matrix M in Theorem 5.3
    The theorem uses M both as the Gaussian covariance in z~N(mu,M) and as the row-stochastic transition matrix in p'=M q'. No such assumption is stated before the proof.
  • ad hoc to paper Rank-1 covariance structure: Omega_ij/tau ≈ theta p_j for all i,j
    Introduced inside the proof of Theorem 5.4 without being stated in the theorem or justified independently. This is the load-bearing structure that makes the Taylor expansion work.
  • ad hoc to paper Post-TLI draft distribution is approximately uniform, N q'_j = 1+O(theta)
    Asserted in Appendix B.2. False for pruned drafters with small intersection I, where q' is concentrated and N q'_j is far from 1.
  • standard math Non-expansiveness of M^T for row-stochastic M (Lemma B.1)
    Used in the proof of Theorem 5.1. This lemma is correct for row-stochastic matrices.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Out-of-Vocabulary Sampling Boosts Speculative Decoding." pith.science (2026). https://pith.science/paper/HZCBW22J

@misc{pith2026250603206,
  author       = {Pith},
  title        = {Pith review of: Out-of-Vocabulary Sampling Boosts Speculative Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HZCBW22J}},
  note         = {Machine review of arXiv:2506.03206}
}
abstract

Speculative decoding relies on fast and accurate drafters. Recent state-of-the-art language models employ larger and larger vocabularies, which significantly slows down drafters. One promising approach to boost the efficiency of speculative decoding is to use drafters with smaller vocabularies. However, existing sampling methods cannot draw out-of-vocabulary tokens, creating a tradeoff between drafters' vocabulary size and acceptance rates. This paper introduces Redistributing Drafter Kernels (RDK), the first out-of-vocabulary sampler that effectively recovers acceptance rates by virtually restoring pruned target tokens. RDK leverages token-affinity priors to reallocate drafter mass towards high-overlap regions. We prove mathematically that RDK can achieve higher acceptance rates than vanilla and state-of-the-art samplers. We provide an efficient first-order approximation of RDK and prove that it reduces redistribution times from $O(N^2)$ to $O(N)$, enabling lightweight implementations for large vocabularies. Our experiments demonstrate that this linear-time RDK significantly boosts acceptance rates even after extreme pruning (removing more than 75% of the drafter's vocabulary), where existing samplers fail. RDK opens the door to extremely pruned drafters, which were previously impractical.

Figures

Figures reproduced from arXiv: 2506.03206 by the authors.

Figure 1
Figure 1. Token frequency analysis. 12.6% of the tokens (16.1k from 128k) support 95% of the [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Histograms of gpt-2 logit statistics across token positions. (Top-left) Mean logits, (Top￾right) logit standard deviations, (Bottom-left) minimum logits, and (Bottom-right) maximum logits. Mean and minimum values concentrate in the strongly negative range, standard deviations lie between approximately 2 and 4, and maximum logits exhibit a heavy tail up to around 0, indicating that although most logits are negative w… view at source ↗
Figure 3
Figure 3. Empirical distribution of logits for a random prompt compared to a fitted Student [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Histograms of gpt-2 output-probability statistics across token positions. (Top-left) Mean probabilities, (Top-right) standard deviations, (Bottom-left) minimum probabilities, and (Bottom￾right) maximum probabilities. Mean and minimum values concentrate at zero, standar…
Figure 5
Figure 5. Figure 5: Random target distribution (test sample). The dashed vertical line marks the 25% quantile, [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Overlay of probability-mass distributions after pruning to 0.00025% of the vocabulary (500 [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Acceptance rates with synthetic target distributions. TLI fails to recover acceptance rates [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: (Left) Histogram of observed logits overlaid with the fitted normal PDF. (Right) Q-Q [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Q-Q plot of gpt-2 output probabilities vs. a Normal distribution. The systematic departure in the upper tail indicates heavier-tailed behavior than Gaussian. C.3 Co-occurrence matrix To illustrate the prior we use for redistribution, [PITH_FULL_IMAGE:figures/full_fig_…
Figure 10
Figure 10. Figure 10: Q-Q plot of gpt-2 output probabilities vs. a Student t distribution (df estimated via MLE). Even the t fit underestimates the extreme quantiles, reinforcing the need for more flexible, heavy-tailed assumptions in our model. i and token j, so rows sum to 1 by construct…
Figure 11
Figure 11. Figure 11: A 200×200 block of the co-occurrence matrix M for our t-distributed synthetic target. Most entries are near zero, indicating that token affinities are highly sparse. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: Acceptance rates with off-the-shelf LMs. RDK substantially outperforms both baselines [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

14 extracted references · 4 canonical work pages

  1. [1]

    Medusa: Simple LLM inference acceleration framework with multiple decoding heads

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774, 2024

    arXiv 2024

  2. [2]

    Accelerating large language model decoding with speculative sampling

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023

    arXiv 2023

  3. [3]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  4. [4]

    Fast inference from transformers via speculative decoding

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274--19286. PMLR, 2023

    work page 2023

  5. [5]

    Eagle: Speculative sampling requires rethinking feature uncertainty

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint arXiv:2401.15077, 2024 a

    arXiv 2024

  6. [6]

    Eagle-2: Faster inference of language models with dynamic draft trees

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: Faster inference of language models with dynamic draft trees. arXiv preprint arXiv:2406.16858, 2024 b

    arXiv 2024

  7. [7]

    Eagle-3: Scaling up inference acceleration of large language models via training-time test

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-3: Scaling up inference acceleration of large language models via training-time test. arXiv preprint arXiv:2503.01840, 2025

    arXiv 2025

  8. [8]

    Specinfer: Accelerating large language model serving with tree-based speculative inference and verification

    Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. In Proceedings of the 29th ACM ...

    arXiv 2024

Show all 14 references
  1. [9]

    SlimPajama: A 627B token cleaned and deduplicated version of RedPajama

    Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama . https://www.cerebras.net/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama, 2023. U...

    2023

  2. [10]

    Block verification accelerates speculative decoding

    Ziteng Sun, Uri Mendlovic, Yaniv Leviathan, Asaf Aharoni, Jae Hun Ro, Ahmad Beirami, and Ananda Theertha Suresh. Block verification accelerates speculative decoding. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?...

    2025

  3. [11]

    Accelerating llm inference with lossless speculative decoding algorithms for heterogeneous vocabularies

    Nadav Timor, Jonathan Mamou, Daniel Korat, Moshe Berchansky, Oren Pereg, Gaurav Jain, Roy Schwartz, Moshe Wasserblat, and David Harel. Accelerating llm inference with lossless speculative decoding algorithms for heterogeneous vocabularies. arXiv preprint arXiv:2502.05202, 2025 a

    2025 arXiv

  4. [12]

    Distributed speculative inference ( DSI ): Speculation parallelism for provably faster lossless language model inference

    Nadav Timor, Jonathan Mamou, Daniel Korat, Moshe Berchansky, Oren Pereg, Moshe Wasserblat, Tomer Galanti, Michal Gordon-Kiwkowitz, and David Harel. Distributed speculative inference ( DSI ): Speculation parallelism for provably faster lossless language model inference. In The ...

    2025

  5. [13]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  6. [14]

    Fr-spec: Accelerating large-vocabulary language models via frequency-ranked speculative sampling

    Weilin Zhao, Tengyu Pan, Xu Han, Yudi Zhang, Ao Sun, Yuxiang Huang, Kaihuo Zhang, Weilun Zhao, Yuxuan Li, Jianyong Wang, et al. Fr-spec: Accelerating large-vocabulary language models via frequency-ranked speculative sampling. arXiv preprint arXiv:2502.14856, 2025

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.