Pith. sign in

REVIEW 2 major objections 5 minor 51 references

A small autoregressive adapter turns factorized draft marginals into proposal trees that raise acceptance length and speed language-model decoding by 4.37×.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 15:56 UTC pith:ZAGE5H3V

load-bearing objection Solid hybrid drafting + first practical GDN tree kernel; 4.37× / 24.7% gains look real and well-ablated. the 2 major comments →

arxiv 2607.06763 v2 pith:ZAGE5H3V submitted 2026-07-07 cs.LG cs.CL

Trees from Marginals: Autoregressive drafting with factorized priors

Yuma Oda , Ryan Mathieu , Roman Knyazhitskiy , Artur Chakhvadze This is my paper
classification cs.LG cs.CL
keywords speculative decodingfactorized draftersproposal treesWeaver adaptergated delta networkstree verificationacceptance lengthinteractivity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Speculative decoding speeds up autoregressive language models by proposing several tokens at once and checking them in one verifier pass. Factorized draft models generate those proposals very cheaply because they predict each future position independently, but that independence assumption makes acceptance rates fall sharply as the draft grows longer. This paper shows that the ceiling is structural rather than a capacity limit, and that a lightweight residual adapter called Weaver can lift it. Weaver takes the top-K marginals of a factorized drafter as a prior, then builds proposal trees autoregressively over that tiny candidate set, restoring conditional dependencies without ever multiplying by a full vocabulary matrix. Combined with a new rollback-free verification kernel for gated-delta-net layers, the hybrid method reaches a 4.37 imes speedup over ordinary autoregressive decoding and a 24.7 % interactivity gain over a strong factorized baseline on a 27 B model. A sympathetic reader cares because the same idea applies to any parallel drafter and any target that uses non-diagonal linear attention.

Core claim

The acceptance ceiling of factorized drafters is caused by their independence assumption, not by model size; a 56.7 M-parameter autoregressive residual adapter that conditions only on the top-K=512 marginal tokens of a factorized prior restores enough conditional structure that the resulting proposal trees exceed the acceptance rates available to any pure marginal drafter at long depths, delivering a 4.37 imes end-to-end speedup and a 24.7 % gain over an optimized DFlash baseline.

What carries the argument

Weaver: a single-layer residual transformer that receives verifier and factorized-drafter hidden states, predicts residual logits only over the top-K candidate pool, and grows a best-first proposal tree that is verified with a masked triangular solve on gated-delta-net layers so that no recurrent state is ever rolled back.

Load-bearing premise

That the top-512 tokens taken from the factorized marginals almost always contain essentially all of the verifier’s probability mass, so restricting both training and inference to that pool does not discard meaningful acceptance.

What would settle it

Measure the fraction of verifier probability mass that falls outside the top-512 pool on a held-out workload; if that out-of-pool mass routinely exceeds a few percent and acceptance length of Weaver trees falls back to pure-marginal levels, the hybrid claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Longer speculative budgets become useful again because acceptance no longer collapses with depth.
  • Any existing factorized or diffusion-style drafter can be upgraded by the same residual adapter without changing its training recipe.
  • Targets that use gated delta networks or other non-diagonal linear attention can now run tree verification without sequential branch scans or state rollbacks.
  • Interactivity gains of roughly 4 imes over plain autoregressive decoding become available on single-sequence local deployments where batching is not an option.
  • Tree shape can be optimized directly against estimated acceptance probability rather than draft probability alone.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Jointly training the factorized prior and Weaver could reshape the candidate pool itself so that residual corrections become even cheaper.
  • The same top-K residual idea could be applied to multi-token prediction heads or early-exit layers that currently still pay a full-vocabulary cost.
  • Once the verification kernel is already batched, the method naturally extends from interactivity to throughput serving without redesigning the draft model.
  • Annealing the factorized prior temperature by draft position may close part of the remaining gap between pure marginals and the hybrid trees.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper addresses the acceptance-rate ceiling of factorized (parallel-marginal) draft models in speculative decoding. It introduces DFlash-TfM: a hybrid that takes top-K=512 marginals from a pretrained DFlash drafter and feeds them as a prior to Weaver, a 56.7 M-parameter single-layer autoregressive residual adapter that builds proposal trees over the restricted candidate set, avoiding a full-vocabulary projection. For targets with Gated Delta Net layers the authors derive a rollback-free tree-verification algebra (ancestor-masked chunk form of the delta rule) and a fused CUDA kernel in SGLang. On Qwen3.6-27B (bf16, B200, batch 1) the combination yields a 4.37 imes speedup over autoregressive decoding and a 24.7 % interactivity gain over an optimized DFlash baseline across eight chat/math/code workloads (Table 2, Fig. 1). Supporting evidence includes an empirical upper bound on pure-marginal acceptance (Fig. 4), verification ablations (Table 3, Fig. 6), budget sweeps (Appendix B), and kernel micro-benchmarks (Table 1, Fig. 5).

Significance. If the reported numbers hold under broader replication, the work is a clear advance for interactive and local LLM inference. Factorized drafters currently set records at small budgets but saturate; the hybrid restores conditional dependence at modest cost and demonstrably exceeds the pure-marginal acceptance bound at long depths. The GDN tree-verification kernel closes an open systems gap for non-diagonal linear-attention targets and is of independent interest. Strengths that raise confidence include multi-workload tables, isolation of residual vs. tree vs. kernel contributions, explicit comparison against a theoretical marginal oracle, and open weights/kernels. Free parameters (K, η, γ, w, c) are modest and the top-K mass retention is measured (97.8 %).

major comments (2)
  1. §4.3 and Table 2: the main speedup comparison selects the best verification scheme per method (naive verification for DFlash, Traversal/speculative sampling for DFlash-TfM). Table 3 and Fig. 6 show that the ranking of schemes is method-dependent, so the 24.7 % headline is not under a single fixed protocol. A uniform-protocol column (or an explicit statement that the reported numbers are “best-of-protocol”) is needed for the central claim to be fully transparent.
  2. §4.1.3 / footnote: the hybrid’s safety rests on the claim that DFlash’s top-K=512 pool retains 97.8 % of verifier mass on held-out data. The paper already measures this average, but does not report the tail (fraction of positions where mass falls below, e.g., 90 %) nor an ablation that forces a smaller K. A short sensitivity table would confirm that the residual advantage does not collapse when the pool is imperfect.
minor comments (5)
  1. Abstract and §1.2 claim a 77 % / 32 % MAL increase; these numbers appear only in prose. Adding them as explicit columns or footnotes in Table 2 would make the isolation of residual vs. tree contributions easier to verify.
  2. §3.4.1, Eqs. (9)–(12): the ancestor-masked chunk form is clear, but a one-line statement that the construction reduces to ordinary chunked GDN on a linear chain would help readers check the special case.
  3. Fig. 4 caption and §3.5: the “true marginal” estimator uses M random continuations; the value of M and the number of contexts are not stated. Adding them would strengthen the bound’s reproducibility.
  4. §4.2 notes that the public Qwen3.6-DFlash checkpoint is still under training; a short remark on expected sensitivity of the relative gains once the official checkpoint is frozen would be useful.
  5. Typos / notation: “DFlash­TfM” hyphenation is inconsistent across abstract and body; “argmax­mar­ginal” line-break artifacts appear in §3.5; Algorithm 1 uses both “cands” and “candsnode”.

Circularity Check

0 steps flagged

No significant circularity: empirical systems paper whose speedups and acceptance gains are measured against external baselines, not forced by definition or self-citation.

full rationale

The central claims (4.37× over AR, 24.7% over optimized DFlash, higher long-depth acceptance than any pure-marginal oracle) rest on end-to-end wall-clock measurements and ablations (Table 2, Figs. 1/4/6) against public baselines (AR, DFlash, DDTree) on public workloads. Weaver is trained with an LK loss that includes a TV term equal to 1−p_accept; that is ordinary supervised training of a residual adapter, not a tautological redefinition of the reported metrics. The top-K=512 pool restriction is justified by a held-out mass measurement (97.8%), not by construction. The GDN tree-verification algebra (Eqs. 9–13) is a mathematical derivation of a masked triangular solve; the kernel timings are hardware measurements. No load-bearing uniqueness theorem or ansatz is imported from overlapping authors. No equation reduces the claimed speedup or acceptance length to a fitted constant by construction. Score 0 is therefore the correct outcome.

Axiom & Free-Parameter Ledger

5 free parameters · 3 axioms · 2 invented entities

The work rests on standard speculative-sampling theory, the published DFlash factorized drafter, and the GDN recurrence of Yang et al. Free parameters are ordinary hyper-parameters of the adapter and tree builder; the only invented entities are the Weaver adapter itself and the tree-chunk algebra for non-commutative GDN states.

free parameters (5)
  • candidate pool size K = 512
    Fixed at 512; controls both the truncated vocabulary projection and the support of the training loss.
  • LK-loss mixing schedule eta = 2
    Controls the transition from forward KL to total-variation; set to 2 (paper notes default 3 was worse).
  • greedy-matching auxiliary weight gamma = 0.1
    Extra log-prob term on the verifier argmax; set to 0.1.
  • tree expansion width w = 2-8
    Number of frontier nodes expanded in parallel; chosen in {2…8} for bandwidth/compute balance.
  • per-node fan-out c = 8
    Children kept when expanding a node; fixed at 8 in the algorithm box.
axioms (3)
  • domain assumption Speculative sampling acceptance probability equals 1 - TV(p_draft, p_verifier) for single-token proposals (Leviathan et al.).
    Used throughout Sections 2.2 and 3.5 to bound and interpret acceptance rates.
  • domain assumption Gated Delta Net state update S_t = α_t (I - β_t k_t k_t^ op) S_{t-1} + β_t k_t v_t^ op (Yang et al.).
    Starting point for the tree-chunk algebra derived in Section 3.4.
  • ad hoc to paper Top-K marginal tokens from a well-trained factorized drafter capture nearly all verifier mass.
    Empirically measured at 97.8 %; required for the truncated-projection and restricted-loss design to be lossless.
invented entities (2)
  • Weaver residual adapter independent evidence
    purpose: Lightweight autoregressive network that corrects factorized marginals into conditional draft distributions over a tiny candidate set.
    New trainable module (56.7 M parameters) not present in prior factorized or EAGLE-style adapters.
  • Rollback-free GDN tree-verification algebra and fused kernel independent evidence
    purpose: Score an entire proposal tree in one masked triangular solve without speculative state writes or per-branch rollback.
    Closes an open systems problem stated for non-diagonal linear attention; independent of the Weaver architecture.

pith-pipeline@v1.1.0-grok45 · 23522 in / 2964 out tokens · 36846 ms · 2026-07-14T15:56:36.063971+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Trees from Marginals: Autoregressive drafting with factorized priors." pith.science (2026). https://pith.science/paper/ZAGE5H3V

@misc{pith2026260706763,
  author       = {Pith},
  title        = {Pith review of: Trees from Marginals: Autoregressive drafting with factorized priors},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZAGE5H3V}},
  note         = {Machine review of arXiv:2607.06763}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Speculative decoding greatly increases the interactivity of autoregressive language models by trading off computation for extra tokens generated in a single forward pass. Factorized draft models are especially efficient because they predict future-token marginals in parallel, but their independence assumption causes acceptance rates to degrade sharply as the speculative budget grows. We analyze this limitation and introduce Weaver, a lightweight autoregressive adapter that constructs proposal trees from the top-K marginals of a factorized drafter. Weaver restores conditional dependencies between proposed tokens while avoiding a full-vocabulary projection. To support fast verification for models with Gated Delta Net layers, we derive a rollback-free tree-verification algorithm and implement optimized CUDA kernels in SGLang. By combining these model and systems contributions we achieve a 4.37-fold speedup over autoregressive decoding, and outperform a highly optimized DFlash baseline by 24.7%.

Figures

Figures reproduced from arXiv: 2607.06763 by Artur Chakhvadze, Roman Knyazhitskiy, Ryan Mathieu, Yuma Oda.

Figure 1
Figure 1. Figure 1: Comparison of decoding speed for Qwen3.6 [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the DFlash-TfM drafting procedure. A factorized drafter emits marginal distributions for several positions in a single forward pass. Because the factorized drafter ignores conditional dependencies between positions, the acceptance rate falls as the draft length grows ( [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Tree verification modifies the GDN chunk algebra to use a partial [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Acceptance probability for a token at a specific future position [PITH_FULL_IMAGE:figures/full_fig_p017_4.png] view at source ↗
Figure 4
Figure 4. Figure 4: Acceptance rates under speculative sampling computed for different [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The scaling behaviour of verification time vs the number of draft [PITH_FULL_IMAGE:figures/full_fig_p022_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Mean accepted length versus draft length for chain proposals on [PITH_FULL_IMAGE:figures/full_fig_p024_6.png] view at source ↗
Figure 6
Figure 6. Figure 6: Mean accepted length versus draft length for chain proposals on [PITH_FULL_IMAGE:figures/full_fig_p023_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Interactivity versus speculative budget with temperature 1.0 and [PITH_FULL_IMAGE:figures/full_fig_p034_7.png] view at source ↗
Figure 7
Figure 7. Figure 7: Interactivity versus speculative budget with temperature 1.0 and [PITH_FULL_IMAGE:figures/full_fig_p031_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Acceptance length (τ) versus speculative budget temperature 1.0 and reasoning on. 34 [PITH_FULL_IMAGE:figures/full_fig_p034_8.png] view at source ↗
Figure 8
Figure 8. Figure 8: Acceptance length (τ) versus speculative budget temperature 1.0 and reasoning on. 31 [PITH_FULL_IMAGE:figures/full_fig_p031_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Interactivity versus speculative budget with temperature 1.0 and [PITH_FULL_IMAGE:figures/full_fig_p035_9.png] view at source ↗
Figure 9
Figure 9. Figure 9: Interactivity versus speculative budget with temperature 1.0 and [PITH_FULL_IMAGE:figures/full_fig_p032_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Acceptance length (τ) versus speculative budget temperature 1.0 and reasoning off. 35 [PITH_FULL_IMAGE:figures/full_fig_p035_10.png] view at source ↗
Figure 10
Figure 10. Figure 10: Acceptance length (τ) versus speculative budget temperature 1.0 and reasoning off. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Interactivity versus speculative budget with greedy decoding and [PITH_FULL_IMAGE:figures/full_fig_p036_11.png] view at source ↗
Figure 11
Figure 11. Figure 11: Interactivity versus speculative budget with greedy decoding and [PITH_FULL_IMAGE:figures/full_fig_p033_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Acceptance length (τ) versus speculative budget greedy decoding and reasoning on. 36 [PITH_FULL_IMAGE:figures/full_fig_p036_12.png] view at source ↗
Figure 12
Figure 12. Figure 12: Acceptance length (τ) versus speculative budget greedy decoding and reasoning on. 33 [PITH_FULL_IMAGE:figures/full_fig_p033_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Interactivity versus speculative budget with greedy decoding and [PITH_FULL_IMAGE:figures/full_fig_p037_13.png] view at source ↗
Figure 13
Figure 13. Figure 13: Interactivity versus speculative budget with greedy decoding and [PITH_FULL_IMAGE:figures/full_fig_p034_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Acceptance length (τ) versus speculative budget greedy decoding and reasoning off. 37 [PITH_FULL_IMAGE:figures/full_fig_p037_14.png] view at source ↗
Figure 14
Figure 14. Figure 14: Acceptance length (τ) versus speculative budget greedy decoding and reasoning off. 34 [PITH_FULL_IMAGE:figures/full_fig_p034_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: DFlash: acceptance by draft token position on MTBench with tem [PITH_FULL_IMAGE:figures/full_fig_p038_15.png] view at source ↗
Figure 15
Figure 15. Figure 15: DFlash: acceptance by draft token position on MTBench with tem [PITH_FULL_IMAGE:figures/full_fig_p035_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: DFlash-TfM: acceptance by draft token position on MTBench with temperature 1.0 38 [PITH_FULL_IMAGE:figures/full_fig_p038_16.png] view at source ↗
Figure 16
Figure 16. Figure 16: DFlash-TfM: acceptance by draft token position on MTBench with temperature 1.0 35 [PITH_FULL_IMAGE:figures/full_fig_p035_16.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

51 extracted references · 28 linked inside Pith

  1. [1]

    Attention Is All You Need,

    A. Vaswani et al., “Attention Is All You Need,” p. 1, Jun. 2017, [Online]. Available: https://arxiv.org/abs/1706.03762v7

    Pith/arXiv arXiv 2017

  2. [2]

    ISSCC 2014 / SESSION 1 / PLENARY / 1.1 Computing's Energy Problem (and what we can do about it),

    M. Horowitz, “ISSCC 2014 / SESSION 1 / PLENARY / 1.1 Computing's Energy Problem (and what we can do about it),” doi: 10.1109/ ISSCC.2014.6757323

    arXiv 2014

  3. [3]

    Asynchronous RLHF: Faster and More Efficient Off­Policy RL for Language Models,

    M. Noukhovitch, S. Huang, S. Xhonneux, A. Hosseini, R. Agarwal, and A. Courville, “Asynchronous RLHF: Faster and More Efficient Off­Policy RL for Language Models,” 13th International Conference on Learning Repre - sentations, ICLR 2025 , pp. 35297–35323, Oct. 2024, [Online]. Available: https://arxiv.org/abs/2410.18252v3

    Pith/arXiv arXiv 2025

  4. [4]

    AReaL: A Large­Scale Asynchronous Reinforcement Learn­ ing System for Language Reasoning,

    W. Fu et al., “AReaL: A Large­Scale Asynchronous Reinforcement Learn­ ing System for Language Reasoning,” May 2025, [Online]. Available: https://arxiv.org/abs/2505.24298v5

    Pith/arXiv arXiv 2025

  5. [5]

    Fast Inference from Transform­ ers via Speculative Decoding,

    Y. Leviathan, M. Kalman, and Y. Matias, “Fast Inference from Transform­ ers via Speculative Decoding,” Proceedings of Machine Learning Research, 24 vol. 202, pp. 19274–19286, Nov. 2022, [Online]. Available: https://arxiv. org/abs/2211.17192v2

    Pith/arXiv arXiv 2022

  6. [6]

    PARD: Accelerating LLM Inference with Low ­Cost PARallel Draft Model Adaptation,

    Z. An, H. Bai, Z. Liu, D. Li, and E. Barsoum, “PARD: Accelerating LLM Inference with Low ­Cost PARallel Draft Model Adaptation,” Apr. 2025, [Online]. Available: https://arxiv.org/abs/2504.18583v4

    arXiv 2025

  7. [7]

    Introspective Diffusion Language Models,

    Y. Yu et al., “Introspective Diffusion Language Models,” Apr. 2026, [On ­ line]. Available: https://arxiv.org/abs/2604.11035v1

    Pith/arXiv arXiv 2026

  8. [8]

    PARD ­2: Target ­ Aligned Parallel Draft Model for Dual­Mode Speculative Decoding,

    Z. An, T. Liu, Z. Liu, D. Li, R. Liu, and E. Barsoum, “PARD ­2: Target ­ Aligned Parallel Draft Model for Dual­Mode Speculative Decoding,” May 2026, [Online]. Available: https://arxiv.org/abs/2605.08632v1

    Pith/arXiv arXiv 2026

  9. [9]

    DFlash: Block Diffusion for Flash Specu ­ lative Decoding,

    J. Chen, Y. Liang, and Z. Liu, “DFlash: Block Diffusion for Flash Specu ­ lative Decoding,” Feb. 2026, [Online]. Available: https://arxiv.org/abs/ 2602.06036v2

    Pith/arXiv arXiv 2026

  10. [10]

    Traversal Verification for Speculative Tree Decoding,

    Y. Weng et al., “Traversal Verification for Speculative Tree Decoding,” May 2025, [Online]. Available: https://arxiv.org/abs/2505.12398v2

    arXiv 2025

  11. [11]

    Accelerating Speculative Decoding with Block Diffusion Draft Trees,

    L. Ringel and Y. Romano, “Accelerating Speculative Decoding with Block Diffusion Draft Trees,” Apr. 2026, [Online]. Available: https://arxiv.org/ abs/2604.12989v1

    Pith/arXiv arXiv 2026

  12. [12]

    SGLang: Efficient Execution of Structured Language Model Programs,

    L. Zheng et al. , “SGLang: Efficient Execution of Structured Language Model Programs,” Advances in Neural Information Processing Systems, vol. 37, Dec. 2023, doi: 10.52202/079017-2000

    doi:10.52202/079017-2000 2023

  13. [13]

    Ac­ celerating Large Language Model Decoding with Speculative Sampling,

    C. Chen, S. Borgeaud, G. Irving, J.­B. Lespiau, L. Sifre, and J. Jumper, “Ac­ celerating Large Language Model Decoding with Speculative Sampling,” pp. 2023–2025, Feb. 2023, [Online]. Available: https://arxiv.org/abs/ 2302.01318v1

    Pith/arXiv arXiv 2023

  14. [14]

    Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads,

    T. Cai et al., “Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads,” Proceedings of Machine Learning Re - search, vol. 235, pp. 5209–5235, Jan. 2024, [Online]. Available: https:// arxiv.org/abs/2401.10774v3

    Pith/arXiv arXiv 2024

  15. [15]

    EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty,

    Y. Li, F. Wei, C. Zhang, and H. Zhang, “EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty,” Proceedings of Machine Learn- 25 ing Research, vol. 235, pp. 28935–28948, Jan. 2024, [Online]. Available: https://arxiv.org/pdf/2401.15077

    Pith/arXiv arXiv 2024

  16. [16]

    PLD+: Accelerating LLM inference by leveraging Language Model Artifacts,

    S. Somasundaram, A. Phukan, and A. Saxena, “PLD+: Accelerating LLM inference by leveraging Language Model Artifacts,” Dec. 2024, [Online]. Available: https://arxiv.org/abs/2412.01447v1

    Pith/arXiv arXiv 2024

  17. [17]

    SAM Decoding: Speculative Decoding via Suffix Automaton,

    Y. Hu et al., “SAM Decoding: Speculative Decoding via Suffix Automaton,” Nov. 2024, [Online]. Available: https://arxiv.org/abs/2411.10666v2

    Pith/arXiv arXiv 2024

  18. [18]

    Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recycling,

    X. Luo et al. , “Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recycling,” Proceedings of the Annual Meeting of the Association for Computational Linguistics , vol. 1, pp. 6816– 6831, Aug. 2024, doi: 10.18653/v1/2025.acl-long.338

    doi:10.18653/v1/2025.acl-long.338 2024

  19. [19]

    Draft & Verify: Lossless Large Language Model Acceler ­ ation via Self ­Speculative Decoding,

    J. Zhang et al., “Draft & Verify: Lossless Large Language Model Acceler ­ ation via Self ­Speculative Decoding,” Proceedings of the Annual Meeting of the Association for Computational Linguistics , vol. 1, pp. 11263–11282, May 2024, doi: 10.18653/v1/2024.acl-long.607

    doi:10.18653/v1/2024.acl-long.607 2024

  20. [20]

    LayerSkip: Enabling Early Exit Inference and Self ­ Speculative Decoding,

    M. Elhoushi et al. , “LayerSkip: Enabling Early Exit Inference and Self ­ Speculative Decoding,” Proceedings of the Annual Meeting of the Associa - tion for Computational Linguistics, vol. 1, pp. 12622–12642, Apr. 2024, doi: 10.18653/v1/2024.acl-long.681

    doi:10.18653/v1/2024.acl-long.681 2024

  21. [21]

    Hydra: Sequentially ­Dependent Draft Heads for Medusa Decoding,

    Z. Ankner, R. Parthasarathy, A. Nrusimha, C. Rinard, J. Ragan­Kelley, and W. Brandon, “Hydra: Sequentially ­Dependent Draft Heads for Medusa Decoding,” Feb. 2024, [Online]. Available: https://arxiv.org/abs/2402. 05109v2

    2024

  22. [22]

    EAGLE ­2: Faster Inference of Language Models with Dynamic Draft Trees,

    Y. Li, F. Wei, C. Zhang, and H. Zhang, “EAGLE ­2: Faster Inference of Language Models with Dynamic Draft Trees,” EMNLP 2024 - 2024 Confer- ence on Empirical Methods in Natural Language Processing, Proceedings of the Conference, pp. 7421–7432, Jun. 2024, doi: 10.18653/v1/2024.emnlp- main.422

    doi:10.18653/v1/2024.emnlp- 2024

  23. [23]

    EAGLE ­3: Scaling up Inference Acceleration of Large Language Models via Training ­Time Test,

    Y. Li, F. Wei, C. Zhang, and H. Zhang, “EAGLE ­3: Scaling up Inference Acceleration of Large Language Models via Training ­Time Test,” Mar. 2025, [Online]. Available: https://arxiv.org/abs/2503.01840v3 26

    Pith/arXiv arXiv 2025

  24. [24]

    Multi­token­prediction in Gemma 4

    “Multi­token­prediction in Gemma 4.” [Online]. Available: https://blog.google/innovation-and-ai/technology/developers- tools/multi-token-prediction-gemma-4/

  25. [25]

    Markov Chains and Mixing Times, second edition

    D. A. Levin, Y. Peres, and E. L. Wilmer, “Markov Chains and Mixing Times, second edition.”

  26. [26]

    Block Verification Accelerates Speculative Decoding,

    Z. Sun et al., “Block Verification Accelerates Speculative Decoding,” Apr. 2025, [Online]. Available: http://arxiv.org/abs/2403.10444

    Pith/arXiv arXiv 2025

  27. [27]

    SpecInfer: Accelerating Generative Large Language Model Serving with Tree­based Speculative Inference and Verification,

    X. Miao et al., “SpecInfer: Accelerating Generative Large Language Model Serving with Tree­based Speculative Inference and Verification,” Interna- tional Conference on Architectural Support for Programming Languages and Operating Systems - ASPLOS , vol. 3, pp. 932–949, Apr. 2024, doi: 10.1145/3620666.3651335

    doi:10.1145/3620666.3651335 2024

  28. [28]

    Sequoia: Scalable, Robust, and Hardware ­aware Specula­ tive Decoding,

    Z. Chen et al., “Sequoia: Scalable, Robust, and Hardware ­aware Specula­ tive Decoding,” Feb. 2024, [Online]. Available: https://arxiv.org/abs/ 2402.12374v3

    Pith/arXiv arXiv 2024

  29. [29]

    SpecTr: Fast Speculative Decoding via Optimal Transport,

    Z. Sun, A. Beirami, A. T. Suresh, H. Jain, J. H. Ro, and F. Yu, “SpecTr: Fast Speculative Decoding via Optimal Transport,” Advances in Neural Information Processing Systems , vol. 36, Oct. 2023, [Online]. Available: https://arxiv.org/abs/2310.15141v2

    Pith/arXiv arXiv 2023

  30. [30]

    Multi­Draft Speculative Sampling: Canonical Decomposition and Theoretical Limits,

    A. Khisti, M. R. Ebrahimi, H. Dbouk, A. Behboodi, † R. Memisevic, and C. Louizos, “Multi­Draft Speculative Sampling: Canonical Decomposition and Theoretical Limits,” Oct. 2024, [Online]. Available: https://arxiv. org/abs/2410.18234v2

    Pith/arXiv arXiv 2024

  31. [31]

    DySpec: Faster Speculative Decoding with Dynamic Token Tree Structure,

    Y. Xiong, R. Zhang, Y. Li, T. Wu, and L. Zou, “DySpec: Faster Speculative Decoding with Dynamic Token Tree Structure,” Oct. 2024, [Online]. Available: https://arxiv.org/abs/2410.11744v1

    Pith/arXiv arXiv 2024

  32. [32]

    The Mamba in the Llama: Distilling and Accelerating Hybrid Models,

    J. Wang, D. Paliotta, A. May, A. M. Rush, and T. Dao, “The Mamba in the Llama: Distilling and Accelerating Hybrid Models,” Advances in Neural Information Processing Systems , vol. 37, Aug. 2024, doi: 10.52202/079017-1996. 27

    doi:10.52202/079017-1996 2024

  33. [33]

    Snakes and Ladders: Accelerating State Space Model Inference with Speculative Decoding

    Y. Wu, Y. Dukler, † M. Trager, A. Achille, W. Xia, and S. Soatto, “Snakes and Ladders: Accelerating State Space Model Inference with Speculative Decoding.”

  34. [34]

    STree: Speculative Tree Decoding for Hybrid State­Space Models,

    Y. Wu, Z. Qin, A. Wong, and S. Soatto, “STree: Speculative Tree Decoding for Hybrid State­Space Models,” May 2025, [Online]. Available: https:// arxiv.org/abs/2505.14969v2

    arXiv 2025

  35. [35]

    Gated Delta Networks: Improving Mamba2 with Delta Rule,

    S. Yang, J. Kautz, and A. Hatamizadeh, “Gated Delta Networks: Improving Mamba2 with Delta Rule,” 13th International Conference on Learning Rep- resentations, ICLR 2025, pp. 73311–73331, Dec. 2024, [Online]. Available: https://arxiv.org/abs/2412.06464v3

    Pith/arXiv arXiv 2025

  36. [36]

    Parallelizing Linear Transformers with the Delta Rule over Sequence Length,

    S. Yang, B. Wang, Y. Zhang, Y. Shen, and Y. Kim, “Parallelizing Linear Transformers with the Delta Rule over Sequence Length,” Advances in Neural Information Processing Systems , vol. 37, Jun. 2024, doi: 10.52202/079017-3668

    doi:10.52202/079017-3668 2024

  37. [37]

    MT ­Bench­101: A Fine ­Grained Benchmark for Evalu ­ ating Large Language Models in Multi ­Turn Dialogues,

    G. Bai et al. , “MT ­Bench­101: A Fine ­Grained Benchmark for Evalu ­ ating Large Language Models in Multi ­Turn Dialogues,” Nov. 2024, doi: 10.18653/v1/2024.acl-long.401

    doi:10.18653/v1/2024.acl-long.401 2024

  38. [38]

    LK Losses: Direct Acceptance Rate Optimization for Speculative Decod­ ing,

    A. Samarin, S. Krutikov, A. Shevtsov, S. Skvortsov, F. Fisin, and A. Golubev, “LK Losses: Direct Acceptance Rate Optimization for Speculative Decod­ ing,” Feb. 2026, [Online]. Available: https://arxiv.org/abs/2602.23881 v2

    Pith/arXiv arXiv 2026

  39. [39]

    Nemotron ­Post­Training­Dataset­v2

    D. Nathawani et al. , “Nemotron ­Post­Training­Dataset­v2.” [Online]. Available: https://huggingface.co/datasets/nvidia/Nemotron-Post- Training-Dataset-v2

  40. [40]

    LMSYS­Chat­1M: A Large ­Scale Real­World LLM Con­ versation Dataset

    L. Zheng et al., “LMSYS­Chat­1M: A Large ­Scale Real­World LLM Con­ versation Dataset.” 2023

    2023

  41. [41]

    OpenHermes 2.5: An Open Dataset of Synthetic Data for Generalist LLM Assistants

    Teknium, “OpenHermes 2.5: An Open Dataset of Synthetic Data for Generalist LLM Assistants.” [Online]. Available: https://huggingface. co/datasets/teknium/OpenHermes-2.5

  42. [42]

    Code Alpaca: An Instruction­following LLaMA model for code generation

    S. Chaudhary, “Code Alpaca: An Instruction­following LLaMA model for code generation.” GitHub, 2023. 28

    2023

  43. [43]

    Muon is Scalable for LLM Training,

    J. Liu et al. , “Muon is Scalable for LLM Training,” Feb. 2025, [Online]. Available: https://arxiv.org/pdf/2502.16982

    Pith/arXiv arXiv 2025

  44. [44]

    Decoupled Weight Decay Regularization,

    I. Loshchilov and F. Hutter, “Decoupled Weight Decay Regularization,” 7th International Conference on Learning Representations, ICLR 2019, Nov. 2017, [Online]. Available: https://arxiv.org/pdf/1711.05101

    Pith/arXiv arXiv 2019

  45. [45]

    ShareChat: A Dataset of Chatbot Conversations in the Wild

    Y. Yan, T. Nguyen, B. Su, M. Lieffers, and T. Le, “ShareChat: A Dataset of Chatbot Conversations in the Wild.” [Online]. Available: https://arxiv. org/abs/2512.17843

    Pith/arXiv arXiv

  46. [46]

    Training Verifiers to Solve Math Word Problems,

    K. Cobbe et al., “Training Verifiers to Solve Math Word Problems,” arXiv preprint arXiv:2110.14168, 2021

    Pith/arXiv arXiv 2021

  47. [47]

    Let's Verify Step by Step,

    H. Lightman et al. , “Let's Verify Step by Step,” arXiv preprint arXiv:2305.20050, 2023

    Pith/arXiv arXiv 2023

  48. [48]

    American Invitational Mathematics Examina­ tion (AIME) 2025

    Y. Zhang and T. Math­AI, “American Invitational Mathematics Examina­ tion (AIME) 2025.” 2025

    2025

  49. [49]

    Evaluating Large Language Models Trained on Code,

    M. Chen et al., “Evaluating Large Language Models Trained on Code,” 2021

    2021

  50. [50]

    Program Synthesis with Large Language Models,

    J. Austin et al., “Program Synthesis with Large Language Models,” arXiv preprint arXiv:2108.07732, 2021

    Pith/arXiv arXiv 2021

  51. [51]

    LiveCodeBench: Holistic and Contamination Free Evalu ­ ation of Large Language Models for Code,

    N. Jain et al., “LiveCodeBench: Holistic and Contamination Free Evalu ­ ation of Large Language Models for Code,” 13th International Conference on Learning Representations, ICLR 2025 , pp. 25479–25519, Mar. 2024, [Online]. Available: https://arxiv.org/pdf/2403.07974 29 Appendix A. Pseudocode for tree construction Algorithm 1: Adaptive tree construction. T...

    Pith/arXiv arXiv 2025