Pith. sign in

REVIEW 4 major objections 5 minor 43 references

Scaling Speculative Decoding with Lookahead Reasoning

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Whole-step guesses lift speculative decoding from 1.4x to 2.1x

desk verdict A credible incremental step: semantic step-level speculation multiplies with token-level SD, but the headline 2.1x speedup is partly a resource effect from an unmeasured 7B judge and two extra H100s. read the letter →

arxiv 2506.19830 v1 pith:BQEWM3B6 submitted 2025-06-24 cs.LG cs.CL

Yichao Fu , Rui Ge , Zelei Shao , Zhijie Deng , Hao Zhang This is my paper · ORCID
classification cs.LGcs.CL
keywords speculativedecodinglookaheadreasoningstep-levelspeculationsemanticverificationmodelschain-of-thoughtLLM-as-a-judgeinferenceacceleration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that speculative decoding for reasoning models is capped by its own token-level assumption, not by hardware, and that the cap can be broken with a second, coarser layer of speculation. A lightweight draft model proposes several future reasoning steps; the target model expands each proposal in one batched pass; and a semantic verifier keeps the steps that match in meaning, letting the target regenerate the rest. Because reasoning steps only need to be semantically correct, whole steps can be accepted even when their exact tokens differ. Across GSM8K, AIME, and other benchmarks, the paper reports that this raises the peak speedup of token-level speculative decoding from about 1.4x to about 2.1x while keeping final-answer accuracy within roughly two points of the autoregressive baseline. It also proves that under a fixed parallelism budget and realistic acceptance rates and draft costs, maximum speedup requires using both step-level and token-level speculation together.

What carries the argument

The carrying mechanism is a step-level guess-and-verify cycle, run asynchronously and repeatedly. The draft model generates $\gamma$ candidate steps; the target model fills in one step per draft prefix in parallel; and a 7B LLM-as-a-Judge verifier checks semantic equivalence between each draft step and the corresponding target step, accepting the longest prefix of aligned steps. The step-level speedup has form $f(k_1)$ depending on acceptance rate $\alpha_1$ and draft cost ratio $c_1$, while token-level SD has speedup $g(k_2)$, and the combined speedup is the product $h(k_1,k_2)=f(k_1)g(k_2)$. The theorem's role is to show that when the two dimensions compete for a fixed parallelism budget $M$, the product is maximized only in the interior, with both $k_1\ge2$ and $k_2\ge2$.

What would settle it

Instrument the judge's forward-pass time and recompute end-to-end speedup with that latency charged to each cycle: if lookahead plus token-level SD falls to or below token-level SD alone, the claim fails. Independently, measure whether the judge's accept/reject decisions track changes to the final answer better than chance on GSM8K and AIME.

Watch

Extended reading notes

Core claim

The central claim is that reasoning models have a step-level axis of parallelism that multiplies with token-level speculative decoding, changing the algorithmic ceiling of acceleration. The paper introduces Lookahead Reasoning: a draft model proposes $\gamma$ future reasoning steps, the target model generates one continuation for each proposal in a single batched pass, and a verifier finds the longest prefix of draft steps that are semantically equivalent to the target continuations, appending those draft steps and then the target's correction. Token-level speculative decoding still runs inside each step, so the two layers multiply. Empirically, the combined method lifts peak speedup from 1.4x to about 2.1x across the evaluated benchmarks and model pairs, with accuracy deviations between about +1.0 and -2.1 points. The theoretical companion, Theorem 4, states that for acceptance rates in $(0.52,0.8)$, draft cost ratios $c_1<1/3$ and $c_2<1/5$, and a parallelism budget $M\ge16$, the maximum speedup is achieved if and only if both step-level and token-level speculation are used.

Load-bearing premise

The load-bearing premise is that the semantic verifier catches wrong draft steps reliably and adds so little wall-clock time that the reported end-to-end speedups hold; the paper does not directly measure the judge's latency contribution.

Editorial extensions

If this is right

  • Peak speedup of speculative decoding is no longer set by token-match probability; step-level speculation raises the ceiling from 1.4x to about 2.1x on the tested models and benchmarks.
  • Step-level and token-level speculation multiply, so a system that already uses token-level SD can add lookahead reasoning without replacing its existing mechanism.
  • Under the theorem's conditions, neither pure step-level nor pure token-level speculation is optimal for a fixed parallelism budget; both dimensions must be active.
  • Verifier quality becomes a first-order design variable: LLM-as-a-judge preserves accuracy, while random acceptance or target-model scoring can cost multiple accuracy points.
  • Because step-level speculation uses additional FLOPs to draft more or deeper steps, the speedup scales better with added GPU throughput than token-level SD alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper leaves implicit: replace the newline-based step splitter with a learned or model-conditional segmenter, which should raise acceptance rates if step boundaries better match semantic units.
  • If the judge's wall-clock cost were measured and amortized, the step-level layer could also be paired with draft-head token speculation instead of n-gram lookup; the multiplicative speedup formula suggests gains would compound.
  • The theorem's parameter regime implies a further prediction the paper does not test: as the parallelism budget $M$ grows, the optimal split between step-level depth $k_1$ and token-level width $k_2$ should shift, so tuning that ratio should yield larger speedups on future hardware.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Lookahead Reasoning (LR), a step-level speculative decoding method for large reasoning models. A small draft model proposes several future reasoning steps; the target model generates candidate continuation steps in a single batched pass; a 7B LLM-as-a-Judge verifies whether each draft step is semantically equivalent to the target's step, accepting the longest prefix of semantically correct drafts and regenerating from the first failure. Token-level n-gram speculative decoding continues to operate inside each step, giving two orthogonal parallelism axes whose speedups are claimed to multiply. The authors derive step-level speedup formulas for synchronous and asynchronous variants, prove a hybrid optimality theorem under a parallelism budget, and report experiments on DeepSeek-R1-Distill and Qwen3 draft/target pairs across GSM8K, AIME, AMC12, HumanEval, GPQA, MT-Bench, and LiveCodeBench. The headline claim is that LR combined with n-gram SD raises the peak end-to-end speedup from about 1.4x to 2.1x relative to autoregressive decoding while keeping task accuracy within roughly 2% of the target-model baseline.

Significance. The core idea is timely and genuinely novel: exploiting the step-level semantic structure of reasoning chains as an additional speculation axis is a natural way to attack the algorithmic ceiling of token-level speculative decoding. If the empirical speedup claim holds, the paper would make a useful contribution to efficient long-CoT inference, and the authors are to be credited for releasing code and for comparing against a concrete token-level SD baseline. The theoretical formulas in Section 3.2 are standard re-derivations of the speculative-decoding bound applied to steps, and the hybrid optimality theorem is conditional on assumed acceptance-rate and cost ranges, so the main load is carried by the experiments. Unfortunately, as described below, the experimental support for the headline wall-clock speedup is incomplete: the verifier's latency is never measured, the GPU count is not held fixed across conditions, and the accuracy-preservation claim is statistically weak on AIME. The idea is significant enough to warrant publication, but the current quantitative claims require revision and additional measurements.

major comments (4)
  1. [Sections 3.2 and 4.1, Table 1] The reported speedups are end-to-end wall-clock numbers, but the 7B judge's inference time is never measured or disclosed. The theoretical speedup analysis explicitly assumes "negligible verification overhead" (Section 3.2 and Appendix B.1.1), and each draft-step comparison costs at least one prefill pass of Qwen2.5-7B-Instruct. Over a long chain of thought this is many judge calls per generation. The paper should report the judge's wall-clock contribution as a fraction of total generation time, or provide an ablation that subtracts or overlaps the judge cost, before the 2.11x claim can be accepted.
  2. [Section 4, Testbed] The resource accounting is asymmetric across conditions. The autoregressive baseline and the n-gram SD baseline run the 32B target on two H100 GPUs, while LR and SD+LR additionally run the 1.5B draft on one H100 and the 7B judge on another H100. Thus the advertised 2.1x speedup is achieved with roughly twice the GPU allocation for a single request. A fixed-GPU comparison, or a throughput-per-GPU normalization, is needed to separate algorithmic gain from additional hardware.
  3. [Section 4.1, Table 1 and Section 4.3, Table 2] The claim that LR "preserves answer quality" is not statistically supported on AIME: LR accuracy is 69.2 +/- 8.1 versus the target baseline's 70.8 +/- 5.2, and the confidence intervals overlap substantially. The verifier ablation in Table 2 shows that weaker verifiers (embedding at 0.85, target scoring at threshold 7) degrade accuracy by 2-11 points, and Table 3 shows that wider trees degrade accuracy even with the 7B judge. The paper does not report false-accept or false-reject rates against any ground-truth step-equivalence set, so the accuracy-preservation claim rests entirely on end-task accuracy with a noisy AIME estimate. Please report verifier precision/recall on a labeled sample and a paired significance test for the accuracy comparisons.
  4. [Appendix B.1.2, Theorem 4 proof, Step 2 Case 2] The proof of Theorem 4 asserts the inequality h(ceil(1/c1),1) = h(M,1) < h(M/2,2) without a supporting argument. In the case ceil(1/c1) < M < 2*ceil(1/c1), we have M/2 < ceil(1/c1) < M, so Lemma 2 gives f(M/2) < f(M) with f(M) at its plateau; the comparison of f(M/2)*g(2) with f(M)*g(1) does not follow from the stated monotonicity and unimodality lemmas. This is a gap in the "if and only if" hybrid-optimality theorem. Either supply the missing argument or restrict the theorem's conditions accordingly. In addition, the sync speedup formula appears to be indexed one token/step off relative to Algorithm 1, where the target generates gamma+1 steps in parallel; please reconcile the notation.
minor comments (5)
  1. [Section 4, Datasets] The AMC12'23 subset is described only as "40 out of 50 problems, selected by Qwen2.5 Math"; the selection procedure is not specified, which leaves room for selection bias in that benchmark. Please describe the selection or use the full set.
  2. [Table 1 and Figure 3] Speedup numbers are reported without error bars or confidence intervals, even though acceptance rates and latencies vary across runs. Please report standard errors or repeated-run ranges.
  3. [Figure 3] The x-axis labels "Speculative Steps" and "Speculative Tokens" range from 5 to 30, while the default configuration uses 6 steps and 8 tokens; the figure would be easier to read if the varied parameter and the fixed counterfactual were stated in the caption.
  4. [Section 5] SpecReason is cited and its accuracy is reported in Table 1, but no speedup comparison with SpecReason is given. A direct latency or throughput comparison would clarify the claimed advantage over this closest prior step-level method.
  5. [Throughout] There are several typographical errors, including "preseve" (Section 4.1), "orcale" (Figure 1 caption), and "Janaury" (Section 4); a proofreading pass is needed.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the speedups are measured against external benchmarks and the theoretical formulas are conditional re-derivations of the standard speculative-decoding bound; the overlapping-author citations are contextual and not load-bearing.

full rationale

The central empirical claims are measured, not derived: Table 1 reports wall-clock speedups and accuracy on external benchmarks (GSM8K, AIME, HumanEval, GPQA, MT-Bench, LiveCodeBench), and the 1.4x-to-2.1x improvement is an experimental result, not the output of a fitted equation. The theoretical analysis in Section 3.2 is explicitly conditional: it states 'We make simplifying assumptions for clarity: negligible verification overhead, constant cost for generating steps, and a single draft branch at each stage,' and the speedup formulas f_sync(k1), f_async(k1), and g(k2) are direct analogues of the known speculative-decoding bound applied at step granularity. Appendix B.1.1 says 'The proof follows the same reasoning as in [3]. The only difference is that our gamma represents the maximum number of tokens the large model generates in parallel,' which is an acknowledged re-derivation rather than a disguised fit. Theorem 4 is a conditional optimization statement over assumed parameter ranges (0.52<alpha1,alpha2<0.8, c1<1/3, c2<1/5, M>=16); it is not used to produce the measured speedups and does not reduce the experimental outcome to its assumptions. The only self-citations by overlapping authors (Lookahead Decoding [12], CLLM [26], and goodput optimization [28]) appear in Related Work and background as contextual comparisons, not as load-bearing justification for the central claim; no uniqueness theorem or prior result by the same authors is invoked to forbid alternatives. The paper's own limitation statement, 'current verifiers trade speed for accuracy; faster, lightweight alternatives remain an open challenge,' and the absence of a wall-clock breakdown for the 7B judge are real resource-accounting and correctness risks, but they are not circularity: the verifier is a system component whose judgments and latency are not defined in terms of the reported speedup. Overall, the derivation chain is self-contained against external benchmarks, so the circularity score is low.

Assumptions & free parameters 6 free parameters · 7 assumptions · 0 invented entities

The central empirical speedup is measured directly and does not depend on the theory. The theoretical claims (hybrid optimality, S1, S2) rest on assumed ranges for α1, α2, c1, c2, M and on independence and negligible-verifier-overhead assumptions. The paper's own limitation section notes that step splitting by '\n\n' is crude and that verifiers trade speed for accuracy, which confirms that axioms 2, 4, and 6 are live fragility points rather than harmless simplifications.

free parameters (6)
  • α_1 (step-level acceptance rate) = Measured per dataset, 0.30 to 0.63 (Table 1)
    Used as the probability of draft step acceptance in the speedup formulas; the paper assumes a 0.52 to 0.8 range for the hybrid optimality theorem, which some empirical rates (Qwen3, 0.30 to 0.53) fall outside.
  • α_2 (token-level acceptance rate) = Not directly reported
    Assumed in the 0.52 to 0.8 range for Theorem 4; token acceptance rate for n-gram SD is not measured or reported.
  • c_1 (draft step cost ratio) = Not measured; assumed less than 1/3
    Cost of one draft step relative to one target step; assumed small for the hybrid optimality theorem.
  • c_2 (token-level draft cost ratio) = Not measured; assumed less than 1/5
    Cost of one token draft relative to target token for token-level SD; assumed small for Theorem 4.
  • M (parallelism budget) = Not specified for experiments; assumed at least 16 (even)
    Total degree of parallelism constraint in the resource allocation problem; the theorem requires M >= 16.
  • γ_1, γ_2 (draft steps and tokens) = γ_1 = 6, γ_2 = 8 by default
    Number of speculative steps and speculative tokens used in experiments; these are hand-chosen hyperparameters.
assumptions (7)
  • domain assumption A reasoning step only needs semantic correctness, not exact token matching (Section 1, 4.1).
    Foundational premise of the method; supported only by the informal substitution observation in Section 1 and by acceptance-rate results, not by a dedicated controlled experiment.
  • domain assumption The LLM-as-a-Judge verifier correctly identifies semantic equivalence (Section 3.1).
    If the judge accepts bad steps, accuracy degrades; if it is slow, reported speedups overstate real latency gains. The paper ablates judge types but never measures judge latency.
  • domain assumption Step acceptances are independent with constant probability α1 (Section 3.2, Appendix B).
    The geometric distribution for accepted steps assumes identical independent acceptances, which is likely false in real reasoning chains where steps are correlated.
  • domain assumption Draft and target step generation costs are constant and verifier overhead is negligible (Section 3.2).
    Used to derive both S1 and S2; in practice step lengths vary and the judge is a 7B model whose compute is not negligible.
  • standard math The standard token-level SD speedup formula holds for the n-gram drafter (Section 2).
    The paper adopts the Leviathan et al. guarantee for its g(γ2) formula; this is an accepted background result.
  • domain assumption '\n\n' reliably separates reasoning steps in the evaluated models (Section 3.1).
    Step boundaries are detected by newlines only; the authors themselves list smarter segmentation as a limitation.
  • standard math Law of large numbers for asymptotic speedup (Appendix B).
    Used to pass from n-stage averages to expectations in S1 and S2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scaling Speculative Decoding with Lookahead Reasoning." pith.science (2026). https://pith.science/paper/BQEWM3B6

@misc{pith2026250619830,
  author       = {Pith},
  title        = {Pith review of: Scaling Speculative Decoding with Lookahead Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BQEWM3B6}},
  note         = {Machine review of arXiv:2506.19830}
}
abstract

Reasoning models excel by generating long chain-of-thoughts, but decoding the resulting thousands of tokens is slow. Token-level speculative decoding (SD) helps, but its benefit is capped, because the chance that an entire $\gamma$-token guess is correct falls exponentially as $\gamma$ grows. This means allocating more compute for longer token drafts faces an algorithmic ceiling -- making the speedup modest and hardware-agnostic. We raise this ceiling with Lookahead Reasoning, which exploits a second, step-level layer of parallelism. Our key insight is that reasoning models generate step-by-step, and each step needs only to be semantically correct, not exact token matching. In Lookahead Reasoning, a lightweight draft model proposes several future steps; the target model expands each proposal in one batched pass, and a verifier keeps semantically correct steps while letting the target regenerate any that fail. Token-level SD still operates within each reasoning step, so the two layers of parallelism multiply. We show Lookahead Reasoning lifts the peak speedup of SD both theoretically and empirically. Across GSM8K, AIME, and other benchmarks, Lookahead Reasoning improves the speedup of SD from 1.4x to 2.1x while preserving answer quality, and its speedup scales better with additional GPU throughput. Our code is available at https://github.com/hao-ai-lab/LookaheadReasoning

Figures

Figures reproduced from arXiv: 2506.19830 by the authors.

Figure 1
Figure 1. One cycle of LOOKAHEAD REASONING. The draft model proposes γ = 3 steps {sˆ1, sˆ2, sˆ3}. The target model then generate {s1, s2, s3} based on prefixes and {sˆ1, sˆ2, sˆ3}, respectively. Verifier checks if draft and target steps are semantically equivalent (e.g., s1 ≈ sˆ1). If the first two steps are equivalent but the third is not, LOOKAHEAD REASONING outputs the verified draft steps (sˆ1, sˆ2) followed by the target… view at source ↗
Figure 2
Figure 2. Speedup vs Draft Tokens. Speedup over autoregressive decoding, comparing LOOKA￾HEAD REASONING combined with token-level SD (NGram-based) (red line) to SD alone (blue line). Our method is orthogonal to token-level SD and improves the maximum speedup from 1.4× to 2.1×. This paper makes a key observation that reasoning is naturally hierarchical: a full chain-of-thought breaks into discrete steps, and each step unrolls … view at source ↗
Figure 3
Figure 3. Orthogonality of Lookahead Reasoning and Speculative Decoding. When used alone, the speedup from both LR and SD is limited by their draft length (γ). However, their combination consistently improves the max achievable speedup. 4.3 Ablation Study Effectiveness of the Verifier. We conducted an ablation study to assess the impact of different verifier mechanisms on task accuracy, utilizing DeepSeek-R1-Distill 32B as th… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Illustration of hybrid approach. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_4.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 15 canonical work pages

  1. [1]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  2. [2]

    OpenAI, :, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondri...

    2024

  3. [3]

    Fast inference from transformers via speculative decoding

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. InInternational Conference on Machine Learning, pages 19274–19286. PMLR, 2023

    2023

  4. [4]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  5. [5]

    Openai o3 and o4-mini system card

    OpenAI. Openai o3 and o4-mini system card. April 2025. 11

    work page 2025

  6. [6]

    Qwen3 technical report

    Qwen Team. Qwen3 technical report. 2025

    work page 2025

  7. [7]

    Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    2023

  8. [8]

    Sentence-bert: Sentence embeddings using siamese bert- networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019

    2019

Show all 43 references
  1. [9]

    Specrea- son: Fast and accurate inference-time compute via speculative reasoning.arXiv preprint arXiv:2504.07891, 2025

    Rui Pan, Yinwei Dai, Zhihao Zhang, Gabriele Oliaro, Zhihao Jia, and Ravi Netravali. Specrea- son: Fast and accurate inference-time compute via speculative reasoning.arXiv preprint arXiv:2504.07891, 2025

    2025 arXiv

  2. [10]

    Eagle-2: Faster inference of language models with dynamic draft trees.arXiv preprint arXiv:2406.16858, 2024

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: Faster inference of language models with dynamic draft trees.arXiv preprint arXiv:2406.16858, 2024

    2024 arXiv

  3. [11]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  4. [12]

    Break the sequential dependency of llm inference using lookahead decoding.arXiv preprint arXiv:2402.02057, 2024

    Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of llm inference using lookahead decoding.arXiv preprint arXiv:2402.02057, 2024

    2024 arXiv

  5. [13]

    Eagle-3: Scaling up inference acceleration of large language models via training-time test.arXiv preprint arXiv:2503.01840, 2025

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-3: Scaling up inference acceleration of large language models via training-time test.arXiv preprint arXiv:2503.01840, 2025

    2025 arXiv

  6. [14]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott G...

    2021

  7. [15]

    Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974, 2024

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  8. [16]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    arXiv

  9. [17]

    Aime problems and solutions, 2025

    AIME. Aime problems and solutions, 2025

    2025

  10. [18]

    Amc 12 problems and solutions, 2025

    AMC12. Amc 12 problems and solutions, 2025

    2025

  11. [19]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. InFirst Conference on Language Modeling, 2024

    2024

  12. [20]

    Qwen2.5-math: The world’s leading open-sourced mathematical llms, 2024

    Qwen Team. Qwen2.5-math: The world’s leading open-sourced mathematical llms, 2024

    2024

  13. [21]

    Prompt lookup decoding, November 2023

    Apoorv Saxena. Prompt lookup decoding, November 2023

    2023

  14. [22]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. InProceedings of the 29th Symposium on Operating Systems Principles, pa...

    2023

  15. [23]

    Medusa: Simple llm inference acceleration framework with multiple decoding heads

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple llm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774, 2024

    2024 arXiv

  16. [24]

    Hydra: Sequentially-dependent draft heads for medusa decoding.arXiv preprint arXiv:2402.05109, 2024

    Zachary Ankner, Rishab Parthasarathy, Aniruddha Nrusimha, Christopher Rinard, Jonathan Ragan-Kelley, and William Brandon. Hydra: Sequentially-dependent draft heads for medusa decoding.arXiv preprint arXiv:2402.05109, 2024

    2024 arXiv

  17. [25]

    Eagle: Speculative sampling requires rethinking feature uncertainty.arXiv preprint arXiv:2401.15077, 2024

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty.arXiv preprint arXiv:2401.15077, 2024

    2024 arXiv

  18. [26]

    Cllms: Consistency large language models

    Siqi Kou, Lanxiang Hu, Zhezhi He, Zhijie Deng, and Hao Zhang. Cllms: Consistency large language models. InForty-first International Conference on Machine Learning, 2024

    2024

  19. [27]

    Specinfer: Accelerating generative large language model serving with tree-based speculative inference and verification

    Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, et al. Specinfer: Accelerating generative large language model serving with tree-based speculative inference and verification. arXiv pre...

    2023 arXiv

  20. [28]

    Optimizing speculative decoding for serving large language models using goodput.arXiv preprint arXiv:2406.14066, 2024

    Xiaoxuan Liu, Cade Daniel, Langxiang Hu, Woosuk Kwon, Zhuohan Li, Xiangxi Mo, Alvin Cheung, Zhijie Deng, Ion Stoica, and Hao Zhang. Optimizing speculative decoding for serving large language models using goodput.arXiv preprint arXiv:2406.14066, 2024

    2024 arXiv

  21. [29]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  22. [30]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  23. [31]

    s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393, 2025

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393, 2025

    2025 arXiv

  24. [32]

    From decoding to meta-generation: Inference-time algorithms for large language models.arXiv preprint arXiv:2406.16838, 2024

    Sean Welleck, Amanda Bertsch, Matthew Finlayson, Hailey Schoelkopf, Alex Xie, Graham Neubig, Ilia Kulikov, and Zaid Harchaoui. From decoding to meta-generation: Inference-time algorithms for large language models.arXiv preprint arXiv:2406.16838, 2024

    2024 arXiv

  25. [33]

    Scaling llm test-time compute opti- mally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314, 2024

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute opti- mally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314, 2024

    2024 arXiv

  26. [34]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599, 2025

    2025 arXiv

  27. [35]

    aligned" or

    Wang Yang, Xiang Yue, Vipin Chaudhary, and Xiaotian Han. Speculative thinking: En- hancing small-model reasoning with large model guidance at inference time.arXiv preprint arXiv:2504.12329, 2025. 13 A Judgement Prompt Template Semantic Equivalence Analysis Prompt <|im_start|>s...

    2025 arXiv

  28. [36]

    The draft model sequentially generate drafts

    If the number of generations the large model performs in parallel is less thanγ. The draft model sequentially generate drafts

  29. [37]

    Each time when we start to generate a draft step, we immediately ask the target model to generate a target step

  30. [38]

    If the draft was reject, fall back to the target model’s original sentence and proceed to the nextDRAFT STAGE

    After the target model finished generation, immediately ask the verifier to verify whether should we accept the draft. If the draft was reject, fall back to the target model’s original sentence and proceed to the nextDRAFT STAGE. 14 Since each draft is accepted independently, ...

  31. [39]

    The parallel dimension of the target model is ⌈ 1 c1 ⌉, and asn→∞, the asymptotic speedup is S1 = 1 c1 + (1−c 1)(1−α 1)

    Ifγ≥⌈ 1 c1 ⌉, the draft tree never saturates. The parallel dimension of the target model is ⌈ 1 c1 ⌉, and asn→∞, the asymptotic speedup is S1 = 1 c1 + (1−c 1)(1−α 1)

  32. [40]

    The parallel dimension of the target model isγ, and asn→∞, the asymptotic speedup is S2 = 1−α γ 1 (1−α 1) +c 1 α1−αγ+1 1 −γ(1−α 1)αγ 1

    Ifγ <⌈ 1 c1 ⌉, the draft tree is depth-limited. The parallel dimension of the target model isγ, and asn→∞, the asymptotic speedup is S2 = 1−α γ 1 (1−α 1) +c 1 α1−αγ+1 1 −γ(1−α 1)αγ 1 . Proof. Overn stages, we compare the total number of sentences generated by our algorithm to ...

  33. [41]

    For each fixedα∈(0.5,0.8), the functionx7→a(α,x)is strictly decreasing on[1,∞)

  34. [42]

    For each fixedx≥1, the functionα7→a(α,x)is strictly increasing on(0.5,0.8)

  35. [43]

    Proof.(i) Monotonicity inx.Fixα∈(0,1)and write f(x) = xαx 1−α x = N(x) D(x), N(x) =xα x, D(x) = 1−α x

    Consequently, for everyx≥10andα∈(0.5,0.8), a(α,x)< a(0.8,10) =−ln(0.8) 10·0.8 10 1−0.8 10 ≈0.26, and for allα∈(0.52,0.8), a(α,2)∈ a(0.52,2), a(0.8,2) ≈(0.48,0.79). Proof.(i) Monotonicity inx.Fixα∈(0,1)and write f(x) = xαx 1−α x = N(x) D(x), N(x) =xα x, D(x) = 1−α x. Then N′(x)...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.