Pith. sign in

REVIEW 3 major objections 7 minor 37 references

Faster and Better LLMs via Latency-Aware Test-Time Scaling

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that on memory-bound hardware, test-time scaling should be budgeted by wall-clock latency rather than generated tokens, and that tuning parallel branches and speculative decoding finds configurations that are both faster…

desk verdict Latency-aware framing is real and useful, but the headline accuracy numbers are selected on the test set, so they predict the past rather than the future. read the letter →

arxiv 2505.19634 v4 pith:6RVCV2MF submitted 2025-05-26 cs.CL

Zili Wang , Tianyu Zhang , Haoli Bai , Lu Hou , Xianzhi Yu , Wulong Liu , Shiming Xiang , Lei Zhu This is my paper
classification cs.CL
keywords test-timescalinglatency-awareinferencespeculativedecodingbranch-wiseparallelismmemory-boundmajorityvotinggreedysearchMATH-500
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that test-time scaling — spending extra inference compute to improve an LLM's answers — should be measured and optimized against wall-clock latency, not generated-token count, in latency-sensitive deployments. Prior work treats token efficiency (accuracy gained per token) as the objective, but the paper shows that a compute-optimal strategy can be slower in real time than a parallel strategy that wastes more tokens yet uses otherwise idle memory bandwidth. The authors propose optimizing two kinds of generation concurrency — multiple parallel solution branches and speculative decoding with a chosen draft length — and give a greedy search that finds the best configuration cheaply. If the claim is right, a 32B model can reach 82.3% accuracy on the MATH-500 math benchmark within one minute, and a 3B model 72.4% within ten seconds.

What carries the argument

The machinery is generation concurrency, treated as two resource-competing dimensions. Branch-wise parallelism runs B independent solution branches and aggregates them by majority voting, adding accuracy almost without adding latency while decoding is memory-bound. Sequence-wise parallelism applies speculative decoding with draft length gamma, letting several tokens be verified in one forward pass and shifting the accuracy-latency curve left. A greedy search algorithm starts at B = 1, gamma = 0 and each step expands whichever dimension improves accuracy more, stopping when neither helps; this finds the latency-optimal configuration with far fewer evaluations than grid search.

What would settle it

Run the same latency-aware configurations on a compute-heavy accelerator or under a large server batch and measure per-token latency as branch count grows from 1 to 64; if latency rises proportionally with branches even at small batch size — or if a roofline measurement shows the decode workload already sits above the machine's ridge point — the premise that extra branches are nearly free fails and the latency-optimal results become hardware-specific artifacts.

Watch

Extended reading notes

Core claim

The central claim is that compute-optimal test-time scaling is not latency-optimal in memory-bound regimes: the strategy that maximizes accuracy per token can require more wall-clock time than a parallel strategy that generates more tokens but exploits otherwise idle memory bandwidth. Demonstrating this on s1.1-32B with MATH-500, the paper finds sequential scaling needs about 9x fewer tokens than parallel majority voting but 1.6x more time to reach the same accuracy. The authors then define latency-optimal TTS as the joint choice of parallel branches B and speculative draft length gamma, and show that a greedy search recovers the same configuration as exhaustive grid search with 8–10 evaluation steps instead of 56. The headline quantitative results are 82.3% accuracy for a 32B model within one minute and 72.4% for a 3B model within ten seconds on MATH-500.

Load-bearing premise

The load-bearing premise is that LLM decoding on the target hardware is memory-bound at small batch sizes, so extra parallel branches and speculative draft tokens add almost no latency; if the workload is compute-bound, the authors' own limitation note concedes that token-count budgets apply and the parallel-branch advantage disappears.

Editorial extensions

If this is right

  • On memory-bound hardware, a latency budget can change which TTS strategy is best: parallel majority voting beat sequential scaling by 1.6x in time to reach the same accuracy for s1.1-32B on MATH-500.
  • Tuning both branch count B and speculative draft length gamma together outperforms either alone: on s1.1-32B the latency-optimal configuration improved accuracy by up to 7.3 points and reduced latency by up to 1.7x versus the baseline.
  • The optimal configuration is model-dependent: reasoning models such as QwQ-32B get more from speculative decoding, while models not trained for long reasoning chains get more from extra branches.
  • Greedy search finds the same (B, gamma) as grid search using 8–10 evaluation steps instead of 56, so latency-optimal configurations can be identified cheaply.
  • When the workload becomes compute-bound, the paper's own analysis says the token-count budget is the right metric again, so latency-optimal TTS is specifically a small- and medium-scale deployment result.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the memory-bound premise holds broadly, then token-efficiency rankings reported on one hardware class may invert on another; benchmark suites should report latency-accuracy curves for the target deployment setting, not only tokens-per-accuracy.
  • The same concurrency-allocation view could be applied to other latency-sensitive LLM services such as interactive agents or edge assistants, where the greedy search could run at calibration time on the actual device.
  • A natural extension the authors do not develop is a hybrid controller that starts in latency-optimal parallel mode and switches to token-count budgeting as request concurrency grows and the workload becomes compute-bound.
  • Because confidence-based aggregation improved accuracy without changing latency, combining confidence scores with the latency-optimal search could push the same curves slightly higher.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper argues that test-time scaling (TTS) should be optimized under a wall-clock latency budget rather than a token-count budget, because LLM decoding is memory-bound at small batch sizes and therefore parallel branches and speculative decoding can increase throughput at little additional latency. The authors propose branch-wise parallelism and sequence-wise parallelism (speculative decoding) as two concurrency mechanisms, formulate latency-optimal TTS as a configuration search over branch count B and draft length γ, and present a greedy search algorithm to find (B, γ). Experiments on MATH-500, AIME24, AIME25, and GPQA-Diamond across several 3B--32B models show that parallel scaling can outperform sequential scaling within a fixed time budget while using more tokens, and that a jointly optimized configuration reaches 82.3% on MATH-500 for s1.1-32B within one minute and 72.4% for s1.1-3B within ten seconds. The paper also reports that greedy search reproduces grid-search configurations with fewer steps and includes ablations on aggregation strategies.

Significance. If the reported accuracy numbers were obtained by a method that selects its concurrency configuration without seeing the test answers, the paper would make a useful and falsifiable systems-level point: on memory-bound, small-batch deployments, latency, not token count, is the appropriate optimization target for test-time scaling. The roofline argument in Section 3.1 and Figure 3 is principled, and the fixed-configuration comparisons in Figures 1--3 and Table 1 are informative. The speculative-decoding-plus-majority-voting combination as a unified concurrency resource-allocation problem is a reasonable framing, and the paper is unusually explicit in its Limitations section that the memory-bound premise does not apply to large-scale server workloads. However, the headline 'latency-optimal' accuracies are not yet supported as deployable, expected-performance claims because the configuration search is evaluated on the same test benchmark used for the final numbers (Algorithm 1 and Table 2, with per-dataset configurations in Appendix A.2). This selection bias directly inflates the headline numbers and must be fixed before the central quantitative claims can be accepted.

major comments (3)
  1. [§3.3, Algorithm 1; §5.3, Table 2; Appendix A.2] The headline results 82.3% on MATH-500 and 72.4% within 10 seconds are chosen as the maximum of the search objective on the same benchmark on which they are reported. Algorithm 1 calls tts_task, an accuracy evaluation on the test set, and Table 2 reports the accuracy of the selected (B, γ) on that same set. Appendix A.2 lists per-dataset optimal configurations (e.g., B=16, γ=5 for s1.1-32B on MATH-500) selected using ground-truth answers. These are test-set-selected maxima, not the expected accuracy of a deployment that must choose its configuration without seeing the answers. This is load-bearing because the abstract and Section 5.1 present these numbers as achieved accuracies within a time budget. The authors should either report accuracy on a held-out set after selecting the configuration on a disjoint validation set, or explicitly re-label the reported numbers as upper bounds / selected-configuration ceilings and provide an unbiased estimate.
  2. [§3.3, Eq. (1)] Equation (1) defines the latency-optimal strategy as the argmax over θ of the expected indicator that the sampled answer equals the ground-truth y*(x). This makes the ground-truth answer part of the optimization objective, which is not a practical procedure. The greedy search in Algorithm 1 is a practical proxy, but its objective is still test-set accuracy. The paper should distinguish between (i) an oracle upper bound that knows ground-truth answers and (ii) a configurable policy that must be selected without them; the current presentation conflates the two, and the reported 82.3% and 72.4% figures belong to the former category as currently measured.
  3. [§5.3, Table 2] The claim that greedy search is 'effective' is supported only by showing that greedy recovers the same (B*, γ*) as grid search on the test set. This does not establish that the selected configuration transfers to unseen problems, nor does it provide an estimate of the accuracy penalty from using a validation-set selection. In addition, Table 2 reports no standard deviations or repeated runs, unlike Table 1, so the equality of greedy and grid accuracies may be within noise. The authors should add a validation/test split experiment and report confidence intervals for the selected configurations.
minor comments (7)
  1. [Table 1] The header 'Bnh-wise' is a typo for 'Branch-wise'; please correct it.
  2. [§4.5] The sentence 'Conversely, for LLMs hat exhibit improvements from branches' contains a typo: 'hat' should be 'that'.
  3. [Appendix A.2] The sentence 'The results of sequence-wise parallelism under different branch counts are shown in Figure 13..' has a double period; please fix the punctuation.
  4. [Figures 9 and 10] The caption label 'Latenct-Optimal scaling' is a typo for 'Latency-Optimal scaling'.
  5. [Appendix A.1] The claim that Qwen2.5-32B-Instruct has 64GB of weights should specify the precision assumption (presumably FP16/BF16); otherwise the number is confusing because 32B parameters at 2 bytes each is 64GB.
  6. [§5.2] The sentence 'latency-optimal TTS can achieve 6% accuracy improvement on average' is not directly derivable from Table 1, which reports improvements of 7.3, 2.5, 6.8, and 7.3 percentage points; please clarify whether '6%' refers to a different aggregation or is a typo.
  7. [Figures 7--10] The scaling curves are plotted without error bars or repeated-run variability, while Table 1 reports standard deviations; adding uncertainty information to the curves would make the comparisons more robust.

Circularity Check

1 steps flagged · score 6.0 of 10

The 82.3% and 72.4% headline accuracies are test-set-selected maxima: Algorithm 1 and Eq. (1) choose (B, gamma) by maximizing accuracy on MATH-500, and Table 2 reports that same maximized accuracy as the method's result.

  1. fitted input called prediction [Section 3.3 Eq. (1); Algorithm 1; Section 5.3 Table 2; Abstract]
    "The latency-optimal test-time scaling is given by: θ∗_{x,y∗(x)}(T) = arg max_θ E_{y∼Target(θ,T,x)} [1{y=y∗(x)}], (1), where y∗(x) indicates the groundtruth ... To validate the effectiveness of our proposed greedy search algorithm, extensive experiments are conducted across various models on MATH-500. ... Model Grid Search Greedy Search ... s1.1-32B(16,5)82.3% 56 (16,5)82.3% 10"

    Eq. (1) defines 'latency-optimal' as the argmax over θ of accuracy computed with ground-truth y*(x). Algorithm 1 selects (B,γ) by calling tts_task, an accuracy evaluation under a latency budget, on the same MATH-500 benchmark. Table 2 then reports the selected (16,5) as 82.3% and (32,5) as 72.4%, and the Abstract presents these as what latency-optimal TTS 'enables.' The reported accuracies are the maximized values of the search objective on the test set, not predictions of a configuration chosen without test answers. The greedy-vs-grid comparison only shows greedy recovers the test-set-optimal configuration, not transfer to unseen problems. The headline claims therefore reduce by construction to the test-set selection objective.

full rationale

The central empirical claim — that latency-optimal TTS reaches 82.3% on MATH-500 within 1 minute and 72.4% within 10 seconds — is compromised by the evaluation protocol. The 'latency-optimal configuration' is defined in Eq. (1) as the argmax over configurations of the probability of matching the ground-truth answer, and Algorithm 1 searches this objective on the same benchmark whose accuracy is later reported. Table 2 explicitly reports the accuracy of the selected configuration, so the headline numbers are the maxima of the search objective, not independent predictions. This is the fitted-input-called-prediction pattern. The memory-bound premise that motivates branch parallelism is separately supported by a roofline model and throughput measurements, so that part is not circular. The only self-citation (MoESD) is a peripheral related-work statement and is not load-bearing; no uniqueness theorem or imported ansatz appears. However, because the headline accuracies reduce by construction to test-set selection, the paper merits a 6 rather than a lower score. A further correctness risk beyond circularity is that the selected configurations may not transfer to unseen problems.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical entities. Its free parameters are the test-time concurrency settings chosen by search on the benchmark, and its key axioms are the memory-bound decoding premise and the dataset-specific prefill-negligibility assumption.

free parameters (2)
  • Concurrency configuration (B*, gamma*) = (16,5) for s1.1-32B on MATH-500
    Chosen by greedy search maximizing test-set accuracy; varies per dataset and model (Appendix A.2). This is the main tuned quantity in the paper.
  • Sequence length / budget forcing target = 1024 to 8192 tokens
    Varies as experimental knob in Table 1 and Figure 1; effective 'thinking length' per branch.
assumptions (4)
  • domain assumption LLM decoding is memory-bound at small batch sizes; adding parallel branches does not increase latency until compute-bound.
    Section 3.1 and Figure 3; the authors acknowledge in Limitations that this holds on small/medium hardware, not large servers.
  • domain assumption Prefill time is negligible for the chosen benchmark dataset because inputs are short.
    Appendix A.3: 88% of MATH-500 questions are under 360 tokens, responses up to 8192 tokens. This justifies using decode latency as total latency.
  • ad hoc to paper The greedy search over (B, gamma) finds the global optimum of the latency-accuracy trade-off.
    Algorithm 1 greedily expands one of two dimensions; the paper only validates against grid search on MATH-500 (Table 2) and provides no proof for other datasets.
  • domain assumption Majority voting is the right aggregation for low-latency parallel scaling.
    Section 2 cites Singhi et al. (2025) that majority voting beats verifier-based methods at low token budgets; Section 5.4 shows confidence-based aggregation can slightly improve results but is not used in the headline.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Faster and Better LLMs via Latency-Aware Test-Time Scaling." pith.science (2026). https://pith.science/paper/6RVCV2MF

@misc{pith2026250519634,
  author       = {Pith},
  title        = {Pith review of: Faster and Better LLMs via Latency-Aware Test-Time Scaling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6RVCV2MF}},
  note         = {Machine review of arXiv:2505.19634}
}
read the original abstract

Test-Time Scaling (TTS) has proven effective in improving the performance of Large Language Models (LLMs) during inference. However, existing research has overlooked the efficiency of TTS from a latency-sensitive perspective. Through a latency-aware evaluation of representative TTS methods, we demonstrate that a compute-optimal TTS does not always result in the lowest latency in scenarios where latency is critical. To address this gap and achieve latency-optimal TTS, we propose two key approaches by optimizing the concurrency configurations: (1) branch-wise parallelism, which leverages multiple concurrent inference branches, and (2) sequence-wise parallelism, enabled by speculative decoding. By integrating these two approaches and allocating computational resources properly to each, our latency-optimal TTS enables a 32B model to reach 82.3% accuracy on MATH-500 within 1 minute and a smaller 3B model to achieve 72.4% within 10 seconds. Our work emphasizes the importance of latency-aware TTS and demonstrates its ability to deliver both speed and accuracy in latency-sensitive scenarios.

Figures

Figures reproduced from arXiv: 2505.19634 by the authors.

Figure 1
Figure 1. “Compute-optimal” does not necessarily translate to “latency-optimal” for test-time scaling. Left: Previous works measure test-time scaling by #token budget, indicating sequential scaling shows superior token efficiency than parallel scaling (majority voting for instance). Right: When considering latency as budget, parallel scaling can be 1.6x faster to achieve the same accuracy than sequential scaling. Experiments … view at source ↗
Figure 2
Figure 2. Latency-aware test-time scaling on MATH-500 with different model types, with sequential scaling in red [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. System state, latency, and throughput of s1.1-32B. Left: The roofline model with s1.1-32B. Increasing computational demand shifts execution from memory-bound to compute-bound. Middle: Throughput scales linearly with batch size before saturating at peak FLOPS. Right: The latency per forward pass under varying batch sizes. better latency-performance trade-off. 2. We provide a unified view for parallel branches and spe… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Overview of how to improve TTS with latency budget. (a): default sequential scaling suffers from [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Latency-aware test-time scaling with different branches on s1.1-32B, LLama-3.1-8B-Instruct and s1.1-3B. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Latency-aware test-time scaling with specula [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Latency-aware test-time scaling curves on MATH-500 with sequential scaling in red, parallel scaling in [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Latency-aware test-time scaling curves of QwQ-32B and DeepSeek-R1-Distill-Qwen-32B under latency [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Latency-aware TTS curves of s1.1-32B on MATH-500, AIME24, AIME25 and GPQA-Diamond. [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Latency-aware TTS curves of s1.1-3B on MATH-500, AIME24, AIME25 and GPQA-Diamond. [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 11
Figure 11. Figure 11: More results of the influence of the number of branches from branch-wise parallelism. [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 12
Figure 12. Figure 12: More results of the influence of the number of branches from branch-wise parallelism under different [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]
Figure 13
Figure 13. Figure 13: More results of the influence of draft length from sequence-wise parallelism under different branch-wise [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

37 extracted references · 7 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Zachary Ankner, Rishab Parthasarathy, Aniruddha Nrusimha, Christopher Rinard, Jonathan Ragan-Kelley, and William Brandon. 2024. https://arxiv.org/abs/2402.05109 Hydra: Sequentially-dependent draft heads for medusa decoding . Preprint, arXiv:2402.05109

    arXiv 2024

  4. [4]

    AoPS. 2024. https://artofproblemsolving.com/wiki/index.php/2024_AIME_I,II Aime 2024 dataset

    work page 2024

  5. [5]

    AoPS. 2025. https://artofproblemsolving.com/wiki/index.php/2025_AIME_I,II Aime 2025 dataset

    work page 2025

  6. [6]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, and 29 others. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

    arXiv 2023

  7. [7]

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher R \'e , and Azalia Mirhoseini. 2024. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787

    arXiv 2024

  8. [8]

    Lee, Deming Chen, and Tri Dao

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. 2024. Medusa: Simple llm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv: 2401.10774

    arXiv 2024

Show all 37 references
  1. [9]

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318

    2023 arXiv

  2. [10]

    Sukmin Cho, Sangjin Choi, Taeho Hwang, Jeongyeon Seo, Soyeong Jeong, Huije Lee, Hoyun Song, Jong C Park, and Youngjin Kwon. 2025. Lossless acceleration of large language models with hierarchical drafting based on temporal locality in speculative decoding. arXiv preprint arXiv:...

    2025 arXiv

  3. [11]

    Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. 2025. rstar-math: Small llms can master math reasoning with self-evolved deep thinking. arXiv preprint arXiv:2501.04519

    2025 arXiv

  4. [12]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    2025 arXiv

  5. [13]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874

    2021 arXiv

  6. [14]

    Zongle Huang, Lei Zhu, Zongyuan Zhan, Ting Hu, Weikai Mao, Xianzhi Yu, Yongpan Liu, and Tianyu Zhang. 2025. Moesd: Unveil speculative decoding's potential for accelerating sparse moe. arXiv preprint arXiv:2505.19645

    2025

  7. [15]

    Kuang-Huei Lee, Ian Fischer, Yueh-Hua Wu, Dave Marwood, Shumeet Baluja, Dale Schuurmans, and Xinyun Chen. 2025. Evolving deeper llm thinking. arXiv preprint arXiv:2501.09891

    2025 arXiv

  8. [16]

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274--19286. PMLR

    2023

  9. [17]

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024 a . EAGLE-2 : Faster inference of language models with dynamic draft trees. In Empirical Methods in Natural Language Processing

    2024

  10. [18]

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024 b . EAGLE : Speculative sampling requires rethinking feature uncertainty. In International Conference on Machine Learning

    2024

  11. [19]

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2025. https://arxiv.org/abs/2503.01840 EAGLE-3 : Scaling up inference acceleration of large language models via training-time test . Preprint, arXiv:2503.01840

    2025 arXiv

  12. [20]

    Runze Liu, Junqi Gao, Jian Zhao, Kaiyan Zhang, Xiu Li, Biqing Qi, Wanli Ouyang, and Bowen Zhou. 2025. Can 1b llm surpass 405b llm? rethinking compute-optimal test-time scaling. arXiv preprint arXiv:2502.06703

    2025 arXiv

  13. [21]

    Meta. 2024. https://ai.meta.com/blog/meta-llama-3-1/ Introducing llama 3.1: Our most capable models to date

    2024

  14. [22]

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Cand \`e s, and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393

    2025 arXiv

  15. [23]

    Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, and 1 others. 2021. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2...

    2021 arXiv

  16. [24]

    September 2024

    OpenAI. September 2024. https://openai.com/index/learning-to-reason-with-llms/ Learning to reason with llms

    2024

  17. [25]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling

    2024

  18. [26]

    Amrith Setlur, Nived Rajaraman, Sergey Levine, and Aviral Kumar. 2025. Scaling test-time compute without verification or rl is suboptimal. arXiv preprint arXiv:2502.12118

    2025 arXiv

  19. [27]

    Wenlei Shi and Xing Jin. 2025. Heimdall: test-time scaling on the generative verification. arXiv preprint arXiv:2504.10337

    2025 arXiv

  20. [28]

    Nishad Singhi, Hritik Bansal, Arian Hosseini, Aditya Grover, Kai-Wei Chang, Marcus Rohrbach, and Anna Rohrbach. 2025. When to solve, when to verify: Compute-optimal problem solving and generative verification for llm reasoning. arXiv preprint arXiv:2504.01005

    2025

  21. [29]

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314

    2024 arXiv

  22. [30]

    Qwen Team. 2025. https://qwenlm.github.io/blog/qwq-32b/ Qwq-32b: Embracing the power of reinforcement learning

    2025

  23. [31]

    Jun Wang, Meng Fang, Ziyu Wan, Muning Wen, Jiachen Zhu, Anjie Liu, Ziqin Gong, Yan Song, Lei Chen, Lionel M Ni, and 1 others. 2024 a . Openr: An open source framework for advanced reasoning with large language models. arXiv preprint arXiv:2410.09671

    2024 arXiv

  24. [32]

    Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. 2024 b . Mixture-of-agents enhances large language model capabilities. arXiv preprint arXiv:2406.04692

    2024 arXiv

  25. [33]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171

    2022 arXiv

  26. [34]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 22 others. 2024. Qwen2.5 technical repo...

    2024 arXiv

  27. [35]

    Wenkai Yang, Shuming Ma, Yankai Lin, and Furu Wei. 2025. Towards thinking-optimal scaling of test-time compute for llm reasoning. arXiv preprint arXiv:2502.18080

    2025

  28. [36]

    Lefan Zhang, Xiaodan Wang, Yanhua Huang, and Ruiwen Xu. 2025. Learning harmonized representations for speculative sampling. In International Conference on Learning Representations

    2025

  29. [37]

    Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. 2024. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408.15240

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.