Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Kinetics: Rethinking Test-Time Scaling Laws

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper establishes the Kinetics Scaling Law: test-time compute is best spent on models above a critical size, and sparse attention is the mechanism that makes further scaling efficient.

desk verdict The block top-k sparse attention implementation is solid and worth using, but the headline Kinetics scaling law (including the 14B threshold) rests on a cost model that drops parameter-access under an infeasible large-batch assumption, so the central claim likely overreaches. read the letter →

arxiv 2506.05333 v3 pith:LNO5WASW submitted 2025-06-05 cs.LG cs.CL

Ranajoy Sadhukhan , Zhuoming Chen , Haizhong Zheng , Yang Zhou , Emma Strubell , Beidi Chen This is my paper
classification cs.LGcs.CL MSC 68T07
keywords test-timescalinglawsattentioncostmemoryaccesssparseKVcacheinferenceefficiencyreasoningmodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Test-time scaling (TTS), the practice of spending extra compute on long chains of thought or repeated sampling to solve hard problems, has usually been analyzed by counting floating-point operations. This paper argues that under realistic serving conditions memory access dominates, and specifically that attention cost, driven by key-value cache reads and attention arithmetic, dwarfs parameter computation once generations are long. The paper's Kinetics Scaling Law says that for a fixed budget, resources should go first to a larger model, empirically around 14B parameters for the Qwen3 family, before lengthening generations or adding samples. It then shows that sparse attention, by cutting the attention cost from quadratic to linear in generation length, lets smaller models re-enter the Pareto frontier and yields up to 60-point accuracy gains in low-cost regimes on AIME problems.

What carries the argument

The load-bearing object is the eFLOPs cost model, $C_{\mathrm{TTS}} = 2N P L_{\mathrm{out}} + 2rN L_{\mathrm{in}} D L_{\mathrm{out}} + rN D L_{\mathrm{out}}^2 + 2I L_{\mathrm{in}} D L_{\mathrm{out}} + I N D L_{\mathrm{out}}^2$, where $P$ is active parameters, $D$ is the KV dimension per token, $r$ is the grouped-query-attention ratio, $L_{\mathrm{out}}$ is generated tokens, and $I$ is the hardware arithmetic intensity. The paper uses this model to form the ratio $\Phi$ of attention-related cost to parameter-related cost, which exceeds 10--1000$\times$ for long generations. Sparse attention is then modeled by replacing the quadratic term with $L_{\mathrm{out}} B D$, and block top-$k$ attention is the tractable instantiation: it scores key blocks by their averaged key vectors and retrieves only the top $k$ blocks per query group, which makes the KV loading cost sub-quadratic in generation length.

What would settle it

Measure wall-clock solve rates on a single consumer GPU with batch size one, where parameter access is not amortized, and check whether small models with longer generations then win the cost-accuracy Pareto frontier; if they do, the Kinetics ordering collapses. A second check is to charge full search overhead to the oracle top-k attention curves and see whether the 60-point low-cost gains survive when retrieval cost is no longer zero.

Watch

Extended reading notes

Core claim

The central claim is that test-time scaling laws change character once inference cost is measured with memory access included. With generation lengths typical of long chains of thought, the cost of attention grows with the square of generated tokens, $L_{\mathrm{out}}^2 D$, while parameter computation grows only linearly in model size $P$. The Kinetics Scaling Law follows: for a fixed budget, Pareto-optimal accuracy is obtained by first scaling the model to a critical size (empirically around 14B for Qwen3 and 7B for DeepSeek-R1-distilled Qwen) and only then spending remaining budget on more generated tokens or more trials. A second claim is that replacing dense attention with sparse attention reshapes the law by turning the quadratic attention term into a linear one, so smaller models become effective again and problem-solving accuracy on AIME improves by over 60 points in low-cost regimes and over 5 points in high-cost regimes, including for mixture-of-experts models.

Load-bearing premise

The cost model assumes model parameter access is amortized away by large-batch serving, leaving attention computation and KV-cache access as the dominant costs; on small-batch or local hardware, parameter access can dominate and the paper's own limitations concede the conclusions may not extend there.

Editorial extensions

If this is right

  • Under a fixed inference budget, buy model capacity up to roughly 14B parameters before investing in longer chains of thought or more parallel samples.
  • Sparse attention should be treated as part of the test-time compute budget, and higher sparsity becomes more valuable as more compute is invested at test time.
  • Mixture-of-experts models reduce active parameter count but not attention cost, so they stand to benefit even more from sparse attention than dense models.
  • Simple block top-$k$ attention, which requires no oracle retrieval, reproduces most of the accuracy gains of oracle top-$k$ attention while being hardware-friendly.
  • Counting cost in eFLOPs, a memory-aware unit, can cut the resources needed to reach a given accuracy by up to $3\times$ compared with FLOPs-based planning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if Kinetics holds, task-level throughput, the number of problems solved per unit time and budget, should become the default comparison metric for reasoning systems; the paper gestures at this shift but does not fully build a metric out of it.
  • Editorial inference: any mechanism that shrinks KV traffic, such as KV quantization or latent attention, should compound with sparse attention, and the paper does not test those combinations jointly; a direct test would be applying sparse attention on top of an MLA-style model under the same eFLOPs budget.
  • Editorial inference: the critical threshold size is empirically tied to the KV-to-parameter ratio of a model family, which suggests a testable prediction that the threshold for a new model family can be estimated from that ratio before running the full Pareto sweeps.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Kinetics, a test-time scaling law based on a cost model that augments FLOPs with memory-access costs (eFLOPs). The authors argue that in long-CoT and Best-of-N inference, attention and KV-cache access dominate parameter-related costs, so FLOPs-only scaling laws overestimate small models. From Pareto-frontier analyses on Qwen3 and DeepSeek-R1-Distilled models across AIME24/25 and LiveCodeBench, they conclude that resources should first be spent on increasing model size up to a critical threshold (around 14B for Qwen3, 7B for DeepSeek-R1-Distilled) before investing in test-time strategies. They further propose sparse attention as a new scaling paradigm and provide a block top-k implementation with measured throughput improvements. The headline empirical claims are that sparse attention gives over 60-point accuracy gains in low-cost regimes and over 5-point gains in high-cost regimes.

Significance. If the cost model is accepted, the paper offers practically important guidance: memory access, not FLOPs, is the binding constraint in test-time scaling, and sparse attention is a promising lever. The paper's strengths include a broad empirical sweep across model families and benchmarks, a released codebase, and a real block top-k implementation with hardware measurements. The qualitative insight that KV-cache access matters more than parameter count in long generation is credible and useful. However, the quantitative 'law' rests on a parameter-access amortization assumption whose validity in the dense long-CoT regime is not demonstrated, the headline accuracy gains rely on oracle upper bounds, and the fitted scaling exponents are presented as laws despite being task-dependent. These issues are fixable with sensitivity analysis, memory-constrained costing, and clearer qualification of oracle results.

major comments (4)
  1. [Section 2, Eq. (1)-(4), Figure 3, Table 2] The cost model in Eq. (4) drops the 2P Lout parameter-access term from Eq. (2) after arguing that large batches amortize weight reads. The empirical support, Figure 3, uses batch size 4096. For Qwen3-32B, Table 2 reports 8GB KV cache per 32K-token sequence, so a batch of 4096 requires approximately 32TB of KV memory, far beyond an 8-GPU B200 node; the dense-attention baseline cannot operate at this batch size in the long-CoT regime. Since the deletion of parameter access is what makes attention dominate in Figure 2a and drives the 14B threshold in Figures 4-5, the authors should either justify the amortization with memory-feasible batch sizes for each model/context configuration or re-run the Pareto analysis with parameter access included under realistic memory constraints. Without this, the central 'spend on model size first' conclusion is not established for the dense models it compares.
  2. [Abstract and Section 4, Figures 1b, 8a, 8d] The abstract states that sparse attention achieves 'over 60 points gains' in low-cost regimes. These numbers come from oracle top-k attention combined with oracle Best-of-N (Section D.2), both upper bounds that are not deployable. The practical block top-k results in Section 5 (Figure 9a) show a 45-point gain and an 8.58x resource reduction, which is materially smaller. The paper should explicitly qualify the headline numbers as oracle upper bounds in the abstract and introduction, so readers are not misled about achievable performance.
  3. [Section 4.2, Figure 7, Appendix C.2] The claimed scaling relations (doubling cost increases optimal KV budget by 1.18x and generated tokens by 1.74x) are least-squares fits to empirical curves: y = 91.68·x^0.80 and y = 81.77·x^0.25 - 212. The exponents are task-dependent (AIME25: 0.74/0.17; LiveCodeBench: 0.92/0.19), the KV-budget fit includes an ad hoc subtractive constant, and no confidence intervals or goodness-of-fit are reported. Framing these fits as a 'Kinetics scaling law' overstates their status; they should be presented as empirical characterizations with appropriate uncertainty and caveats.
  4. [Section 3.1, Eq. (5), Figures 5a-5c] The Best-of-N Pareto analysis uses oracle selection (pass@N), which assumes access to the correct answer to choose among N samples; this is an upper bound on achievable inference-time strategies, not a deployable protocol. The threshold claims about models above and below 14B rely substantially on this oracle setting. While Long-CoT results are also provided, the paper should consistently label the Best-of-N threshold as an upper-bound analysis and state how the conclusion changes when a learned verifier or majority vote is used.
minor comments (4)
  1. [Figure 1b and Abstract] The Figure 1b caption says 'over 50-point improvements' while the abstract says 'over 60 points'; these numbers should be reconciled and both should state the oracle nature of the result.
  2. [Section 2 heading and Figure 4 caption] There are typos in the headings and captions, including 'T est-Time' in Section 2 and 'abfor' in the Figure 4 caption; the paper should be proofread.
  3. [Section 5.2] The text 'torch compile' should be 'torch.compile' to match the tool name.
  4. [Table 1 and Table 2] The notation D is described as 'KV size per token', but Table 2 reports total KV memory for 32K tokens; clarifying the relationship between D and the reported GB values would improve reproducibility.

Circularity Check

2 steps flagged · score 5.0 of 10

Partially circular: attention dominance is baked into the cost model by dropping parameter-access memory, and the 1.18x/1.74x doubling ratios restate fitted power-law exponents.

  1. self definitional [Section 2, Cost Model, Eqs. (1)-(4) and Analysis paragraph]
    "In real serving scenarios, a large batch size will be used (DeepSeek-AI, 2025) with growing GPU VRAM (Tirumala and Wong, 2024) and model parallelism (Pope et al., 2023). The access to the model parameters will be amortized across requests in a batch; Figure 3 shows parameter access time is negligible when the batch size is large. Thus, we only consider the second term (i.e., KV cache loading) in our cost function."

    The paper's 'key insight is attention-related cost dominates in long CoTs' is presented through Phi = [2rLinD + (rD + ID)Lout] / 2P, but this ratio is taken after Eq. (4) has already removed the 2P Lout parameter-access term from the memory cost via the large-batch amortization assumption. Attention dominance is therefore in part a restatement of the cost-model construction rather than an independent empirical discovery.

  2. fitted input called prediction [Section 3.2 / Figure 7 caption and Section 4.2]
    "Using Qwen3-8B as a representative model, we fit curves to characterize this tradeoff. For Best-of-N, we find that for every doubling of the total compute cost, the optimal KV budget increases by a factor of 1.18x, while the total number of generated tokens increases by 1.74x."

    The claimed 'finding' is the fitted curve itself: Figure 7 reports y = 91.68·x^0.80 for optimal generated tokens and y = 81.77·x^0.25 − 212 for optimal KV budget. Doubling x gives 2^0.80 ≈ 1.74x tokens and 2^0.25 ≈ 1.19x KV, exactly the 1.74x/1.18x values presented. The paper says it 'fit curves to characterize this tradeoff,' so the numbers are transparently fit-derived, but Section 4.2 then presents 'our analysis reveals a consistent trend' as if the doubling ratios were an independent scaling result. This is a fitted input elevated to a finding, not a prediction.

full rationale

The paper's central empirical content—the Pareto frontiers on AIME24/AIME25/LiveCodeBench and the 14B (Qwen3) / 7B (DeepSeek-distilled) critical-size observations—is measured on external benchmarks and is not itself derived from the cost model, so the work is not globally circular against its data. However, two load-bearing pieces reduce to their own inputs. First, the 'attention dominates' insight that motivates Kinetics and Sparse Kinetics is partially definitional: Eq. (4) deletes the 2P Lout parameter-access memory term on a large-batch amortization argument, and the ratio Phi used to 'show' attention dominance is then computed with only parameter compute in the denominator. The paper also measures a latency breakdown (Figure 3) supporting the same conclusion, so this is a partial, not total, reduction. Second, the 1.18x/1.74x doubling ratios in Figure 7 and Section 4.2 are arithmetic consequences of power-law fits to the same data, not independent predictions; the fitted exponents are simply restated. Self-citations (Yuan et al. 2024a; Sadhukhan et al. 2024) are methodological references and are not load-bearing uniqueness claims. The Limitations section concedes small-batch/local settings are out of scope, which narrows but does not remove the definitional character of the attention-dominance step: in the dense long-CoT regime the large batches that justify dropping parameter access are memory-infeasible (Table 2: Qwen3-32B has 8GB KV per 32K tokens; Figure 3 uses batch 4096), so the cost model's premise is only comfortably realized under sparse attention, the very intervention the paper recommends.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No new physical entities are proposed; the new named constructs (Kinetics Scaling Law, Sparse Kinetics, eFLOPs) are analytical frameworks, not entities. The free parameters are the fitted scaling-curve constants used to state the KV-budget versus token-generation tradeoff.

free parameters (2)
  • Scaling curve exponents and coefficients (Figure 7) = 91.68, 0.80; 81.77, 0.25, -212 (AIME24, Qwen3-8B)
    Used to claim optimal KV budget grows 1.18x and generated tokens 1.74x per doubling of eFLOPs; the -212 is a model-specific constant added to account for small-budget regime.
  • Additional fitted curves (Appendix C.2) = 213.58·x^0.74 and 140.92·x^0.17 - 212 (AIME25); 22.08·x^0.92 and 161.83·x^0.19 - 212 (LiveCodeBench)
    Same tradeoff analysis on other tasks; constants are fit to the authors' own empirical data.
assumptions (4)
  • domain assumption Generated length Lout is much larger than or proportional to prompt length Lin
    Invoked in Section 2 Analysis to justify shifting the bottleneck from linear parameter terms to quadratic attention terms; if Lin dominates, the cost model and conclusions change.
  • domain assumption In large-batch serving, model parameter access is fully amortized and can be neglected
    Used in Section 2 to reduce Cmem to KV cache loading only; acknowledged in Limitations as not applying to local or low-batch deployment.
  • domain assumption Additive compute-plus-memory cost model is appropriate (rather than max cost model)
    The paper uses Ccomp + Cmem·I in the main text; Footnote 1 and Section A.1 show the max cost model also supports the claims, but the additive choice affects quantitative conclusions.
  • ad hoc to paper Oracle top-k attention and oracle Best-of-N are legitimate proxies for achievable performance
    The main sparse-attention Pareto frontiers and headline gains use oracle top-k (perfect selection) and oracle Best-of-N (knowing correct answers), which are upper bounds, not implementable systems.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Kinetics: Rethinking Test-Time Scaling Laws." pith.science (2026). https://pith.science/paper/LNO5WASW

@misc{pith2026250605333,
  author       = {Pith},
  title        = {Pith review of: Kinetics: Rethinking Test-Time Scaling Laws},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LNO5WASW}},
  note         = {Machine review of arXiv:2506.05333}
}
abstract

We rethink test-time scaling laws from a practical efficiency perspective, revealing that the effectiveness of smaller models is significantly overestimated. Prior work, grounded in compute-optimality, overlooks critical memory access bottlenecks introduced by inference-time strategies (e.g., Best-of-$N$, long CoTs). Our holistic analysis, spanning models from 0.6B to 32B parameters, reveals a new Kinetics Scaling Law that better guides resource allocation by incorporating both computation and memory access costs. Kinetics Scaling Law suggests that test-time compute is more effective when used on models above a threshold than smaller ones. A key reason is that in TTS, attention, rather than parameter count, emerges as the dominant cost factor. Motivated by this, we propose a new scaling paradigm centered on sparse attention, which lowers per-token cost and enables longer generations and more parallel samples within the same resource budget. Empirically, we show that sparse attention models consistently outperform dense counterparts, achieving over 60 points gains in low-cost regimes and over 5 points gains in high-cost regimes for problem-solving accuracy on AIME, encompassing evaluations on state-of-the-art MoEs. These results suggest that sparse attention is essential and increasingly important with more computing invested, for realizing the full potential of test-time scaling where, unlike training, accuracy has yet to saturate as a function of computation, and continues to improve through increased generation. The code is available at https://github.com/Infini-AI-Lab/Kinetics.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Interpretable Adaptive Sampling for LLM Test-Time Scaling

    cs.AI 2026-08 reject novelty 4.0 of 10

    A fuzzy controller that allocates a per-prompt sampling budget keeps LLM accuracy near a fixed full-budget baseline while reducing the average number of candidate answers on some datasets.

Reference graph

Works this paper leans on

71 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [6]

    Generating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509,

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509,

    arXiv 1904

  2. [8]

    Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models.arXiv preprint arXiv:2401.06066,

    Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models.arXiv preprint arXiv:2401.06066,

    arXiv

  3. [9]

    Flashattention-2: Faster attention with better parallelism and work partitioning.CoRR, abs/2307.08691,

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning.CoRR, abs/2307.08691,

    arXiv

  4. [10]

    Tri Dao, Beidi Chen, Kaizhao Liang, Jiaming Yang, Zhao Song, Atri Rudra, and Christopher R´ e

    doi: 10.48550/ARXIV.2307.08691.https://doi.org/10.48550/arXiv.2307.08691. Tri Dao, Beidi Chen, Kaizhao Liang, Jiaming Yang, Zhao Song, Atri Rudra, and Christopher R´ e. Pixelated butterfly: Simple and efficient sparse training for neural network models. InInternational Conference on Learning Representations (ICLR), 2021.https://arxiv.org/abs/2112.00029. T...

    arXiv doi:10.48550/arxiv.2307.08691

  5. [12]

    Palm-e: An embodied multimodal language model.arXiv preprint arXiv:2303.03378,

    13 Danny Driess, Minh Nguyen, Fei Xia, et al. Palm-e: An embodied multimodal language model.arXiv preprint arXiv:2303.03378,

    arXiv

  6. [13]

    Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Dehao Chen, Yonghui Wu, and Jeff Dean

    Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Dehao Chen, Yonghui Wu, and Jeff Dean. Glam: Efficient scaling of language models with mixture-of-experts.arXiv preprint arXiv:2112.06905,

    arXiv

  7. [14]

    Alphazero-like tree-search can guide large language model decoding and training.arXiv preprint arXiv:2309.17179,

    Xidong Feng, Ziyu Wan, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero-like tree-search can guide large language model decoding and training.arXiv preprint arXiv:2309.17179,

    arXiv

  8. [15]

    Gptq: Accurate post-training quantization for generative pre-trained transformers.arXiv preprint arXiv:2210.17323,

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers.arXiv preprint arXiv:2210.17323,

    arXiv

Show all 71 references
  1. [16]

    Efficiently serving llm reasoning programs with certaindex.arXiv preprint arXiv:2412.20993,

    Yichao Fu, Junda Chen, Siqi Zhu, Zheyu Fu, Zhongdongming Dai, Aurick Qiao, and Hao Zhang. Efficiently serving llm reasoning programs with certaindex.arXiv preprint arXiv:2412.20993,

    arXiv

  2. [17]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Let- man, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    arXiv

  3. [18]

    Mamba: Linear-time sequence modeling with selective state spaces.CoRR, abs/2312.00752,

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces.CoRR, abs/2312.00752,

    arXiv

  4. [19]

    Albert Gu, Karan Goel, and Christopher R´ e

    doi: 10.48550/ARXIV.2312.00752.https://doi.org/10.48550/arXiv.2312.00752. Albert Gu, Karan Goel, and Christopher R´ e. Efficiently modeling long sequences with structured state spaces. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event,...

    doi arXiv

  5. [20]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al

    OpenReview.net, 2022.https://openreview.net/forum?id=uYLFoz1vlAC. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.ar...

    2022 arXiv

  6. [21]

    Training large language models to reason in a continuous latent space.arXiv preprint arXiv:2412.06769,

    Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space.arXiv preprint arXiv:2412.06769,

    arXiv

  7. [22]

    Fastdecode: High-throughput gpu-efficient llm serving using heterogeneous pipelines.arXiv preprint arXiv:2403.11421,

    Jiaao He and Jidong Zhai. Fastdecode: High-throughput gpu-efficient llm serving using heterogeneous pipelines.arXiv preprint arXiv:2403.11421,

    arXiv

  8. [23]

    Training compute-optimal large language models

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556,

    arXiv

  9. [24]

    Junhao Hu, Wenrui Huang, Weidong Wang, Zhenwen Li, Tiancheng Hu, Zhixia Liu, Xusheng Chen, Tao Xie, and Yizhou Shan

    https://proceedings.neurips.cc/ paper files/paper/2024/file/028fcbcf85435d39a40c4d61b42c99a4-Paper-Conference.pdf. Junhao Hu, Wenrui Huang, Weidong Wang, Zhenwen Li, Tiancheng Hu, Zhixia Liu, Xusheng Chen, Tao Xie, and Yizhou Shan. Efficient long-decoding inference with reason...

    2024 arXiv

  10. [25]

    Language models as zero-shot planners: Extracting actionable knowledge for embodied agents.arXiv preprint arXiv:2201.07207,

    Wenlong Huang, Fei Fei, and Chelsea Finn. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents.arXiv preprint arXiv:2201.07207,

    arXiv

  11. [26]

    Openai o1 system card.arXiv preprint arXiv:2412.16720,

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720,

    arXiv

  12. [27]

    Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974,

    14 Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974,

    arXiv

  13. [28]

    Mixtral of experts.arXiv preprint arXiv:2401.04088,

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts.arXiv preprint arXiv:2401.04088,

    arXiv

  14. [29]

    Hydragen: High-throughput llm inference with shared prefixes.arXiv preprint arXiv:2402.05099,

    Jordan Juravsky, Bradley Brown, Ryan Ehrlich, Daniel Y Fu, Christopher R´ e, and Azalia Mirhoseini. Hydragen: High-throughput llm inference with shared prefixes.arXiv preprint arXiv:2402.05099,

    arXiv

  15. [30]

    Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,

    2001 arXiv

  16. [31]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Fran¸ cois Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. InProceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume...

    2020

  17. [32]

    Scaling laws for precision.arXiv preprint arXiv:2411.04330,

    Tanishq Kumar, Zachary Ankner, Benjamin F Spector, Blake Bordelon, Niklas Muennighoff, Mansheej Paul, Cengiz Pehlevan, Christopher R´ e, and Aditi Raghunathan. Scaling laws for precision.arXiv preprint arXiv:2411.04330,

    arXiv

  18. [33]

    Yaniv Leviathan, Matan Kalman, and Yossi Matias

    https://arxiv.org/abs/2309.06180. Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274–19286. PMLR,

    arXiv

  19. [34]

    https://arxiv.org/abs/ 2404.14469. Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, Omri Abend, Raz Alon, Tomer Asida, Amir Bergman, Roman Glozman, Michael Gokhman, Avashalom Ma...

    2024 arXiv

  20. [35]

    Awq: Activation-aware weight quantization for on-device llm compression and acceleration

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87–100,...

    arXiv

  21. [36]

    Kivi: A tuning-free asymmetric 2bit quantization for kv cache.arXiv preprint arXiv:2402.02750, 2024c

    Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache.arXiv preprint arXiv:2402.02750, 2024c. Wenjie Ma, Jingxuan He, Charlie Snell, Tyler Griggs, Sewon Min, and...

    2024 arXiv

  22. [37]

    https:// artofproblemsolving.com/wiki/index.php/American Invitational Mathematics Examination?srsltid= AfmBOoqiDCiaGTLQrsRTKsZui8RFnjOZqM4qIqY3yGB3sBaqOaxwf Xt. MAA. American invitational mathematics examination 2025,

    2025

  23. [38]

    Pierre-Emmanuel Mazar´ e, Gergely Szilvasy, Maria Lomeli, Francisco Massa, Naila Murray, Herv´ e J´ egou, and Matthijs Douze

    https:// artofproblemsolving.com/wiki/index.php/American Invitational Mathematics Examination?srsltid= AfmBOoqiDCiaGTLQrsRTKsZui8RFnjOZqM4qIqY3yGB3sBaqOaxwf Xt. Pierre-Emmanuel Mazar´ e, Gergely Szilvasy, Maria Lomeli, Francisco Massa, Naila Murray, Herv´ e J´ egou, and Matthi...

    arXiv

  24. [39]

    Specinfer: Accelerating generative large language model serving with tree-based speculative inference and verification.arXiv preprint arXiv:2305.09781,

    Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, et al. Specinfer: Accelerating generative large language model serving with tree-based speculative inference and verification.arXiv prep...

    arXiv

  25. [40]

    Accelerating sparse deep neural networks.arXiv preprint arXiv:2104.08378,

    Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks.arXiv preprint arXiv:2104.08378,

    arXiv

  26. [41]

    Cotformer: A chain-of-thought driven architecture with budget-adaptive computation cost at inference.arXiv preprint arXiv:2310.10845,

    Amirkeivan Mohtashami, Matteo Pagliardini, and Martin Jaggi. Cotformer: A chain-of-thought driven architecture with budget-adaptive computation cost at inference.arXiv preprint arXiv:2310.10845,

    arXiv

  27. [43]

    Webgpt: Browser-assisted question-answering with human feedback

    Reiichiro Nakano, Jacob Hilton, Jeffrey Wu, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332,

    arXiv

  28. [44]

    The sparse frontier: Sparse attention trade-offs in transformer llms.arXiv preprint arXiv:2504.17768,

    Piotr Nawrot, Robert Li, Renjie Huang, Sebastian Ruder, Kelly Marchisio, and Edoardo M Ponti. The sparse frontier: Sparse attention trade-offs in transformer llms.arXiv preprint arXiv:2504.17768,

    arXiv

  29. [45]

    Skeleton-of-thought: Prompting llms for efficient parallel generation.arXiv preprint arXiv:2307.15337,

    Xuefei Ning, Zinan Lin, Zixuan Zhou, Zifu Wang, Huazhong Yang, and Yu Wang. Skeleton-of-thought: Prompting llms for efficient parallel generation.arXiv preprint arXiv:2307.15337,

    arXiv

  30. [46]

    Daniele Paliotta, Junxiong Wang, Matteo Pagliardini, Kevin Y

    Accessed: 2025-01-09. Daniele Paliotta, Junxiong Wang, Matteo Pagliardini, Kevin Y. Li, Aviv Bick, J. Zico Kolter, Albert Gu, Fran¸ cois Fleuret, and Tri Dao. Thinking slow, fast: Scaling inference compute with distilled reasoners,

    2025

  31. [47]

    org/abs/2502.20339

    https://arxiv. org/abs/2502.20339. Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference.Proceedings of Machine Learning and Systems, 5:606–624,

    arXiv

  32. [48]

    Ranajoy Sadhukhan, Jian Chen, Zhuoming Chen, Vashisth Tiwari, Ruihang Lai, Jinyuan Shi, Ian En-Hsu Yen, Avner May, Tianqi Chen, and Beidi Chen

    https://qwenlm.github.io/blog/ qwq-32b/. Ranajoy Sadhukhan, Jian Chen, Zhuoming Chen, Vashisth Tiwari, Ruihang Lai, Jinyuan Shi, Ian En-Hsu Yen, Avner May, Tianqi Chen, and Beidi Chen. Magicdec: Breaking the latency-throughput tradeoff for long context generation with speculat...

    arXiv

  33. [49]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538,

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538,

    arXiv

  34. [50]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314,

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314,

    arXiv

  35. [51]

    16 Hanshi Sun, Li-Wen Chang, Wenlei Bao, Size Zheng, Ningxin Zheng, Xin Liu, Harry Dong, Yuejie Chi, and Beidi Chen

    Apache 2.0 License. 16 Hanshi Sun, Li-Wen Chang, Wenlei Bao, Size Zheng, Ningxin Zheng, Xin Liu, Harry Dong, Yuejie Chi, and Beidi Chen. Shadowkv: Kv cache in shadows for high-throughput long-context llm inference.arXiv preprint arXiv:2410.21465, 2024a. Hanshi Sun, Zhuoming Ch...

    arXiv

  36. [52]

    Llm pretraining with continuous concepts.arXiv preprint arXiv:2502.08524,

    Jihoon Tack, Jack Lanchantin, Jane Yu, Andrew Cohen, Ilia Kulikov, Janice Lan, Shibo Hao, Yuandong Tian, Jason Weston, and Xian Li. Llm pretraining with continuous concepts.arXiv preprint arXiv:2502.08524,

    arXiv

  37. [53]

    Quest: Query-aware sparsity for efficient long-context llm inference.arXiv preprint arXiv:2406.10774,

    Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context llm inference.arXiv preprint arXiv:2406.10774,

    arXiv

  38. [55]

    Rush, and Tri Dao

    Junxiong Wang, Wen-Ding Li, Daniele Paliotta, Daniel Ritter, Alexander M. Rush, and Tri Dao. M1: Towards scalable test-time compute with mamba reasoning models, 2025.https://arxiv.org/abs/2504.10449. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aaka...

    2025 arXiv

  39. [56]

    Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models.arXiv preprint arXiv:2408.00724,

    Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models.arXiv preprint arXiv:2408.00724,

    arXiv

  40. [57]

    Efficient streaming language models with attention sinks, 2024.https://arxiv.org/abs/2309.17453

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks, 2024.https://arxiv.org/abs/2309.17453. Ruyi Xu, Guangxuan Xiao, Haofeng Huang, Junxian Guo, and Song Han. Xattention: Block sparse attention with anti...

    2024 arXiv

  41. [58]

    Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024a

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng...

    arXiv

  42. [59]

    Shinn Yao, Jiaming Zhao, Dian Yu, et al

    https://arxiv.org/abs/2505.09388. Shinn Yao, Jiaming Zhao, Dian Yu, et al. React: Synergizing reasoning and acting in language models.Advances in Neural Information Processing Systems (NeurIPS), 2023a. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, ...

    2025 arXiv

  43. [60]

    ISBN 978-1-939133-28-1

    USENIX Association. ISBN 978-1-939133-28-1. https://www.usenix.org/conference/osdi22/presentation/yu. Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, YX Wei, Lean Wang, Zhiping Xiao, et al. Native sparse attention: Hardware-aligned and ...

    arXiv

  44. [61]

    Llm inference unveiled: Survey and roofline model insights, 2024a.https://arxiv.org/abs/2402.16363

    Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Zhe Zhou, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu, Yong Jae Lee, Yan Yan, Beidi Chen, Guangyu Sun, and Kurt Keutzer. Llm inference unveiled: Survey and roofline model insights, 2024a.https://arxiv.org/abs/2402.16363. Zhi...

    arXiv

  45. [62]

    Opt: Open pre-trained transformer language models.arXiv preprint arXiv:2205.01068,

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models.arXiv preprint arXiv:2205.01068,

    arXiv

  46. [63]

    Nanoflow: Towards optimal large language model serving throughput.arXiv preprint arXiv:2408.12757,

    Kan Zhu, Yilong Zhao, Liangyu Zhao, Gefei Zuo, Yile Gu, Dedong Xie, Yufei Gao, Qinyu Xu, Tian Tang, Zihao Ye, et al. Nanoflow: Towards optimal large language model serving throughput.arXiv preprint arXiv:2408.12757,

    arXiv

  47. [64]

    For the Qwen3 series, this additional overhead is bounded by 3.57% for the 0.6B model and by 1.56% for the 32B model

    Many sparse attention algorithms skip the first layer (Tang et al., 2024; Chen et al., 2024; Zhang et al., 2023), resulting in only a minor increase in total cost. For the Qwen3 series, this additional overhead is bounded by 3.57% for the 0.6B model and by 1.56% for the 32B mo...

    2024

  48. [65]

    LiveCodeBench features complex programming problems from recent coding contests, while AIME25 consists of challenging math problems

    and AIME25 (MAA, 2025). LiveCodeBench features complex programming problems from recent coding contests, while AIME25 consists of challenging math problems. In both cases, sparse attention—particularly oracle top- k—consistently outperforms dense attention. Block top-kattentio...

    2024

  49. [66]

    In parallel, approaches like FlashAttention (Dao et al., 2022; Dao,

    has been comprehensively studied to reduce the attention cost when processing long sequeces. In parallel, approaches like FlashAttention (Dao et al., 2022; Dao,

    2022

  50. [67]

    accelerate attention by maximizing hardware efficiency. To address the quadratic complexity of standard attention, researchers have also explored linear attention architectures (Gu and Dao, 2023; Gu et al., 2022; Katharopoulos et al., 2020; Choromanski et al., 2020). Additiona...

    2023

  51. [68]

    Our analysis builds on the practical designs and implementations of these systems

    are widely adopted to enhance the efficiency of LLM serving. Our analysis builds on the practical designs and implementations of these systems. In parallel, speculative decoding (Leviathan et al., 2023; Chen et al., 2023; Miao et al., 2023; Sadhukhan et al.,

    2023

  52. [69]

    Additionally, model compression and offloading (Dettmers et al., 2022; Lin et al., 29 2024a; Svirschevski et al., 2024; Sheng et al., 2023; Frantar et al.,

    has been proposed to mitigate the memory-bandwidth bottleneck during LLM decoding. Additionally, model compression and offloading (Dettmers et al., 2022; Lin et al., 29 2024a; Svirschevski et al., 2024; Sheng et al., 2023; Frantar et al.,

    2022

  53. [70]

    techniques are playing a crucial role in democratizing LLM deployment. Efficient T est-time Strategies.Optimizing reasoning models to generate fewer tokens has been shown to directly reduce inference-time cost (NovaSky-Team, 2025; Arora and Zanette; Ma et al., 2025b). Recent w...

    2025

  54. [71]

    Efficient reward-model-based (Wu et al., 2024; Snell et al., 2024; Sun et al., 2024c) test-time scaling algorithms are also comprehensively studied

    further enhance efficiency by adaptively allocating computational resources across tokens. Efficient reward-model-based (Wu et al., 2024; Snell et al., 2024; Sun et al., 2024c) test-time scaling algorithms are also comprehensively studied. 30

    2024

  55. [1996]

    Nvidia blackwell platform: Advancing generative ai and accelerated computing

    Ajay Tirumala and Raymond Wong. Nvidia blackwell platform: Advancing generative ai and accelerated computing. In 2024 IEEE Hot Chips 36 Symposium (HCS), pages 1–33. IEEE Computer Society,

    2024

  56. [2017]

    s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393,

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Cand` es, and Tatsunori Hashimoto. s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393,

    arXiv

  57. [2019]

    Rethinking attention with performers.arXiv preprint arXiv:2009.14794,

    Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers.arXiv preprint arXiv:2009.14794,

    2009 arXiv

  58. [2020]

    Large language monkeys: Scaling inference compute with repeated sampling.arXiv preprint arXiv:2407.21787,

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher R´ e, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling.arXiv preprint arXiv:2407.21787,

    arXiv

  59. [2021]

    Accelerating large language model decoding with speculative sampling.arXiv preprint arXiv:2302.01318,

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling.arXiv preprint arXiv:2302.01318,

    arXiv

  60. [2022]

    Smyrf: Efficient attention using asymmetric clustering.arXiv preprint arXiv:2010.05315,

    Giannis Daras, Nikita Kitaev, Augustus Odena, and Alexandros G Dimakis. Smyrf: Efficient attention using asymmetric clustering.arXiv preprint arXiv:2010.05315,

    2010 arXiv

  61. [2023]

    Reinforcement learning for long-horizon interactive llm agents.arXiv preprint arXiv:2502.01600, 2025a

    Kevin Chen, Marco Cusumano-Towner, Brody Huval, Aleksei Petrenko, Jackson Hamburger, Vladlen Koltun, and Philipp Kr¨ ahenb¨ uhl. Reinforcement learning for long-horizon interactive llm agents.arXiv preprint arXiv:2502.01600, 2025a. Mouxiang Chen, Binyuan Hui, Zeyu Cui, Jiaxi Y...

    arXiv

  62. [2024]

    Lococo: Dropping in convolutions for long context compression.arXiv preprint arXiv:2406.05317,

    Ruisi Cai, Yuandong Tian, Zhangyang Wang, and Beidi Chen. Lococo: Dropping in convolutions for long context compression.arXiv preprint arXiv:2406.05317,

    arXiv

  63. [2025]

    Daman Arora and Andrea Zanette

    https://github.com/meta-llama/llama-models/blob/main/models/llama4/ MODEL CARD.md. Daman Arora and Andrea Zanette. Training language models to reason efficiently, 2025.URL https://arxiv. org/abs/2502.04463. Edward Beeching, Lewis Tunstall, and Sasha Rush. Scaling test-time com...

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.