Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Scaling Inference-Efficient Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that adding a model-shape term to the Chinchilla scaling law lets designers pick wider, shallower language models that serve faster without losing accuracy.

desk verdict A useful empirical extension of Chinchilla scaling laws for inference efficiency, with a real released 1B model at 1.8x lower latency, but the scaling-law form is an ad hoc ansatz and the ranking validation rests on three data points. read the letter →

arxiv 2501.18107 v2 pith:SQCPJ3XW submitted 2025-01-30 cs.LG cs.AIcs.CL

Song Bian , Minghao Yan , Shivaram Venkataraman This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords inference-efficientscalinglawsmodelaspectratiolatency-accuracytradeoffChinchillalawwidershallowermodelsselectionovertrainingdownstreamtaskaccuracy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that model architecture — specifically the aspect ratio of hidden width to number of layers — belongs inside a scaling law for language models, alongside parameter count and training tokens. It proposes $L(N,D,R) = (E + A N^{-\alpha} + B D^{-\beta})(1 + \varepsilon R^\gamma)$ and shows this fits 63 trained models and predicts the loss of scaled-up 1B models better than the standard Chinchilla formula, especially when ranking candidate shapes. If the claim is right, practitioners can treat shape as a dial for inference latency: choose a wider, shallower model to cut serving cost at fixed accuracy, instead of treating parameter count as the only design lever. The concrete payoff is Morph-1B, which runs 1.8x faster than similar open models at the same downstream accuracy.

What carries the argument

The central object is the inference-efficient scaling law of Eq. (4), $L(N,D,R) = (E + A N^{-\alpha} + B D^{-\beta}) \cdot (1 + \varepsilon R^\gamma)$, with $R = d_{\text{model}}/n_{\text{layers}}$. The new machinery is the multiplicative shape factor $(1 + \varepsilon R^\gamma)$: it makes loss depend on architecture by a power law while keeping the Chinchilla form intact, and the paper sets $\alpha = \beta = \gamma$ to keep the fit tractable. The law is fitted with Levenberg-Marquardt to 63 models from 80M to 339M parameters trained on DCLM-Baseline data at 20N, 40N, and 160N tokens, and the fitted law is then used to rank candidate 1B shapes before training them.

What would settle it

Train two families of models at fixed $N$ and $D$ with widely separated aspect ratios (for example $R=64$ and $R=256$) across several sizes ($200$M, $500$M, $1$B, $3$B) and fit Eq. (4) with and without an interaction term $N^{-\delta} R^\gamma$; if the interaction term improves the fit, or if the fitted $\varepsilon$ varies systematically with size, the multiplicative-shape assumption is wrong. A cheaper check is to fit Eq. (4) separately on models with $N \le 200$M and $N > 200$M and see whether the two fitted $\varepsilon$ values agree within the fit uncertainty.

Watch

Extended reading notes

Core claim

The central claim is that the effect of model shape on training loss is captured by multiplying the Chinchilla loss by a factor $(1+\varepsilon R^\gamma)$, where $R = d_{\text{model}}/n_{\text{layers}}$ is the aspect ratio. With this term, the scaling law both predicts held-out loss values (relative error under 1.2%) and, more importantly for model selection, ranks candidate 1B architectures correctly (Spearman correlation 1.00) where the plain Chinchilla law ranks them almost anti-correlated (-0.40). The paper argues that the ranking, not the absolute loss value, is what matters when choosing a model to train, because models with very similar loss can differ in downstream accuracy.

Load-bearing premise

The load-bearing premise is that the whole effect of model shape on loss is a single multiplicative power-law factor $(1+\varepsilon R^\gamma)$ whose exponent equals the parameter and token exponents, with no interaction between shape and model size or training duration.

Editorial extensions

If this is right

  • Designers can use the scaling law to pick the aspect ratio at fixed parameter and token budgets so that predicted loss stays within tolerance while inference latency drops.
  • Wider and shallower models consistently improve throughput and time-to-first-token across batch sizes, hardware (A100 and A30), and inference systems (Hugging Face and vLLM).
  • Including over-training data is essential for fitting: excluding the 160N data points degrades prediction quality, so the law is calibrated in the over-trained regime that matches real deployment.
  • The predict-and-rank methodology extends to other architectural changes such as grouped-query attention or multi-head latent attention, giving a way to quantify their accuracy-efficiency tradeoffs.
  • With only 6 data points and 85 A100 GPU-hours for fitting, the inference-efficient law can still predict losses of scaled-up models, cutting the cost of building scaling laws.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the multiplicative shape factor implies a computable 'shape tax' — if $\varepsilon > 0$, loss rises with $R$ regardless of $N$ and $D$ — so the optimal aspect ratio for a latency budget could be derived without retraining, but only if the shared-exponent assumption survives stress tests.
  • Beyond the paper: because the law is fit up to 339M parameters and validated at 1B, a natural next test is to train a 3B wide-shallow variant and a 3B narrow-deep variant and check whether the predicted loss ranking still holds at that scale.
  • Beyond the paper: the 1.8x latency gain of Morph-1B comes from shape alone at fixed tokens; the method should be roughly orthogonal to quantization and speculative decoding, so combining all three could compound the efficiency gain.
  • Beyond the paper: the paper leaves the irreducible-loss term $E$ independent of shape; a variant where $E$ also depends on $R$ would change predictions for very large models, and the two forms could be compared on the released data.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper extends the Chinchilla scaling law to include model shape, measured by the aspect ratio R = d_model / n_layers. It proposes L(N,D,R) = (E + A N^{-alpha} + B D^{-beta})(1 + epsilon R^gamma), fits this law on 63 small models (80M-339M parameters, up to 12.8B tokens), and uses predicted loss rankings to select 1B-scale candidates for training. The selected Morph-1B (d_model = 3072, 12 layers) is reported to reduce inference latency by 1.8x relative to Open-LM-1B while matching its average downstream accuracy, with all three 1B variants trained on the same 30B-token DCLM-Baseline split. The paper also ablates the fitting procedure by excluding over-training data and randomly selecting model shapes, and reports that the inference-efficient law predicts 1B losses within 1.2% and ranks the three 1B variants with Spearman 1.00 versus Chinchilla's -0.40.

Significance. The significance is conditional on validating the shape term. The empirical contribution is solid: 63 trained models, latency measurements across multiple GPUs and serving systems, and open-sourced training code and checkpoints. The controlled comparison among Morph-1B variants (same data, same token budget) directly supports the practical claim that wider, shallower models can be faster at equal accuracy. The scaling-law extension itself is an ansatz rather than a derivation, and the out-of-distribution ranking evidence is thin (three 1B models), so the general claim that the law ranks architectures reliably is not yet established. If the authors add alternative-form comparisons and broader held-out validation, the paper would be a useful step toward inference-aware model design. The 1B losses are genuinely out-of-sample, so the comparison is not circular by construction, but the functional form and the small number of test points are the main risks to the methodological claims.

major comments (3)
  1. [Section 2.2, Eq. (4)] The shape dependence is introduced as the multiplicative factor (1 + epsilon R^gamma), and Section 3.2 further constrains alpha = beta = gamma 'to simplify the fitting procedure.' This is the only mechanism by which the scaling law ranks architectures, so the functional form is load-bearing for the paper's central methodological claim. The paper motivates the form by visual inspection of Figure 5 and does not compare it with alternatives such as an additive shape term, a separate gamma exponent, or an interaction term R^gamma N^{-delta}. With the fitted values in Table 7 (epsilon = 0.0011, gamma = 0.61), the shape term changes predicted loss by only a few percent over the fitted R range, so a modest mis-specification could reorder near-equal candidate shapes and change the architecture selection. Please provide a model-comparison experiment on held-out data, for example fitting several candidate forms on the small-model data and scoring them on the 1B variants, or at least a residual analysis of Eq. (4) against N, D, and R. Reporting confidence intervals or standard errors for the fitted parameters would also help assess whether the shape term is statistically distinguishable from zero.
  2. [Section 4.1, Figure 8(c)] The headline ranking result is based on exactly three held-out 1B variants. Spearman's rho = 1.00 with n = 3 is not evidence of a reliable ordering: any strictly monotone ordering of three points yields rho = 1.00, and a single point can change the comparison against Chinchilla's -0.40. Since the paper explicitly argues that ranking is more important than absolute loss, this evidence is too thin to support the claimed ranking advantage. Please report the individual predicted and actual losses for the three 1B models, provide a confidence interval or permutation test for the Spearman comparison, and ideally validate on additional held-out shapes and sizes.
  3. [Section 4.1 and Section 4.3] The out-of-distribution validation consists of 1B models trained on 30B tokens, while the fit uses N <= 339M and D <= 12.8B, apart from the single 80M model at D = 160N. The claim that the law 'accurately predicts the loss of scaled-up models' therefore rests on three models at one scale and one token budget. Section 4.3 shows that removing the single over-trained 80M point degrades the fit substantially (MSE 0.1165, R2 0.6293 in Figure 9), so the extrapolation to D = 30B depends heavily on one anchor. The authors acknowledge in Section 6 that they cannot scale to 7B, but intermediate-scale validation (for example 500M-800M models at two or more token budgets) would substantially strengthen the predictive claim. I view this as a strengthening request rather than a demonstration of an error, but it is load-bearing for the general methodology.
minor comments (6)
  1. [Section 4.2] The sentence 'the inference latency for Morph-1B-v1 is 1.8x lower compared to Morph-1B' appears to have the comparison reversed; based on Table 2, Morph-1B is 1.8x faster than Morph-1B-v1.
  2. [Section 4.2 vs. Section 3.1] Section 4.2 says the trained models are evaluated 'over 20 downstream tasks,' but Section 3.1 lists 11 datasets and the Figure 1 caption says 11 downstream tasks; please reconcile the number and provide the full list.
  3. [Section 7] The conclusion contains a garbled sentence: 'We conduct count number, each point is a number experiments' should be rewritten.
  4. [Table 7] The text states that alpha, beta, and gamma are set equal, but the table reports only alpha and epsilon; please make the equality explicit in the table or report the values of beta and gamma.
  5. [Table 2] The comparison with open-source baselines would be easier to interpret if the training token counts for Open-LM-1B, OPT-1.3B, Pythia-1.3B, and Neox-1.3B were listed, since token budget is a known confound in accuracy comparisons.
  6. [Section 3.1] The paper does not report the variance or number of repeats of the latency measurements; because latency can be noisy, a short statement about measurement stability would strengthen the empirical claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the scaling law is an empirical fit evaluated on out-of-sample 1B models, and Morph-1B is actually trained and measured.

full rationale

The paper's central derivation is an empirical scaling-law fit, not a circular one. Eq. (4) introduces a shape term (1 + epsilon R^gamma) as a multiplicative factor on the Chinchilla loss; the functional form is an ansatz chosen after inspecting Figure 5 ('From the figure, we can see that the most suitable model shape adjustment is the inclusion of the term (1+epsilon R^gamma)'), but the parameters A, B, E, alpha, epsilon are fitted to small models (80M-339M) and the resulting law is then evaluated on 1B models that were not used in the fit. The 1B loss predictions in Section 4.1 are therefore out-of-sample, and the Morph-1B result is verified by actually training and evaluating the model on downstream tasks, so the headline 'wider and shallower models can yield efficiency gains while preserving accuracy' does not reduce by construction to the fitting inputs. The only mildly self-referential element is that the same law is used to select candidate shapes (Morph-1B variants) which then serve as test points for the law's ranking; this creates a selection-bias concern about the reported Spearman correlations on three 1B models, but it does not make the measured losses or accuracies equal to the model's predictions by definition. There are no load-bearing self-citations, no imported uniqueness theorems, and no ansatz smuggled in via citation. Accordingly, no circular step can be exhibited under the required standard.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central methodology is an empirical fit: Eq. (4) has five fitted parameters (A, B, E, alpha=beta=gamma, epsilon) and no first-principles derivation. The paper relies on the Chinchilla base form, the multiplicative shape ansatz, and standard scaling-law extrapolation. No new physical entities are introduced.

free parameters (5)
  • A = 54754.14 (Figure 8 fit)
    Scale coefficient for the N^{-alpha} term in the Chinchilla base, fitted to losses of 63 small models.
  • B = 778340.38 (Figure 8 fit)
    Scale coefficient for the D^{-beta} term, fitted to the same data.
  • E = 2.45 (Figure 8 fit)
    Irreducible loss floor, fitted.
  • alpha (and beta, gamma) = 0.61 (Figure 8 fit)
    Exponent shared by N, D, and R after the paper sets alpha=beta=gamma to simplify fitting.
  • epsilon = 0.0011 (Figure 8 fit)
    Coefficient of the aspect-ratio factor (1+epsilon R^gamma); fitted.
assumptions (5)
  • domain assumption Chinchilla loss form L(N,D)=E+AN^{-alpha}+BD^{-beta} is valid for these decoder-only LMs.
    Adopted from Hoffmann et al. (2022) without re-derivation; used as the base of Eq. (2) and Eq. (4).
  • domain assumption Smoothed training loss approximates test loss.
    Following Chinchilla, the paper uses smoothed training loss to fit the scaling laws.
  • ad hoc to paper Aspect ratio modifies loss through a multiplicative power-law factor (1+epsilon R^gamma).
    This functional form is an ansatz motivated by Figure 5; it is not derived from architecture or optimization theory.
  • domain assumption Scaling laws fitted at 80M-339M parameters and up to 12.8B tokens extrapolate to 1B parameters and 30B tokens.
    The 1B loss predictions in Section 4.1 rely on this extrapolation.
  • domain assumption Inference latency at batch size 1 on a single A100 is a representative proxy for deployment latency.
    The latency measurements and the recommendation of wider/shallower shapes are based on this setup, with only limited checks on A30, vLLM, and different token counts.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scaling Inference-Efficient Language Models." pith.science (2026). https://pith.science/paper/SQCPJ3XW

@misc{pith2026250118107,
  author       = {Pith},
  title        = {Pith review of: Scaling Inference-Efficient Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SQCPJ3XW}},
  note         = {Machine review of arXiv:2501.18107}
}
read the original abstract

Scaling laws are powerful tools to predict the performance of large language models. However, current scaling laws fall short of accounting for inference costs. In this work, we first show that model architecture affects inference latency, where models of the same size can have up to 3.5x difference in latency. To tackle this challenge, we modify the Chinchilla scaling laws to co-optimize the model parameter count, the number of training tokens, and the model architecture. Due to the reason that models of similar training loss exhibit gaps in downstream evaluation, we also propose a novel method to train inference-efficient models based on the revised scaling laws. We perform extensive empirical studies to fit and evaluate our inference-aware scaling laws. We vary model parameters from 80M to 1B, training tokens from 1.6B to 30B, and model shapes, training 63 models. Guided by our inference-efficient scaling law and model selection method, we release the Morph-1B model, which improves inference latency by 1.8x while maintaining accuracy on downstream tasks compared to open-source models, pushing the Pareto frontier of accuracy-latency tradeoff. Notably, our experiments reveal that wider and shallower models can yield efficiency gains while preserving accuracy.

Figures

Figures reproduced from arXiv: 2501.18107 by the authors.

Figure 1
Figure 1. We train Morph-1B and its variant models on 30B to￾kens. The results indicate that Morph-1B maintains high accuracy on downstream tasks and achieves faster inference than open￾source models and their variants. OPT-IML-1.3B achieves slightly higher performance on downstream tasks than Morph-1B since it is trained on 180B tokens (Iyer et al., 2022) and is instruction-tuned. We obtain the accuracy by evaluating models … view at source ↗
Figure 2
Figure 2. Open-Source LLM’s Inference Latency: An overview of inference latency in open-source LLMs. The evaluated models include LLaMA (Touvron et al., 2023b), Qwen (Yang et al., 2024), Gemma (Team et al., 2024a;b), and MiniCPM (Hu et al., 2024). All evaluations were performed using the Hugging Face generate function on a single NVIDIA Ampere 40GB A100 GPU with batch size 1, input length 128, and output length 256. laws. We … view at source ↗
Figure 3
Figure 3. Model Shape on End-to-End Inference Latency: (Left) We illustrate the correlation between inference latency and the number of layers, with constant hidden size. Due to the sequential nature of LLM execution, latency increases linearly with the number of layers. (Center) We plot the relationship between inference latency and hidden size with the number of layers fixed. We see that model width does not affect latency … view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: Model Shape on Throughput: We examine the relationship between inference throughput and model architecture by fixing the total parameter count and varying the hidden size and number of layers. Across different batch sizes, wider and shallower models consistently yield …
Figure 5
Figure 5. Figure 5: Inference-Efficient Scaling Laws: In this plot, each data point represents a training run with the given configuration. The dashed lines represent predictions based on the inference-efficient scaling laws outlined in Eq. (4). (Left) The number of training tokens is 20N…
Figure 6
Figure 6. Figure 6: An Overview of Methodology: (A) The model training team first selects several candidate models with various model sizes and configurations; (B) Measure the inference latency using open-source inference systems and predict model loss with fitted scaling laws; (C) Select…
Figure 7
Figure 7. Figure 7: Accuracy vs. Loss: (Left) We illustrate the correlation between accuracy and model loss on PIQA (Bisk et al., 2020). (Center) We present the connection between accuracy and model loss on BoolQ (Clark et al., 2019). (Right) We show the connection between accuracy and mo…
Figure 8
Figure 8. Figure 8: Comparison: (Left) We illustrate the predicted versus actual loss using Eq. (2). (Center) We display the comparison of predicted to actual loss based on Eq. (4). Dots represent data points used for curve-fitting, while cross marks represent test data points. (Right) We…
Figure 9
Figure 9. Figure 9: Excluding Over-training Data: We avoid using over-training data to fit the scaling laws. (Left) The figure is plotted by using Eq. (2). (Center) the center figure is created with Eq. (4). (Right) We plot the Spearman correlation of our scaling law versus the Chinchilla…
Figure 10
Figure 10. Figure 10: Random Choice of Model Shape: We randomly select the model shape to fit the scaling laws. (Left) The figure is plotted by using Eq. (2). (Center) The center figure is created with Eq. (4). (Right) We plot the Spearman correlation of our scaling law versus the Chinchil…
Figure 11
Figure 11. Figure 11: Open-Source LLM’s Inference Latency: An overview of inference latency in open-source LLMs. The evaluated models include LLaMA (Touvron et al., 2023a), Qwen (Yang et al., 2024), Gemma (Team et al., 2024a;b), and MiniCPM (Hu et al., 2024). All evaluations were performed…
Figure 12
Figure 12. Figure 12: Model Shape on Inference Latency over A30 GPU: (Left) We illustrate the correlation between inference latency and the number of layers, with the constant hidden size. (Center) We indicate the relationship between inference latency and hidden size with the number of la…
Figure 13
Figure 13. Figure 13: Model Shape on Inference Latency over A30 GPU with different number of input and output tokens: (Left) We illustrate the correlation between inference latency and the number of layers, with the constant hidden size. (Center) We indicate the relationship between infere…
Figure 14
Figure 14. Figure 14: Model Shape on Time To First Token (TTFT): We examine the relationship between TTFT and model architecture by fixing the total parameter count and varying the hidden size and number of layers. Across different batch sizes, wider and shallower models consistently achie…
Figure 15
Figure 15. Figure 15: Open-Source LLM’s Inference Latency over vLLM (Kwon et al., 2023) using A100 GPU: An overview of inference latency in open-source LLMs. The evaluated models include LLaMA (Touvron et al., 2023a), Qwen (Yang et al., 2024), Gemma (Team et al., 2024a;b), and MiniCPM (Hu …
Figure 16
Figure 16. Figure 16: Model Shape on Inference Latency over vLLM (Kwon et al., 2023) using A100 GPU: (Left) We illustrate the correlation between inference latency and the number of layers, with the constant hidden size. (Center) We indicate the relationship between inference latency and h…
Figure 17
Figure 17. Figure 17: Random Choice of Model Shape - Trial 2: We randomly select the model shape to fit the scaling laws. (Left) The figure is plotted by using Eq. (2). (Center) The center figure is created with Eq. (4). (Right) We plot the Spearman correlation of our scaling law versus th…
Figure 18
Figure 18. Figure 18: Random Choice of Model Shape - Trial 3: We randomly select the model shape to fit the scaling laws. (Left) The figure is plotted by using Eq. (2). (Center) The center figure is created with Eq. (4). (Right) We plot the Spearman correlation of our scaling law versus th…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Search

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A skill-graph random-walk model gives closed-form accuracy-versus-compute formulas for four reasoning strategies and connects them to training scaling.

Reference graph

Works this paper leans on

60 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [1]

    Results over vLLM In this section, we first evaluate the inference efficiency of open-source large language models over vLLM using NVIDIA Tesla A100 Ampere 40 GB GPU

    16 Scaling Inference-Efficient Language Models D. Results over vLLM In this section, we first evaluate the inference efficiency of open-source large language models over vLLM using NVIDIA Tesla A100 Ampere 40 GB GPU. From Figure 15, we find that the efficiency of model inference is influenced not only by the number of parameters but also by the model’s ar...

    work page 2023

  2. [2]

    Abnar, S., Shah, H., Busbridge, D., Ali, A. M. E., Susskind, J., and Thilak, V . Parameters vs flops: Scaling laws for optimal sparsity for mixture-of-experts language models. arXiv preprint arXiv:2501.12370,

    arXiv

  3. [4]

    dmodel is the hidden size, fsize is the intermediate size,n layers is the number of layers, andn heads is the number of attention heads

    Model Architectures:We list the architectural configurations of all models trained in this paper. dmodel is the hidden size, fsize is the intermediate size,n layers is the number of layers, andn heads is the number of attention heads. Model Size Variantd model fsize nlayers nheads 80M v1 512 1536 8 8 80M v2 576 1536 5 8 80M v3 640 1792 3 8 80M v4 448 1280...

    work page 2024

  4. [7]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901,

    1901

  5. [8]

    Boolq: Exploring the surprising difficulty of natural yes/no questions.arXiv preprint arXiv:1905.10044,

    Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions.arXiv preprint arXiv:1905.10044,

    arXiv 1905

  6. [11]

    The evaluated models include LLaMA (Touvron et al., 2023a), Qwen (Yang et al., 2024), Gemma (Team et al., 2024a;b), and MiniCPM (Hu et al., 2024)

    Open-Source LLM’s Inference Latency:An overview of inference latency in open-source LLMs. The evaluated models include LLaMA (Touvron et al., 2023a), Qwen (Yang et al., 2024), Gemma (Team et al., 2024a;b), and MiniCPM (Hu et al., 2024). All evaluations were performed using the Hugging Face generate function on a single NVIDIA A30 Tensor Core GPU. In defau...

    work page 2024

  7. [12]

    Gadre, S

    URL https://www.flux.utah.edu/ paper/duplyakin-atc19. Gadre, S. Y ., Smyrnis, G., Shankar, V ., Gururangan, S., Wortsman, M., Shao, R., Mercat, J., Fang, A., Li, J., Keh, S., et al. Language models scale reliably with over-training and on downstream tasks.arXiv preprint arXiv:2403.08540,

    arXiv

  8. [13]

    K.-H., Cao, T., Yang, F., et al

    Gao, Y ., Zeng, Z., Du, D., Cao, S., Zhou, P., Qi, J., Lai, J., So, H. K.-H., Cao, T., Yang, F., et al. Seerattention: Learning intrinsic sparse attention in your llms.arXiv preprint arXiv:2410.13276,

    arXiv

Show all 60 references
  1. [14]

    L., Liu, Y ., Shang, N., Sun, Y ., Zhu, Y ., Yang, F., and Yang, M

    Guan, X., Zhang, L. L., Liu, Y ., Shang, N., Sun, Y ., Zhu, Y ., Yang, F., and Yang, M. rstar-math: Small llms can master math reasoning with self-evolved deep thinking.arXiv preprint arXiv:2501.04519,

    arXiv

  2. [15]

    Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    arXiv

  3. [16]

    GitHub repository

    URL https://github.com/ mlfoundations/open_lm/. GitHub repository. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding.arXiv preprint arXiv:2009.03300,

    2009 arXiv

  4. [17]

    Measuring math- ematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring math- ematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    arXiv

  5. [18]

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models.arXiv preprint arXiv:2203.15556,

    arXiv

  6. [19]

    Minicpm: Unveiling the potential of small language models with scalable train- ing strategies.arXiv preprint arXiv:2404.06395,

    Hu, S., Tu, Y ., Han, X., He, C., Cui, G., Long, X., Zheng, Z., Fang, Y ., Huang, Y ., Zhao, W., et al. Minicpm: Unveiling the potential of small language models with scalable train- ing strategies.arXiv preprint arXiv:2404.06395,

    arXiv

  7. [20]

    V ., Pasunuru, R., Mihaylov, T., Simig, D., Yu, P., Shuster, K., Wang, T., Liu, Q., Koura, P

    Iyer, S., Lin, X. V ., Pasunuru, R., Mihaylov, T., Simig, D., Yu, P., Shuster, K., Wang, T., Liu, Q., Koura, P. S., et al. Opt-iml: Scaling language model instruction meta learning through the lens of generalization.arXiv preprint arXiv:2212.12017,

    arXiv

  8. [21]

    H., Li, D., Lin, C.-Y ., et al

    Jiang, H., Li, Y ., Zhang, C., Wu, Q., Luo, X., Ahn, S., Han, Z., Abdi, A. H., Li, D., Lin, C.-Y ., et al. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention.arXiv preprint arXiv:2407.02490,

    arXiv

  9. [22]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361,

    2001 arXiv

  10. [23]

    Scaling laws for fine-grained mixture of experts.arXiv preprint arXiv:2402.07871,

    Krajewski, J., Ludziejewski, J., Adamczewski, K., Pi ´oro, M., Krutul, M., Antoniak, S., Ciebiera, K., Kr ´ol, K., Odrzyg´o´zd´z, T., Sankowski, P., et al. Scaling laws for fine-grained mixture of experts.arXiv preprint arXiv:2402.07871,

    arXiv

  11. [24]

    F., Bordelon, B., Muen- nighoff, N., Paul, M., Pehlevan, C., R´e, C., and Raghu- nathan, A

    Kumar, T., Ankner, Z., Spector, B. F., Bordelon, B., Muen- nighoff, N., Paul, M., Pehlevan, C., R´e, C., and Raghu- nathan, A. Scaling laws for precision.arXiv preprint arXiv:2411.04330,

    arXiv

  12. [25]

    Datacomp- lm: In search of the next generation of training sets for lan- guage models.arXiv preprint arXiv:2406.11794,

    Li, J., Fang, A., Smyrnis, G., Ivgi, M., Jordan, M., Gadre, S., Bansal, H., Guha, E., Keh, S., Arora, K., et al. Datacomp- lm: In search of the next generation of training sets for lan- guage models.arXiv preprint arXiv:2406.11794,

    arXiv

  13. [26]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434, 2024a

    Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Dengr, C., Ruan, C., Dai, D., Guo, D., et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434, 2024a. Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, ...

    arXiv

  14. [27]

    MLC-LLM, 2023-2025

    MLC team. MLC-LLM, 2023-2025. URL https:// github.com/mlc-ai/mlc-llm. Muennighoff, N., Rush, A., Barak, B., Le Scao, T., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., and Raffel, C. A. Scaling data-constrained language models.Advances in Neural Information Processing Systems, 3...

    2023

  15. [28]

    Tensorflow- serving: Flexible, high-performance ml serving.arXiv preprint arXiv:1712.06139,

    Olston, C., Fiedel, N., Gorovoy, K., Harmsen, J., Lao, L., Li, F., Rajashekhar, V ., Ramesh, S., and Soyke, J. Tensorflow- serving: Flexible, high-performance ml serving.arXiv preprint arXiv:1712.06139,

    arXiv

  16. [32]

    Beyond chinchilla-optimal: Accounting for inference in language model scaling laws.arXiv preprint arXiv:2401.00448,

    Sardana, N., Portes, J., Doubov, S., and Frankle, J. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws.arXiv preprint arXiv:2401.00448,

    arXiv

  17. [33]

    Fast transformer decoding: One write-head is all you need.arXiv preprint arXiv:1911.02150,

    Shazeer, N. Fast transformer decoding: One write-head is all you need.arXiv preprint arXiv:1911.02150,

    1911 arXiv

  18. [34]

    Scaling llm test- time compute optimally can be more effective than scal- ing model parameters.arXiv preprint arXiv:2408.03314,

    Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling llm test- time compute optimally can be more effective than scal- ing model parameters.arXiv preprint arXiv:2408.03314,

    arXiv

  19. [36]

    Scaling laws with vocabulary: Larger models deserve larger vocabularies.arXiv preprint arXiv:2407.13623,

    Tao, C., Liu, Q., Dou, L., Muennighoff, N., Wan, Z., Luo, P., Lin, M., and Wong, N. Scaling laws with vocabulary: Larger models deserve larger vocabularies.arXiv preprint arXiv:2407.13623,

    arXiv

  20. [37]

    W., Narang, S., Yogatama, D., Vaswani, A., and Metzler, D

    Tay, Y ., Dehghani, M., Rao, J., Fedus, W., Abnar, S., Chung, H. W., Narang, S., Yogatama, D., Vaswani, A., and Metzler, D. Scale efficiently: Insights from pre- training and fine-tuning transformers.arXiv preprint arXiv:2109.10686,

    arXiv

  21. [38]

    S., Love, J., et al

    Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivi`ere, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini research and technology.arXiv preprint arXiv:2403.08295, 2024a. Team, G., Riviere, M., Pathak, S., Sessa, P. G...

    arXiv

  22. [39]

    Glue: A multi-task benchmark and analysis plat- form for natural language understanding.arXiv preprint arXiv:1804.07461,

    Wang, A. Glue: A multi-task benchmark and analysis plat- form for natural language understanding.arXiv preprint arXiv:1804.07461,

    arXiv

  23. [40]

    Huggingface’s transformers: State-of-the- art natural language processing.arXiv preprint arXiv:1910.03771,

    Wolf, T. Huggingface’s transformers: State-of-the- art natural language processing.arXiv preprint arXiv:1910.03771,

    1910 arXiv

  24. [42]

    Duoattention: Efficient long-context llm inference with retrieval and streaming heads.arXiv preprint arXiv:2410.10819,

    Xiao, G., Tang, J., Zuo, J., Guo, J., Yang, S., Tang, H., Fu, Y ., and Han, S. Duoattention: Efficient long-context llm inference with retrieval and streaming heads.arXiv preprint arXiv:2410.10819,

    arXiv

  25. [43]

    Decoding speculative decoding.arXiv preprint arXiv:2402.01528,

    Yan, M., Agarwal, S., and Venkataraman, S. Decoding speculative decoding.arXiv preprint arXiv:2402.01528,

    arXiv

  26. [44]

    Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115,

    arXiv

  27. [45]

    Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800,

    Yao, Y ., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Li, H., Zhao, W., He, Z., et al. Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800,

    arXiv

  28. [46]

    Flash- infer: Efficient and customizable attention engine for llm inference serving.arXiv preprint arXiv:2501.01005,

    Ye, Z., Chen, L., Lai, R., Lin, W., Zhang, Y ., Wang, S., Chen, T., Kasikci, B., Grover, V ., Krishnamurthy, A., et al. Flash- infer: Efficient and customizable attention engine for llm inference serving.arXiv preprint arXiv:2501.01005,

    arXiv

  29. [47]

    Native sparse attention: Hardware-aligned and natively trainable sparse attention.arXiv preprint arXiv:2502.11089,

    Yuan, J., Gao, H., Dai, D., Luo, J., Zhao, L., Zhang, Z., Xie, Z., Wei, Y ., Wang, L., Xiao, Z., et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention.arXiv preprint arXiv:2502.11089,

    arXiv

  30. [48]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830,

    Zellers, R., Holtzman, A., Bisk, Y ., Farhadi, A., and Choi, Y . Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830,

    1905 arXiv

  31. [49]

    V ., et al

    Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V ., et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068,

    arXiv

  32. [54]

    (Center) We indicate the relationship between inference latency and hidden size with the number of layers fixed

    Model Shape on Inference Latency over A30 GPU:(Left) We illustrate the correlation between inference latency and the number of layers, with the constant hidden size. (Center) We indicate the relationship between inference latency and hidden size with the number of layers fixed...

    2000

  33. [56]

    The evaluated models include LLaMA (Touvron et al., 2023a), Qwen (Yang et al., 2024), Gemma (Team et al., 2024a;b), and MiniCPM (Hu et al., 2024)

    using A100 GPU:An overview of inference latency in open-source LLMs. The evaluated models include LLaMA (Touvron et al., 2023a), Qwen (Yang et al., 2024), Gemma (Team et al., 2024a;b), and MiniCPM (Hu et al., 2024). All evaluations were performed using the Hugging Face generat...

    2024

  34. [57]

    0 5 10 15 20 nlayers 0 2 4 6 8Latency (s) 4096 5120 6144 8192 (a) Vary layers (nlayers), fix hidden size 0 2000 4000 6000 8000 10000 dmodel 0 2 4 6 8Latency (s) 4 8 12 16 (b) Vary hidden size (dmodel), fix layers 102 103 dmodel / nlayers 0 1 2 3 4 5 6Latency (s) 1B 3B 7B (c) V...

    2000

  35. [58]

    2048 16 128 Llama-3.2-3B (Dubey et al.,

  36. [59]

    2048 36 56.9 Qwen2.5-7B (Yang et al.,

  37. [60]

    5120 48 106.7 gemma-2b (Team et al., 2024a) 2048 18 113.8 gemma-7b (Team et al., 2024a) 3072 28 109.7 gemma-2-2b (Team et al., 2024b) 2304 26 88.6 gemma-2-9b (Team et al., 2024b) 3584 42 85.3 gemma-2-27b (Team et al., 2024b) 4608 46 100.2 microsoft-phi-2 (Phi,

  38. [256]

    0 5 10 15 20 25 30 nlayers 0 2 4 6 8 10 12Latency (s) 4096 5120 6144 8192 (a) Vary layers (nlayers), fix hidden size 0 2000 4000 6000 8000 10000 dmodel 0 2 4 6 8 10 12Latency (s) 4 8 12 16 (b) Vary hidden size (dmodel), fix layers 102 103 dmodel / nlayers 0 2 4 6 8 10 12Latenc...

    2000

  39. [1921]

    Ef- ficient streaming language models with attention sinks

    Xiao, G., Tian, Y ., Chen, B., Han, S., and Lewis, M. Ef- ficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453,

    arXiv

  40. [1961]

    Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.arXiv preprint arXiv:2206.04615,

    arXiv

  41. [2011]

    J., and Hashimoto, T

    Ruan, Y ., Maddison, C. J., and Hashimoto, T. Observational scaling laws and the predictability of language model performance.arXiv preprint arXiv:2405.10938,

    arXiv

  42. [2016]

    L., Yang, F., and Yang, M

    Qi, Z., Ma, M., Xu, J., Zhang, L. L., Yang, F., and Yang, M. Mutual reasoning makes smaller llms stronger problem- solvers.arXiv preprint arXiv:2408.06195,

    arXiv

  43. [2017]

    N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fern´andez, R

    Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q. N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fern´andez, R. The lambada dataset: Word prediction requiring a broad discourse context.arXiv preprint arXiv:1606.06031,

    arXiv

  44. [2018]

    Training verifiers to solve math word problems

    Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168,

    arXiv

  45. [2019]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457,

    arXiv

  46. [2020]

    Gpt-neox-20b: An open-source autoregressive language model.arXiv preprint arXiv:2204.06745,

    Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al. Gpt-neox-20b: An open-source autoregressive language model.arXiv preprint arXiv:2204.06745,

    arXiv

  47. [2021]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    10 Scaling Inference-Efficient Language Models Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    arXiv

  48. [2022]

    V ., R´e, C., and Mirhoseini, A

    Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V ., R´e, C., and Mirhoseini, A. Large language monkeys: Scaling inference compute with repeated sampling.arXiv preprint arXiv:2407.21787,

    arXiv

  49. [2023]

    Qwen technical report.arXiv preprint arXiv:2309.16609,

    Bai, J., Bai, S., Chu, Y ., Cui, Z., Dang, K., Deng, X., Fan, Y ., Ge, W., Han, Y ., Huang, F., et al. Qwen technical report.arXiv preprint arXiv:2309.16609,

    arXiv

  50. [2024]

    J., Javaheripi, M., Kauffmann, P., et al

    Abdin, M., Aneja, J., Behl, H., Bubeck, S., Eldan, R., Gunasekar, S., Harrison, M., Hewett, R. J., Javaheripi, M., Kauffmann, P., et al. Phi-4 technical report.arXiv preprint arXiv:2412.08905,

    arXiv

  51. [2025]

    Chai: Clustered head attention for efficient llm inference.arXiv preprint arXiv:2403.08058, 2024a

    Agarwal, S., Acun, B., Hosmer, B., Elhoushi, M., Lee, Y ., Venkataraman, S., Papailiopoulos, D., and Wu, C.-J. Chai: Clustered head attention for efficient llm inference.arXiv preprint arXiv:2403.08058, 2024a. Agarwal, S., Mao, A., Akella, A., and Venkataraman, S. Symphony: Im...

    arXiv

  52. [2048]

    Model Size Warmup Learning rate Weight decay z-loss Batch size <400M 2000 3e-3 0.033 1e-4 512 1B 5000 3e-3 0.033 1e-4 256 Table

    2000

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.