Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Predictable Scale: Part II, Farseer: A Refined Scaling Law in Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proposes Farseer, a scaling law that predicts a language model's validation loss from its parameter count and training tokens, and claims it extrapolates far beyond the data it was fitted on.

desk verdict A genuinely useful empirical scaling-law paper that deserves a serious referee; the extrapolation headline is real but thinner than the abstract suggests. read the letter →

arxiv 2506.10972 v3 pith:YEM6NZCP submitted 2025-06-12 cs.LG cs.AI

Houyi Li , Wenzhen Zheng , Qiufeng Wang , Zhenyu Ding , Haoying Wang , Zili Wang , Shijie Xuyang , Ning Ding
show 3 more authors
Shuigeng Zhou Xiangyu Zhang Daxin Jiang
This is my paper
classification cs.LGcs.AI
keywords scalinglawslargelanguagemodelsloss-surfaceextrapolationdataexponentcompute-optimaltrainingdifferentialpiecewisefittingbitspercharactertoken-to-parameterratio
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Farseer, a scaling law that predicts a language model's validation loss from its non-embedding parameter count N and training token count D. Its central move is to let the exponent and coefficient of the data-scaling term depend on model size, rather than treating data benefits as a fixed power law as Chinchilla does. On roughly 1,000 trained models, Farseer fits the measured loss surface, measured in bits per character, far better and extrapolates to a 25.1B-parameter model and off-grid points with about 0.5% average relative error, versus 2.68% for Chinchilla's law. The payoff is that cheap small-scale runs can be used to forecast large-scale training loss and compute-optimal data allocation.

What carries the argument

The load-bearing object is the decomposition L(N,D)=E+U(N)+V(D)+H(N,D) together with the empirical ansatz that the data-dependent part equals B(N)$D^{{-A(N)}}$. The paper establishes this via finite differences: the data-difference term is approximately B(N)(1-$lambda^{{-A(N)}}$)$D^{{-A(N)}}$, a power law in D whose coefficient and exponent depend on N. A small dictionary of identity, log, and power transformations selects stretched-exponential forms A(N)=exp(a_1 $N^{{alpha}}$+b_1) and B(N)=exp(a_2 $N^{{beta}}$+b_2), and the model-size residual is fit as exp(a_3 $N^{{gamma}}$+b_3). This three-stage procedure, called Differential Piecewise Fitting, converts per-model-size linear fits into globally refined continuous functions, and that continuity is what makes extrapolation stable.

What would settle it

Train one dense decoder-only model of about 50B non-embedding parameters on roughly 2 x $10^{12}$ tokens under the same architecture family, data recipe, and hyperparameter procedure, measure bits per character on the same validation set, and compare with the Farseer formula. A relative error much above 1% would show that the fitted form does not continue beyond the calibration grid; an alternative is to fit Farseer only on models up to 3B parameters and check whether its predicted curves pass through the observed 6.4B and 25.1B points.

Watch

Extended reading notes

Core claim

The paper asserts that the full loss surface can be written as L(N,D)=U(N)+B(N)$D^{{-A(N)}}$, where B(N) and A(N) are smooth functions of model size, and the fitted instance is L(N,D)=exp(-0.021 $N^{{0.169}}$-0.091)+exp(88.01 $N^{{-0.1}}$-6.287) $D^{{-exp(-0.124 N^{0.123}}$+0.424)}. The data-dependent part is not a constant power law in D: both the exponent A(N) and the coefficient B(N) vary smoothly with model size. The paper argues that this N-dependence is forced by finite-difference analysis of the measured loss, which shows the data-difference term depends on both N and D and follows a power law in D. With this form and a three-stage fitting procedure, the law predicts validation loss at scales beyond its fitted grid, including a 25.1B model, and implies that the optimal token-to-parameter ratio grows with compute budget instead of staying near 20.

Load-bearing premise

The load-bearing premise is that the mathematical shape fitted on small models and modest token counts continues to hold for much larger models and much larger datasets; if the loss curve bends differently just past the fitted range, the 0.5% extrapolation claim has no support.

Editorial extensions

If this is right

  • Small-scale training runs with models up to roughly 6.4B parameters and token counts up to about 5 x 10^11 can predict validation loss at much larger scales, including model sizes over an order of magnitude beyond the fitted range.
  • The optimal token-to-parameter ratio is not a constant near 20; Farseer predicts it rises with compute budget, matching the training configurations used by recent large models.
  • The fitted loss surface allows training strategies such as data mixtures or architectural choices to be compared across arbitrary (N,D) settings, not just at isolated points.
  • A 25.1B-parameter validation model is predicted with 0.47% relative error, and extrapolation errors across six out-of-domain points average 0.50%, versus 2.68% for Chinchilla's law.
  • The open release of roughly 1,000 trained models and their logs lets other groups refit the law or test it on new data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if this functional form generalizes, data quality and data-mixture changes plausibly shift the smooth functions A(N), B(N), and U(N) in systematic ways, making Farseer a common yardstick for comparing pretraining recipes before full-scale training.
  • Beyond the paper: the differential piecewise fitting strategy could transfer to other smooth bivariate performance surfaces, such as inference-time compute or fine-tuning tokens, wherever a finite-difference power-law check can be run first.
  • Beyond the paper: a sharper curve-level test would fit Farseer only on models up to about 3B parameters and check whether its predicted A(N), B(N), and U(N) curves pass through observed values at 6.4B and 25.1B; the paper reports aggregate errors but not this exact diagnostic.
  • Beyond the paper: if the optimal token-to-parameter ratio keeps rising, compute-optimal training on future budgets will use notably more tokens per parameter than Chinchilla's rule, which would shift infrastructure planning toward data pipelines rather than further concentrating compute in model parameters.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper proposes Farseer, a three-term scaling law L(N,D)=exp(a3 N^gamma + b3) + exp(a2 N^beta + b2) D^(-exp(a1 N^alpha + b1)), together with a differential piecewise fitting procedure that estimates the data-dependent term B(N) D^(-A(N)) from finite differences of loss and fits the residual as E+U(N). The model is fit to roughly 1,000 Llama-style LLMs with non-embedding sizes from about 2e8 to 6.4e9 and token budgets from about 1e9 to 5e11, evaluated by BPC. The authors report a better in-range fit than Chinchilla, extrapolation to six held-out points including a 25.1B model with 0.50% average relative error versus 2.68% for Chinchilla, and a predicted D/N ratio that grows with compute. They also study robustness to fitting volume, generalization to an English-Chinese data mix, and monotonicity of the fitted surface.

Significance. The main contribution is a candidate scaling law with an N-dependent exponent and coefficient and a very large empirical dataset. If the extrapolation result holds, it is practically valuable for forecasting the loss of large training runs from small-scale experiments. Strengths include open-sourcing roughly a thousand models, logs, and fitting code; a genuine held-out 25.1B validation point; a monotonicity verification; and a bilingual-data generalization experiment. However, the extrapolation claim currently rests on six hand-picked points without confidence intervals or repeated seeds, and near-tie functional forms are not tested out of sample, so the significance depends on the additional validation requested.

major comments (4)
  1. [Section 4.3 and Figure 7] The central extrapolation claim rests on six validation points with no confidence intervals, no repeated seeds, and no statement of run-to-run variability in BPC. The 25.1B model is only 25.1/6.37, approximately 3.9, times larger than the largest fitted model size, not 'more than an order of magnitude'; please correct this wording. Report bootstrap or parameter-uncertainty intervals for the predicted BPC values and for the mean relative errors, so the reader can judge whether the 0.50% versus 2.68% difference is outside training noise.
  2. [Appendix C.3, Table 2] The stretched-exponential/stretched-exponential form is selected from the dictionary G={identity, log, power} by minimizing the projection residuals ell_A plus ell_B in Eq. (8), but the top alternatives are only 2-16% worse in that criterion (0.1125 versus 0.1148 and 0.1306). The paper never evaluates these near-equally-fitting alternatives at the held-out 25.1B point. If a runner-up form, such as stretched-exponential/power-law, gives a much larger error at 25.1B, the claimed 0.47% extrapolation accuracy is an artifact of a near-tie in the model-selection criterion. Please report the held-out extrapolation errors for the top candidate functional forms.
  3. [Algorithm 2, lines 8-13; Eq. (12)] The exponents alpha and beta are refined by minimizing the difference residual ell_R = sum(R_N(D) - tilde R_N(D))^2, not the residual of the loss L(N,D) itself. The subsequent stage defines G(N) = Avg_D[L(N,D) - B(N)D^(-A(N))], so any systematic in-range approximation error of the power-law term is absorbed into E+U(N). Such absorbed errors need not persist outside the fitted N range, which is exactly the regime of the headline extrapolation. Please provide a quantitative loss-residual diagnostic for the fitted decomposition O(N,D)-G(N) and report whether a loss-based global refinement criterion changes the extrapolated predictions materially.
  4. [Section 5.3] The optimal D/N allocation guidance is a mathematical consequence of the fitted formula rather than an independent prediction, so the agreement with configurations of Qwen and Llama should be quantified rather than asserted visually. Please provide a numerical comparison of predicted versus actual D/N for the cited models, or a sensitivity analysis showing how much the optimal ratio changes under the parameter uncertainty highlighted in the extrapolation section.
minor comments (5)
  1. [Abstract and Section 4.3] The phrase 'reducing extrapolation error by 433%' is mathematically awkward: a relative error decrease from 2.68% to 0.50% is a factor of 5.4, so the error is 81% lower, or equivalently Chinchilla's error is 433% higher. Please rephrase.
  2. [Eq. (3)] The exponential expressions such as 'ea3·N^gamma + b3' lack visible braces and can be misread as products; please use exp(...) notation consistently.
  3. [Figure 1 caption] There is a typo in the caption: 'Chichilla' should be 'Chinchilla'.
  4. [Section 2.2] The text contains the duplicated phrase 'detailed in detailed in Appendix B'; please remove the duplication.
  5. [Appendix J] The closing sentence 'Farseer's experimental methodology will spark an cognitive revolution' is an overstatement for a formal journal article; please consider a more measured statement.

Circularity Check

2 steps flagged · score 4.0 of 10

Headline extrapolation claim is partly selected on the 25.1B validation point; the fitted formula itself is otherwise not circular.

  1. fitted input called prediction [Appendix G; Section 2.2 (definition of N); Section 4.3 (25.1B extrapolation claim)]
    "We validated the fitted laws at a 25.1 billion parameter scale (denoted as Nval = 25.1B). At this larger validation point, the discrepancy between the two an N with embedding layers scaled law, the error in prediction increased substantially to 0.029. In contrast, the scaling law derived using N exclusive of embedding parameters demonstrated significantly better extrapolation, with an error more than four times smaller than its counterpart."

    Section 2.2 fixes the definition of N by excluding embeddings and refers to Appendix G. Appendix G selects that definition by comparing extrapolation error at Nval = 25.1B. Section 4.3 then uses the same 25.1B point as the headline validation of Farseer's extrapolation, reporting a 0.47% relative error. The validation target was therefore used to choose a modeling input, so the reported 25.1B accuracy is not an independent out-of-sample prediction; it is partly a consequence of selecting the modeling option that minimized error at that exact point.

  2. fitted input called prediction [Appendix H; Section 4.3 (extrapolation evaluation)]
    "For the Farseer model, the standard non-linear regression approach struggled to converge to an optimal global solution, even with the extensive initialization strategy. The resulting fit produced errors approximately an order of magnitude (10 times) higher than those achieved with our method. ... Given these findings, and to ensure each model was represented by its most effective and accurately fitted version, the comparisons presented in the main body of this paper are based on: ... Farseer model fitted using our Differential Piecewise Fitting method."

    Appendix H selects the Differential Piecewise Fitting method after comparing extrapolation error at the 6B and 25B validation points, as shown in Fig. 17(b). Section 4.3 then reports the same 6B/25B points as evidence of Farseer's extrapolation capability, including the 0.50% average relative error. The fitting method was chosen using the prediction targets themselves, so the extrapolation performance of the final method is not a clean out-of-sample result; it is biased by the model-selection step.

full rationale

The main Farseer formula is obtained by least-squares-style fitting to the training grid, and the 25.1B model and off-grid points are not used as data in that parameter fit, so the core extrapolation is not forced by the fitted parameters alone. The power-law ansatz V(D)+H(N,D)=B(N)D^{-A(N)} is an empirical assumption rather than a derivation, but that is a correctness and robustness concern, not circularity by itself. However, the paper's headline extrapolation evidence is weakened by validation-based model selection: the choice to exclude embedding parameters (Appendix G) and the choice of Differential Piecewise Fitting over end-to-end regression (Appendix H) were both made after observing errors at the same 25.1B and 6B points later reported as predictions in Section 4.3. This makes the 0.50% mean extrapolation error partly a selected result. The optimal D/N guidance in Section 5.3 is a mathematical consequence of the fitted surface, not an independent experimental prediction. The self-citation to the authors' Step Law [25] for hyperparameter selection is a minor methodological dependency and is not load-bearing for the scaling-law formula itself.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on five domain assumptions plus nine fitted parameters. No new physical entities are postulated. The most important dependency is the power-law ansatz for the data-dependent loss term and the smoothness of its coefficient and exponent functions beyond the fitted range; if either fails, the extrapolation claim loses support.

free parameters (3)
  • A(N) stretched-exponential parameters (a1, b1, alpha) = a1=-0.124, b1=0.424, alpha=0.123
    Exponent function A(N)=exp(a1 N^alpha + b1), fit to per-model exponents obtained from finite-difference regression in Stage 1.
  • B(N) stretched-exponential parameters (a2, b2, beta) = a2=88.01, b2=-6.287, beta=-0.1
    Coefficient function B(N)=exp(a2 N^beta + b2), fit to per-model coefficients obtained from finite-difference regression in Stage 1.
  • U(N) stretched-exponential parameters (a3, b3, gamma) = a3=-0.021, b3=-0.091, gamma=0.169
    Model-dependent residual E+U(N) is approximated by exp(a3 N^gamma + b3), fit in Stage 3 after subtracting the fitted data-dependent term.
assumptions (5)
  • domain assumption The loss decomposes additively as E + U(N) + V(D) + H(N,D)
    Introduced in Section 2.1, Eq. (4). The differential method relies on this decomposition and on the cancellation of E and U(N) when taking data differences.
  • domain assumption The data-dependent component V(D)+H(N,D) follows a power law B(N)D^{-A(N)} in D at each model size N
    Central empirical observation from log-log finite differences in Section 3.2, Eqs. (5)-(6), with R^2=0.9807 in Figure 3(a). The entire fitting pipeline assumes this form holds.
  • ad hoc to paper A(N), B(N), and U(N) are smooth stretched-exponential functions of N that continue outside the fitted N range
    Functional forms are selected from a small transformation dictionary by minimizing residuals (Appendix C.3, C.4), not derived from theory. Extrapolation to 25.1B and beyond assumes these forms persist.
  • domain assumption Hyperparameters such as learning rate, batch size, aspect ratio, and FFN ratio are held at appropriate values so L depends only on N and D
    Appendix A.1 relies on the authors' prior Step Law paper [25] and on the claim that the performance landscape is flat near good hyperparameter choices.
  • domain assumption BPC on the private 30M-token validation set is an unbiased, contamination-free performance metric
    Section 2.2 and Appendix D describe a validation set filtered with proprietary pipelines and GPT-4o scoring; comparability across model sizes assumes no systematic bias or leakage.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Predictable Scale: Part II, Farseer: A Refined Scaling Law in Large Language Models." pith.science (2026). https://pith.science/paper/YEM6NZCP

@misc{pith2026250610972,
  author       = {Pith},
  title        = {Pith review of: Predictable Scale: Part II, Farseer: A Refined Scaling Law in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YEM6NZCP}},
  note         = {Machine review of arXiv:2506.10972}
}
abstract

Training Large Language Models (LLMs) is prohibitively expensive, creating a critical scaling gap where insights from small-scale experiments often fail to transfer to resource-intensive production systems, thereby hindering efficient innovation. To bridge this, we introduce Farseer, a novel and refined scaling law offering enhanced predictive accuracy across scales. By systematically constructing a model loss surface $L(N,D)$, Farseer achieves a significantly better fit to empirical data than prior laws (e.g., Chinchilla's law). Our methodology yields accurate, robust, and highly generalizable predictions, demonstrating excellent extrapolation capabilities, improving upon Chinchilla's law by reducing extrapolation error by 433\%. This allows for the reliable evaluation of competing training strategies across all $(N,D)$ settings, enabling conclusions from small-scale ablation studies to be confidently extrapolated to predict large-scale performance. Furthermore, Farseer provides new insights into optimal compute allocation, better reflecting the nuanced demands of modern LLM training. To validate our approach, we trained an extensive suite of approximately 1,000 LLMs across diverse scales and configurations, consuming roughly 3 million NVIDIA H100 GPU hours. We are comprehensively open-sourcing all models, data, results, and logs at https://github.com/Farseer-Scaling-Law/Farseer to foster further research.

Figures

Figures reproduced from arXiv: 2506.10972 by the authors.

Figure 1
Figure 1. Farseer beats Chichilla [20]. (a) Average relative error (BPC) vs. model size N for Farseer (red) and Chinchilla (blue). Chinchilla, lacking high-order cross terms, fits only near the central N and its error diverges as model size grows. In contrast, Farseer’s error is 232% lower within the fitted range and remains stable across the full N range. (b) Chinchilla’s rule of thumb (D/N ≈ 20) is valid only at moderate bu… view at source ↗
Figure 2
Figure 2. Empirical BPC values (Ground Truth) [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Log-Log analysis of the differential BPC terms ∆DL and ∆N L as univariate functions of D and N, respectively: (a) ∆DL vs. D at fixed N (R2 = 0.9807); (b) ∆DL vs. N at fixed D (R2 = 0.7457); (c) ∆N L vs. D at fixed N (R2 = 0.8484); (d) ∆N L vs. N at fixed D ( R2 = 0.8733). The high R2 in (a) (0.9807) demonstrates a consistent power-law relationship between ∆DL and D, so we adopt this form in our main analysis. As det… view at source ↗
Figures from the paper (17 more)
Figure 4
Figure 4. Figure 4: Fits of the terms A(N) and B(N) using stretched exponential functions. Blue dots denote actual values; orange crosses denote predictions. The fits align well across a wide range of N. 3.3 Functional Form for Model-Dependent Residual E + U(N) By construction, E + U(N) =…
Figure 5
Figure 5. Figure 5: Predicted vs. actual values at residuals. (a) Comparison of actual [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Robustness and data distribution generalizability of [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Extrapolation of Farseer. Blue circles represent the grid of (N, D) employed to fit. Red stars denote validation points beyond that distribu￾tion, including a 25.1 B model, larger dataset sizes, and off-grid combinations. Annotated percentages give the relative errors …
Figure 8
Figure 8. Figure 8: Farseer vs Chinchilla at robustness and extrapolation. (a) For 6.4 B models, Farseer shows lower and steadier errors as N upper bound increases. (b) For 25.1 B model, far beyond the fitting range, Farseer achieves a clear error reduction as N increases, while Chinchill…
Figure 9
Figure 9. Figure 9: Validation BPC (see appendix D) under a fixed parameter N versus (a) FFN ratio (interme￾diate size / hidden dimension) and (b) aspect ratio (hidden dimension / layer count). Blue circles: no embeddings; orange squares: with embeddings. Dashed lines mark min BPC (green)…
Figure 10
Figure 10. Figure 10: Log-Log projections of the coefficient functions derived from four differential loss [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Log-Log projections of the exponent functions derived from four differential loss perspec [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: Example of the power-law relationship between the finite difference of the loss, denoted [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 13
Figure 13. Figure 13: Example of the power-law relationship between the finite difference of the loss, denoted [PITH_FULL_IMAGE:figures/full_fig_p023_13.png]
Figure 14
Figure 14. Figure 14: Example of the power-law relationship between the finite difference of the loss, denoted [PITH_FULL_IMAGE:figures/full_fig_p024_14.png]
Figure 15
Figure 15. Figure 15: shows the 3D surface of ∆ over log10 N and log10 D, with the zero plane marking parity between the two mixtures. Regions above the plane indicate that the 50% English mixture yields lower error, while regions below favor the 85% mixture. Green squares denote individua…
Figure 16
Figure 16. Figure 16: Scaling law extrapolation with and without counting embedding parameters in model [PITH_FULL_IMAGE:figures/full_fig_p029_16.png]
Figure 17
Figure 17. Figure 17: (a) Chinchilla performs slightly better with standard regression, while Farseer shows a [PITH_FULL_IMAGE:figures/full_fig_p030_17.png]
Figure 18
Figure 18. Figure 18: Visualization of L(N, D) as a function of model size N and data size D. log10(N) 8 9 10 11 12 13 log10(D) 8 9 10 11 12 13 14 15 L/ D 1e 9 2.0 1.5 1.0 0.5 0.0 L/ D 2.00 1.75 1.50 1.25 1.00 0.75 0.50 0.25 0.00 L/ D 1e 9 [PITH_FULL_IMAGE:figures/full_fig_p032_18.png]
Figure 19
Figure 19. Figure 19: Visualization of ∂L/∂D as a function of model size N and data size D, showing a consistently negative gradient across the domain. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_19.png]
Figure 20
Figure 20. Figure 20: Visualization of ∂L/∂N over varying N and D, also showing a consistently negative gradient across the domain. J Discussion and Future Work Through a rigorous experimental design, a massive number of experiments, and extensive experimen￾tal data collected over 18 month…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. From Zipf's Law to Neural Scaling through Heaps' Law and Hilberg's Hypothesis

    cs.IT 2025-12 conditional novelty 6.0 of 10

    Zipf's law, via differential Heaps and Hilberg laws, forces a power-law lower bound on the excess cross entropy of any entropy-bounded foundation model.

Reference graph

Works this paper leans on

55 extracted references · 22 canonical work pages · cited by 1 Pith paper

  1. [1]

    Parameters vs flops: Scaling laws for optimal sparsity for mixture-of-experts language models.arXiv preprint arXiv:2501.12370, 2025

    Samira Abnar, Harshay Shah, Dan Busbridge, Alaaeldin Mohamed Elnouby Ali, Josh Susskind, and Vimal Thilak. Parameters vs flops: Scaling laws for optimal sparsity for mixture-of-experts language models.arXiv preprint arXiv:2501.12370, 2025

    arXiv 2025

  2. [2]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Explaining neural scaling laws.Proceedings of the National Academy of Sciences, 121(27):e2311878121, 2024

    Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws.Proceedings of the National Academy of Sciences, 121(27):e2311878121, 2024

    2024

  4. [4]

    Chinchilla scaling: A replication attempt.arXiv preprint arXiv:2404.10102, 2024

    Tamay Besiroglu, Ege Erdil, Matthew Barnett, and Josh You. Chinchilla scaling: A replication attempt.arXiv preprint arXiv:2404.10102, 2024

    arXiv 2024

  5. [5]

    How feature learning can improve neural scaling laws.arXiv preprint arXiv:2409.17858, 2024

    Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. How feature learning can improve neural scaling laws.arXiv preprint arXiv:2409.17858, 2024

    arXiv 2024

  6. [6]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  7. [7]

    A survey on mixture of experts.arXiv preprint arXiv:2407.06204, 2024

    Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts.arXiv preprint arXiv:2407.06204, 2024

    arXiv 2024

  8. [8]

    Communication-efficient language model training scales reliably and robustly: Scaling laws for diloco.arXiv preprint arXiv:2503.09799, 2025

    Zachary Charles, Gabriel Teston, Lucio Dery, Keith Rush, Nova Fallen, Zachary Garrett, Arthur Szlam, and Arthur Douillard. Communication-efficient language model training scales reliably and robustly: Scaling laws for diloco.arXiv preprint arXiv:2503.09799, 2025

    arXiv 2025

Show all 55 references
  1. [9]

    Scaling laws for predicting downstream performance in llms.arXiv preprint arXiv:2410.08527, 2024

    Yangyi Chen, Binxuan Huang, Yifan Gao, Zhengyang Wang, Jingfeng Yang, and Heng Ji. Scaling laws for predicting downstream performance in llms.arXiv preprint arXiv:2410.08527, 2024. 10

    2024 arXiv

  2. [10]

    Reproducible scaling laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. InProceedings of the IEEE/CVF conference on computer vision ...

    2023

  3. [11]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human langua...

    2019

  4. [12]

    Scaling laws do not scale

    Fernando Diaz and Michael Madaio. Scaling laws do not scale. InProceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 7, pages 341–357, 2024

    2024

  5. [13]

    Scaling laws for dense retrieval

    Yan Fang, Jingtao Zhan, Qingyao Ai, Jiaxin Mao, Weihang Su, Jia Chen, and Yiqun Liu. Scaling laws for dense retrieval. InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1339–1349, 2024

    2024

  6. [14]

    A new algorithm for data compression.The C Users Journal archive, 12:23–38, 1994

    Philip Gage. A new algorithm for data compression.The C Users Journal archive, 12:23–38, 1994

    1994

  7. [15]

    Carl Friedrich Gauss.Theoria motus corporum coelestium in sectionibus conicis solem ambien- tium. Frid. Perthes et I. H. Besser, Hamburg, 1809

  8. [16]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and Amy Yang et al. The llama 3 herd of models. 2024

    2024

  9. [17]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

    2023 arXiv

  10. [18]

    Alexander Havrilla and Wenjing Liao. Understanding scaling laws with statistical and approxi- mation theory for transformer neural networks on intrinsically low-dimensional data.Advances in Neural Information Processing Systems, 37:42162–42210, 2024

    2024

  11. [19]

    Scaling laws for transfer.arXiv preprint arXiv:2102.01293, 2021

    Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer.arXiv preprint arXiv:2102.01293, 2021

    2021 arXiv

  12. [20]

    Rae, Oriol Vinyals, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osin...

    2022 arXiv

  13. [21]

    Scaling laws for downstream task performance of large language models

    Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, and Sanmi Koyejo. Scaling laws for downstream task performance of large language models. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024

    2024

  14. [22]

    Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  15. [23]

    Scaling laws for precision.arXiv preprint arXiv:2411.04330, 2024

    Tanishq Kumar, Zachary Ankner, Benjamin F Spector, Blake Bordelon, Niklas Muennighoff, Mansheej Paul, Cengiz Pehlevan, Christopher Ré, and Aditi Raghunathan. Scaling laws for precision.arXiv preprint arXiv:2411.04330, 2024

    2024 arXiv

  16. [24]

    A simple model of inference scaling laws.arXiv preprint arXiv:2410.16377, 2024

    Noam Levi. A simple model of inference scaling laws.arXiv preprint arXiv:2410.16377, 2024

    2024 arXiv

  17. [25]

    Predictable scale: Part i–optimal hyperparameter scaling law in large language model pretraining.arXiv preprint arXiv:2503.04715, 2025

    Houyi Li, Wenzhen Zheng, Jingcheng Hu, Qiufeng Wang, Hanshan Zhang, Zili Wang, Shi- jie Xuyang, Yuantao Fan, Shuigeng Zhou, Xiangyu Zhang, et al. Predictable scale: Part i–optimal hyperparameter scaling law in large language model pretraining.arXiv preprint arXiv:2503.04715, 2025. 11

    2025 arXiv

  18. [26]

    (mis) fitting scaling laws: A survey of scaling law fitting techniques in deep learning

    Margaret Li, Sneha Kudugunta, and Luke Zettlemoyer. (mis) fitting scaling laws: A survey of scaling law fitting techniques in deep learning. InThe Thirteenth International Conference on Learning Representations, 2024

    2024

  19. [27]

    Scal- ingfilter: Assessing data quality through inverse utilization of scaling laws.arXiv preprint arXiv:2408.08310, 2024

    Ruihang Li, Yixuan Wei, Miaosen Zhang, Nenghai Yu, Han Hu, and Houwen Peng. Scal- ingfilter: Assessing data quality through inverse utilization of scaling laws.arXiv preprint arXiv:2408.08310, 2024

    2024 arXiv

  20. [28]

    Scaling laws for black box adversarial attacks.arXiv preprint arXiv:2411.16782, 2024

    Chuan Liu, Huanran Chen, Yichi Zhang, Yinpeng Dong, and Jun Zhu. Scaling laws for black box adversarial attacks.arXiv preprint arXiv:2411.16782, 2024

    2024

  21. [29]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InInternational Conference on Learning Representations, 2019

    2019

  22. [30]

    The quantization model of neural scaling.Advances in Neural Information Processing Systems, 36:28699–28722, 2023

    Eric Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling.Advances in Neural Information Processing Systems, 36:28699–28722, 2023

    2023

  23. [31]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018

    2018

  24. [32]

    Scaling language models: Methods, analysis & insights from training gopher.arXiv preprint arXiv:2112.11446, 2021

    Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher.arXiv preprint arXiv:2112.11446, 2021

    2021 arXiv

  25. [33]

    Observational scaling laws and the predictability of language model performance.arXiv preprint arXiv:2405.10938, 2024

    Yangjun Ruan, Chris J Maddison, and Tatsunori Hashimoto. Observational scaling laws and the predictability of language model performance.arXiv preprint arXiv:2405.10938, 2024

    2024 arXiv

  26. [34]

    Beyond chinchilla-optimal: accounting for inference in language model scaling laws

    Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. Beyond chinchilla-optimal: accounting for inference in language model scaling laws. InProceedings of the 41st Interna- tional Conference on Machine Learning, ICML’24. JMLR.org, 2024

    2024

  27. [35]

    Scaling laws from the data manifold dimension.Journal of Machine Learning Research, 23(9):1–34, 2022

    Utkarsh Sharma and Jared Kaplan. Scaling laws from the data manifold dimension.Journal of Machine Learning Research, 23(9):1–34, 2022

    2022

  28. [36]

    Scaling law for recommendation models: Towards general-purpose user representations

    Kyuyong Shin, Hanock Kwak, Su Young Kim, Max Nihlén Ramström, Jisu Jeong, Jung-Woo Ha, and Kyung-Min Kim. Scaling law for recommendation models: Towards general-purpose user representations. InProceedings of the AAAI conference on artificial intelligence, volume 37, pages 4596...

    2023

  29. [37]

    Beyond neural scaling laws: beating power law scaling via data pruning.Advances in Neural Information Processing Systems, 35:19523–19536, 2022

    Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning.Advances in Neural Information Processing Systems, 35:19523–19536, 2022

    2022

  30. [38]

    Unraveling the mystery of scaling laws: Part i.arXiv preprint arXiv:2403.06563, 2024

    Hui Su, Zhi Tian, Xiaoyu Shen, and Xunliang Cai. Unraveling the mystery of scaling laws: Part i.arXiv preprint arXiv:2403.06563, 2024

    2024 arXiv

  31. [39]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation langua...

    2023

  32. [40]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, and Shruti Bhosale et al. Llama 2: Open foundation and fine-tuned chat models. 7 2023

    2023

  33. [41]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

    2017

  34. [42]

    Data efficient neural scaling law via model reusing

    Peihao Wang, Rameswar Panda, and Zhangyang Wang. Data efficient neural scaling law via model reusing. InInternational Conference on Machine Learning, pages 36193–36204. PMLR, 2023. 12

    2023

  35. [43]

    Inference scaling laws: An empirical analysis of compute-optimal inference for llm problem-solving

    Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for llm problem-solving. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  36. [44]

    Unveiling downstream performance scaling of llms: A clustering-based perspective.arXiv preprint arXiv:2502.17262, 2025

    Chengyin Xu, Kaiyuan Chen, Xiao Li, Ke Shen, and Chenggang Li. Unveiling downstream performance scaling of llms: A clustering-based perspective.arXiv preprint arXiv:2502.17262, 2025

    2025

  37. [45]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

    2025 arXiv

  38. [46]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Qwen Team, and Alibaba Group et al. Qwen2 technical report. 2024

    2024

  39. [47]

    Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao

    Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. 2022

    2022

  40. [48]

    Scaling laws for data- efficient visual transfer learning.arXiv preprint arXiv:2504.13219, 2025

    Wenxuan Yang, Qingqu Wei, Chenxi Ma, Weimin Tan, and Bo Yan. Scaling laws for data- efficient visual transfer learning.arXiv preprint arXiv:2504.13219, 2025. 13 Appendix Contents A General Loss Function Metric Formulas 15 A.1 Standardizing Hyperparameter Settings . . . . . . ....

    2025 arXiv

  41. [49]

    Optimization Hyperparameters:Primarily the learning rate ( lr) and batch size ( bs), which are set to suitable values (potentially following scale-aware schedules or determined through preliminary sweeps) to ensure model training does not deviate significantly or exhibit marke...

  42. [50]

    Architectural Hyperparameters:The specific configuration defining the LLM family A (e.g., layer counts, hidden dimensions relative to N, activation functions) must be consis- tently defined or scaled according to precise rules. By carefully controlling these factors, we ensure...

    2000

  43. [51]

    Stage 1: Initial Estimation of AN and BN : For each model size N, the parameters AN and BN of the data-dependent term B(N)D −A(N) are estimated by analyzing the finite difference∆ DL(N, D), following the specific regression steps detailed in Algorithm 2

  44. [52]

    Stage 2: Parameterization and Iterative Refinement of fA(N;θ A) and fB(N;θ B): Continuous functions fA(N;θ A) and fB(N;θ B) are derived by fitting the discrete {AN } and {BN } estimates. While Algorithm 2 specifies particular functional forms (stretched- exponentials), the und...

  45. [53]

    Minimized Error

    Stage 3: Fitting the Model-Dependent Residual E+U(N) : The residual term E+U(N) is estimated (typically by averaging L(N, D)−B(N;θ ∗ B)D−A(N;θ ∗ A) over D to get G(N)) and modeled as fU (N;θ ∗ U ). Similar to Stage 2, Algorithm 2 employs a specific stretched- exponential funct...

    2024

  46. [54]

    Differentiating with respect toD, we obtain: ∂L ∂D = ∂ ∂D h U(N) +B(N)D −A(N) i =−A(N)B(N)D −A(N)−1

    Partial derivative w.r.t.D. Differentiating with respect toD, we obtain: ∂L ∂D = ∂ ∂D h U(N) +B(N)D −A(N) i =−A(N)B(N)D −A(N)−1 . This expression is strictly negative since all multiplicative components— A(N), B(N) , and D−A(N)−1 —are positive for admissible values of N and D....

  47. [55]

    Architecture Family

    Partial derivative w.r.t.N. By the product and chain rules, ∂L ∂N =U ′(N) +B ′(N)D −A(N) +B(N) d dN D−A(N) . Because the analytical form of ∂L/∂N is prohibitively complex, we resort to numerical verification to determine its sign. As illustrated in Fig. 20, the numerically com...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.