Skaling: Chinchilla's Exponents Meet Kaplan's Coupling

Badr Youbi-Idrissi; David Lopez-Paz; Kartik Ahuja; Mathurin Videau

REVIEW 4 major objections 6 minor 16 references

Skaling: Chinchilla's Exponents Meet Kaplan's Coupling

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single coupling exponent between model size and training data—the Skaling law—removes the boundary bias of additive scaling laws and cuts prediction error by 1.5–3×.

desk verdict The paper's headline claim is overreaching: the measured N–D coupling may be a StepLaw schedule artifact, and the paper's own dominated-pair fit has Chinchilla beating Skaling on Farseer far extrapolation; still, the L-shape sampling and careful evaluation make it worth refereeing. read the letter →

arxiv 2608.07222 v1 pith:5ACROKCS submitted 2026-08-07 cs.CL

Mathurin Videau , Badr Youbi-Idrissi , David Lopez-Paz , Kartik Ahuja This is my paper · ORCID

classification cs.CL

keywords neuralscalinglawslanguagemodelpretrainingcompute-optimalallocationsize-datacouplingmixedpartialderivativeextrapolationsparsegridprofilingSkalinglaw

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the standard additive Chinchilla scaling law fails at the extremes of the model-size/data grid because it assumes model size and training data affect loss independently, forcing the cross-derivative to zero. Using derivative estimates on two large training grids, it shows the loss surface has a nonzero negative mixed derivative, and proposes the Skaling law, $L(N,D)=(A/N^\alpha+B/D^\beta)^k+E$, which adds one coupling exponent $k$ to Chinchilla's form. With that one parameter, extrapolation MAPE drops 1.5–3× across regimes, and an L-shaped sampling grid restricted to cheap runs predicts as well as a full-grid Chinchilla fit at approximately 10× less compute. If right, this gives laboratories a cheaper, more reliable way to choose model size and data budgets, and it revives Kaplan's coupling idea without abandoning Chinchilla's tractable structure.

What carries the argument

The central object is the Skaling functional form and its mixed derivative. Writing $u=A/N^\alpha+B/D^\beta$, the form $L=u^k+E$ has $\partial^2 L/\partial N\partial D = k(k-1)u^{k-2}\alpha A N^{-\alpha-1}\beta B D^{-\beta-1}$, which is nonzero for $k\neq 1$ and negative for $0<k<1$, matching the measured sign while keeping both first derivatives negative. The paper uses moving-least-squares and Gaussian-process surrogates to estimate first and second derivatives directly from runs, and the L-shape sampling strategy exploits the asymptotic limits ($D\to\infty$ isolates the $N$ terms; $N\to\infty$ isolates the $D$ terms) to fit parameters from cheap edge runs. The same machinery yields the theoretical allocation formula $R_{\mathrm{opt}} = 6^{(\beta-\alpha)/(\alpha+\beta)}(\beta B/\alpha A)^{2/(\alpha+\beta)}C^{(\alpha-\beta)/(\alpha+\beta)}$, inherited from Chinchilla's stationarity condition.

What would settle it

Train a small grid with a deliberately different recipe, such as constant learning rate and batch size independent of $N$ and $D$, and estimate $\partial^2 L/\partial N\partial D$ with the same MLS and GP surrogates; if the mixed derivative vanishes or changes sign, the coupling is a recipe artifact. Alternatively, fit Skaling on a grid generated with hyperparameters known to remove cross-variable effects: a fitted $k\approx 1$ with boundary errors returning to Chinchilla's level would falsify the claim that the form itself captures an intrinsic interaction.

Watch

Extended reading notes

Core claim

Model size and training data do not act independently on the loss; the mixed derivative $\partial^2 L/\partial N\partial D$ is measurably negative across the grid, which no additive law can represent. The Skaling form restores interaction through an outer exponent $k$, with $0<k<1$ observed (about $0.31$–$0.45$ on the main grids), yielding a strictly sub-additive coupling that keeps loss monotone decreasing in both $N$ and $D$. This removes the saddle-shaped boundary bias of Chinchilla, and because the compute-optimal stationarity condition $Z'(N)=0$ is unchanged by the outer map, Skaling inherits Chinchilla's closed-form allocation formula even though fitted parameters shift the recommended $D^*/N^*$; empirical GP and MLS gradient estimates on Farseer support Skaling's decreasing token-to-parameter ratio with compute rather than Chinchilla's near-flat one.

Load-bearing premise

The load-bearing premise is that the nonzero mixed derivative measured on these finite, noisy training grids is a property of the model family itself, not an artifact of the StepLaw hyperparameter schedule or of the specific runs; if the recipe created the coupling, Skaling's advantage and its allocation advice would not transfer to other training setups.

Editorial extensions

If this is right

If the coupling is real, additive scaling laws systematically mispredict performance for imbalanced $N$–$D$ configurations, so allocation decisions based on them are wrong at scale; Skaling's fitted parameters imply the optimal token-to-parameter ratio drifts with compute, with the direction of the drift depending on the dataset.
Sparse L-shape profiling becomes a viable substitute for dense grids: training only low-compute edges recovers full-grid accuracy with about 10× less compute, lowering the cost of scaling-law studies.
The improvement does not come from adding parameters (the nine-parameter Farseer law is no better in most regimes) but from matching the form of the interaction, so future scaling laws should test for nonzero mixed derivatives before choosing a form.
Because $k=1$ recovers Chinchilla, Skaling degrades gracefully to additive behavior when data show little coupling, making it a safe drop-in replacement in settings where coupling is weak.
The paper expects similar couplings to matter along other scaling axes such as data mixtures and repeated data, where additive forms may also fail at the extremes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the measured coupling may be partly produced by the StepLaw hyperparameter schedule; the paper itself notes that different recipe parameterizations alter the apparent $N$–$D$ interaction, so Skaling would likely need recipe-specific refits whenever the optimizer, learning-rate policy, or batch-size policy changes.
Editorial inference: the $k$–$E$ tradeoff means irreducible-loss estimates are not directly comparable across laws; reporting $E$ alone as a loss floor could mislead, and dominated-pair fitting offers a way to separate shape from floor when comparing functional forms.
Editorial inference: the same one-parameter coupling could serve as a cheap screening tool on a new architecture: fit Skaling versus Chinchilla on a small L-shape grid; if $k$ is close to 1, additive planning is safe, and if $k<1$, the coupled allocation formula should be used.
Editorial inference: the claimed 10× compute saving assumes the L-shape grid's edge runs are representative of the full surface; a natural stress test is to hold out an interior band and check whether Skaling still predicts it from edges alone.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the paper's headline claim is overreaching. The measured N–D coupling may come from the StepLaw hyperparameter schedule, and the paper's own dominated-pair fit has Chinchilla beating Skaling on Farseer far extrapolation. That said, the L-shape sampling result is genuinely useful, and the paper is honest enough to include the evidence against its own headline.

What's new: Busbridge et al. already used the form E+(A/N^alpha+B/D^beta)^k, but this paper is the first to study it standalone as a supervised scaling law, with careful cross-validation, a mixed-derivative diagnostic, and a sparse L-shape sampling protocol. The same-optimizer comparison against Chinchilla and Farseer is fair, and the compute-extrapolation experiment in Table 3 is a nice operational test.

Soft spots: first, the schedule confound. Equation (9) sets LR and batch size as power laws in N and D, so the fitted surface is L(N,D; eta(N,D), B(N,D)). The mixed derivative conflates architectural interaction with schedule-induced interaction. Section E.2 admits this: 'different parameterizations of hyperparameters in the training recipe can alter the apparent N–D interaction and shift the inferred compute-optimal ratio.' That is a load-bearing concession: the claimed 1.5-3x advantage and the allocation recommendations may be properties of the StepLaw recipe, not of the model family. Second, the headline claim is not uniform: Table 9 shows Chinchilla with dominated-pair fitting achieves 0.79% far-extrapolation MAPE on Farseer versus Skaling's 2.31%. The authors include this, but the abstract does not. Third, SK-Grid is internal and not released, so the strongest result is not independently checkable. Fourth, the derivative evidence (Figure 3) is presented without uncertainty quantification, even though the GP section describes how to compute it.

In proportion: the central idea is still defensible as a recipe-specific scaling form. The paper is a solid empirical study with an overstated generality claim. The formal derivation of the compute-optimal allocation is correct and independent of k, and the L-shape protocol is a practical contribution.

Who should read it: anyone fitting scaling laws or allocating pretraining compute. It deserves serious peer review; the referee should push for re-scoped claims and ideally experiments with fixed hyperparameters.

Referee Report

4 major / 6 minor

Summary. The paper proposes replacing the additive Chinchilla scaling law L(N,D)=A/N^alpha+B/D^beta+E with a coupled form L(N,D)=(A/N^alpha+B/D^beta)^k+E, arguing that a single interaction exponent captures an N–D coupling that additive laws miss. The authors support this with derivative diagnostics on the Farseer grid, cross-validation on two LLM pretraining grids, an L-shape sparse sampling strategy claimed to match full-grid accuracy at roughly 10x less compute, and a derivation showing that the closed-form compute-optimal allocation is inherited from Chinchilla. Appendices add results on Farseer-code and replicated Chinchilla data, plus a dominated-pair fitting alternative.

Significance. If the empirical advantage holds under controlled hyperparameter variation, the Skaling form is a simple and practical improvement for profiling and compute-allocation experiments. The paper has real strengths: two independently collected grids, a uniform fitting protocol across laws, multiple cross-validation regimes, an exact stationarity derivation, and honest caveats in Section E.2 about recipe dependence. The main reservations are that the measured coupling is confounded with the StepLaw hyperparameter schedule and that the headline 1.5–3x MAPE claim is not uniform across the paper's own tables, including a dominated-pair fit in which Chinchilla outperforms Skaling in far extrapolation on Farseer.

major comments (4)

[§2, §E.2, Eq. (9)] The derivative evidence that N and D interact is measured on runs whose learning rate and batch size are themselves functions of N and D: eta=0.0709 F^{-0.4303} D^{0.2785} and B=896.07 F^{0.231} (Eq. 9). The fitted surface is therefore L(N,D; eta(N,D), B(N,D)), and the estimated mixed derivative conflates an architectural interaction with a schedule-induced interaction. Section E.2 concedes that different hyperparameter parameterizations "can alter the apparent N–D interaction and shift the inferred compute-optimal ratio" and that cross-dataset differences partly reflect the training recipes. This is load-bearing because all downstream claims—L-shape efficiency, allocation recommendations, transferability to other setups—assume that the coupled form is the right inductive bias of the model family, not merely a description of one hyperparameter policy. A minimal additional experiment that varies the hyperparameter recipe, or at least a reframing of the claims to the StepLaw recipe, is needed to separate these; as it stands, Figure 3 cannot by itself justify the causal reading that additive forms fail because model size and data act independently on the architecture's loss.
[Appendix F, Table 9] The dominated-pair fit in Table 9 is the most serious internal challenge to the paper's causal story. With the floor E removed from the objective, Chinchilla's Farseer far-extrapolation MAPE drops from 2.46% to 0.79%, while Skaling's is 2.31% (or 1.55% with dominated-pair fitting); on SK-Grid full, Chinchilla's far error drops from 5.17% to 3.67%. The paper itself concludes that "much of Chinchilla's extrapolation error stems from weak identification of the floor E." That conclusion is in tension with the main-text claim that the additive form's zero cross-derivative is the structural cause of the boundary bias, because a better estimator of the same additive form removes most of that bias. The main text should report this robust-fit baseline alongside Table 1 and reconcile the two narratives.
[Table 1, Table 5, Table 6; Abstract] The abstract and Section 4.2 claim a 1.5–3x MAPE reduction across interpolation and extrapolation, but the paper's own tables show this is not uniform. On Farseer full-grid far extrapolation the ratio is 2.46/2.31 ≈ 1.06; on Farseer-code extrapolation-N it is 0.93/0.67 ≈ 1.39; and on the replicated Chinchilla data Skaling is worse than Chinchilla on extrapolation-N (1.28 vs 1.16). The additional datasets in Table 6 yield k≈0.77–0.90, close to additive, which the paper attributes to recipe differences. The headline claim should be qualified to the regimes and datasets where it holds, with far-extrapolation and near-additive cases reported as exceptions rather than being covered by the 1.5–3x statement.
[Section 4.2, Table 1] The "approximately 10x less compute" claim for the L-shape strategy is not supported by SK-Grid. Table 1 shows fitting compute 3.1x10^21 FLOPs (full grid) versus 6.5x10^20 FLOPs (L-shape), a factor of about 4.8, whereas Farseer is about 9.8x. The text in Section 4.2 says "roughly 10x less fitting compute" without noting this variation, and the abstract states "approximately 10x less compute" without qualification. The compute-savings claim should be stated per dataset or as a range.

minor comments (6)

[Section 4.1 vs Section E.1, Table 7] SK-Grid is described as 134 configurations across 15 model sizes in Section 4.1, but Section E.1 says 125 runs across 14 model sizes, and the Budgets column of Table 7 sums to 125; please reconcile these numbers and ensure the reported compute totals use the correct configuration count.
[Figure 3] Figure 3 plots the absolute value of the mixed derivative, but the text describes the sign as "predominantly negative"; please show the signed field or state explicitly how the sign is inferred, since a reader cannot verify the sign from the figure as printed.
[Section 3.1 vs Figure 6] Section 3.1 says the allocation discrepancy accumulates to a "100-fold discrepancy at frontier compute scales," while Figure 6 and its caption report "more than 10x" (Chinchilla near 380 tokens per parameter versus 20–40 for Skaling and the empirical fits); these numbers should be made consistent.
[Figure 1] Please clarify whether the error maps in Figure 1 are in-sample fits on the full grid or cross-validated predictions; if they are in-sample, they are descriptive and should not be presented as predictive evidence without the cross-validation context of Table 1.
[Section 3.1] There is a duplicated sentence fragment: "We give the full derivation in Section B. Section B for the full derivation)."
[Section E.2] There is a typo: "the specific training recipes aand data" should be "and data."

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Skaling is a cross-validated empirical fit; the disclosed StepLaw confound is a validity risk, not a circular reduction.

full rationale

The Skaling law is a parametric extension fitted to measured losses and evaluated with repeated cross-validation on multiple grids, including external Farseer, Farseer-code, and Chinchilla datasets, so the MAPE gains are out-of-sample empirical comparisons rather than fitted inputs renamed as predictions. The mixed-derivative analysis that motivates the form is model-free (MLS and GP surrogates), and the later MAPE comparisons do not fit k to the derivative target and then predict the derivative back; the fitted k is obtained from the loss surface directly. The compute-optimal allocation derivation is exact: the stationarity condition reduces to Z'(N)=0, which is independent of k, so the claim that Skaling inherits Chinchilla's closed-form allocation is a mathematical identity, not a circular step. The GP/MLS empirical optimum is an internal consistency check on the same data, but it is not a self-definitional prediction because the empirical ratio is estimated without the parametric law and the comparison is presented as corroboration rather than as a deduction. The one substantial caveat is Section E.2, where the authors admit that different hyperparameter parameterizations can alter the apparent N-D interaction and shift the inferred optimal ratio; this is a confounding and generalization threat, not a circular reduction, because Eq. (9) is an input recipe, not a definition of the Skaling law or of the coupling claim. Self-citations are limited to implementation details (Meta Lingua) and are not load-bearing. No step in the derivation chain reduces by construction to its own inputs, so the circularity score is 0.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

All six parameters of the Skaling law are fitted to measured validation losses, so the central improvement claims rest on fitted values. The analysis also assumes smoothness of the loss surface and that the hyperparameter schedule does not create the interaction. No new physical or conceptual entities are introduced.

free parameters (6)

A = 2.9e2 (Farseer full), 5.3e6 (SK-Grid full)
Amplitude of the model-size term in Equation 3, fitted to the loss grid; varies by dataset.
B = 6.0e3 (Farseer full), 7.1e6 (SK-Grid full)
Amplitude of the data term, fitted; per Table 2.
alpha = 0.32 (Farseer full), 0.73 (SK-Grid full)
Inner exponent on model size, fitted.
beta = 0.39 (Farseer full), 0.63 (SK-Grid full)
Inner exponent on data, fitted.
k = 0.41 (Farseer full), 0.31 (SK-Grid full), 0.77-0.90 on Farseer-code and Chinchilla data
The single coupling exponent at the heart of the claim; if k were 1 the law reduces to Chinchilla. Fitted, not derived.
E = 0.03 (Farseer full), 1.14 (SK-Grid full)
Irreducible loss; weakly identified and trades off against k as the paper acknowledges.

assumptions (4)

domain assumption The loss surface is smooth enough that MLS polynomial and GP surrogate derivatives recover the true partial and mixed derivatives.
Section 2 and A.1 use these estimates to conclude the mixed derivative is nonzero; no statistical test or error bars are reported.
domain assumption StepLaw hyperparameter prescriptions at every (N,D) produce a loss surface that reflects the architecture's scaling behavior, not the tuning rule.
Section E.2 says different hyperparameter policies can alter the apparent N-D interaction; if true, the coupling is partly a property of the recipe.
domain assumption The compute budget is C=6ND and the log-slope balance condition identifies the compute-optimal allocation.
Section B and A.3; standard in the field but an assumption about FLOP accounting and differentiability.
standard math Taylor expansion, weighted ridge regression, and GP differentiation are valid tools for these finite grid points.
Appendix A.1; standard, but their reliability depends on neighborhood size, kernel length scales, and noise.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Skaling: Chinchilla's Exponents Meet Kaplan's Coupling." pith.science (2026). https://pith.science/paper/5ACROKCS

@misc{pith2026260807222,
  author       = {Pith},
  title        = {Pith review of: Skaling: Chinchilla's Exponents Meet Kaplan's Coupling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5ACROKCS}},
  note         = {Machine review of arXiv:2608.07222}
}

read the original abstract

Neural scaling laws are foundational for language model development, yet standard formulations systematically under- and overestimate loss at data-scarce and overtraining extremes. This failure originates in the underlying assumption that model size and training data impact the loss independently. To address this, we introduce the Skaling law, a generalized functional form that couples model capacity and data through a single interaction exponent. This simple extension reduces the Mean Absolute Percentage Error (MAPE) by 1.5-3x across both interpolation and extrapolation regimes. When paired with a sparse grid strategy restricted to low-compute regimes, the Skaling law achieves accurate full-grid extrapolation using approximately 10x less compute than uniform sweeps. By enabling reliable performance prediction from small-scale experiments, the Skaling law provides a more robust and resource-efficient framework for allocating compute budgets in next-generation model training.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 4 canonical work pages

[1]

How text quality interventions reshape neural scaling laws for LLMs: Empirical study

NewshaArdalani, FeiyangKang, MichaelKuchnik, MostafaElhoushi, ShubhabrataSengupta, Shang-WenLi, andCarole- Jean Wu. How text quality interventions reshape neural scaling laws for LLMs: Empirical study. InThe Fourteenth International Conference on Learning Representations, 2026.https://openreview.net/forum?id=ZC5QBfdOw7. Tamay Besiroglu, Ege Erdil, Matthew...

arXiv 2026
[2]

Differentiatingln|∂L/∂N| and ln|∂L/∂D| gives γN = (1 −k ) β wD and γD = (1 −k ) α wN, soγN /γD = (β/α) (wD/wN )

Let wD = BD −β/u and wN = AN −α/u denote the data and size shares of the inner sum. Differentiatingln|∂L/∂N| and ln|∂L/∂D| gives γN = (1 −k ) β wD and γD = (1 −k ) α wN, soγN /γD = (β/α) (wD/wN ). This ratio exceeds one whenever β > αand the inner sum is data-leaning, reproducing the measuredγN ≈ 0.13 > γD ≈ 0.07through the single coupling exponent rather...

work page 2022
[3]

Distillation scaling laws.arXiv preprint arXiv:2502.08606,

Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, and Russ Webb. Distillation scaling laws.arXiv preprint arXiv:2502.08606,

arXiv
[4]

+dom”) fitting versus the default joint L-BFGS fit, for Chinchilla and Skaling on every dataset and grid. Each “+dom

Vocabulary128,256 Sequence len.2048 Optimization Optimizer AdamW (β1, β2) (0.9,0.95) Weight decay0.1 Grad. clip0.1 LR schedule cosine Warmup10% Final LR1×10 −6 18 Table 9Dominated-pair (“+dom”) fitting versus the default joint L-BFGS fit, for Chinchilla and Skaling on every dataset and grid. Each “+dom” row is the dominated-pair fit (Equation (10)) of the...

work page
[7]

For instance, a broadly mistuned grid might artificially dampen the measured interaction or skew the optimal ratio

E.2 Hyperparameters The two run-specific hyperparameters, the global batch sizeB (tokens) and the peak learning rateη, follow the StepLaw prescription (Li et al., 2025b) as power laws in the per-token computeF (FLOPs per token, a monotone proxy for model sizeN) and the token budgetD, with coefficients refit to our own setup: B= 896.07F 0.231, η= 0.0709F −...

work page 2024
[10]

Resolving discrepancies in compute-optimal scaling of language models.arXiv preprint arXiv:2406.19146,

Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, and Yair Carmon. Resolving discrepancies in compute-optimal scaling of language models.arXiv preprint arXiv:2406.19146,

arXiv
[11]

A constructive prediction of the generalization error across scales.arXiv preprint arXiv:1909.12673,

Jonathan S Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales.arXiv preprint arXiv:1909.12673,

arXiv 1909
[13]

At each grid point, we estimate ∂z/∂x j using two mesh-free procedures: a local moving least-squares estimator (MLS) and a global Gaussian- process estimator (GP)

11 Appendix A Differential analyses A.1 Numerical gradients We estimate derivatives of the loss surface directly from the measured runs. At each grid point, we estimate ∂z/∂x j using two mesh-free procedures: a local moving least-squares estimator (MLS) and a global Gaussian- process estimator (GP). Moving least squares (MLS).The local estimator fits a tr...

work page 1981

Show all 16 references

[1981]

Predictable scale: Part ii, farseer: A refined scaling law in large language models

Houyi Li, Wenzhen Zheng, Qiufeng Wang, Zhenyu Ding, Haoying Wang, Zili Wang, Shijie Xuyang, Ning Ding, Shuigeng Zhou, Xiangyu Zhang, et al. Predictable scale: Part ii, farseer: A refined scaling law in large language models. arXiv preprint arXiv:2506.10972, 2025a. Houyi Li, We...

arXiv
[2016]

Deep learning scaling is predictable, empirically.arXiv preprint arXiv:1712.00409,

Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Pat- wary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically.arXiv preprint arXiv:1712.00409,

arXiv
[2017]

Training compute-optimal large language models.arXiv preprint arXiv:2203.15556, 10,

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, DDL Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models.arXiv preprint arXiv:2203.15556, 10,

arXiv
[2022]

Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,

2001 arXiv
[2023]

Reconciling kaplan and chinchilla scaling laws.arXiv preprint arXiv:2406.12907,

Tim Pearce and Jinyeop Song. Reconciling kaplan and chinchilla scaling laws.arXiv preprint arXiv:2406.12907,

arXiv
[2024]

Deepseek llm: Scaling open-source language models with longtermism.arXiv preprint arXiv:2401.02954,

Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek llm: Scaling open-source language models with longtermism.arXiv preprint arXiv:2401.02954,

arXiv
[2025]

The cma evolution strategy: A tutorial.arXiv preprint arXiv:1604.00772,

Nikolaus Hansen. The cma evolution strategy: A tutorial.arXiv preprint arXiv:1604.00772,

arXiv
[2026]

Meta Lingua: A minimal PyTorch LLM training library, 2024.https://github.com/facebookresearch/lingua

Mathurin Videau, Badr Youbi Idrissi, Daniel Haziza, Luca Wehrstedt, Jade Copet, Olivier Teytaud, and David Lopez- Paz. Meta Lingua: A minimal PyTorch LLM training library, 2024.https://github.com/facebookresearch/lingua. Jiasheng Ye, Peiju Liu, Tianxiang Sun, Yunhua Zhou, Jun ...

2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

How text quality interventions reshape neural scaling laws for LLMs: Empirical study

NewshaArdalani, FeiyangKang, MichaelKuchnik, MostafaElhoushi, ShubhabrataSengupta, Shang-WenLi, andCarole- Jean Wu. How text quality interventions reshape neural scaling laws for LLMs: Empirical study. InThe Fourteenth International Conference on Learning Representations, 2026.https://openreview.net/forum?id=ZC5QBfdOw7. Tamay Besiroglu, Ege Erdil, Matthew...

arXiv 2026

[2] [2]

Differentiatingln|∂L/∂N| and ln|∂L/∂D| gives γN = (1 −k ) β wD and γD = (1 −k ) α wN, soγN /γD = (β/α) (wD/wN )

Let wD = BD −β/u and wN = AN −α/u denote the data and size shares of the inner sum. Differentiatingln|∂L/∂N| and ln|∂L/∂D| gives γN = (1 −k ) β wD and γD = (1 −k ) α wN, soγN /γD = (β/α) (wD/wN ). This ratio exceeds one whenever β > αand the inner sum is data-leaning, reproducing the measuredγN ≈ 0.13 > γD ≈ 0.07through the single coupling exponent rather...

work page 2022

[3] [3]

Distillation scaling laws.arXiv preprint arXiv:2502.08606,

Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, and Russ Webb. Distillation scaling laws.arXiv preprint arXiv:2502.08606,

arXiv

[4] [4]

+dom”) fitting versus the default joint L-BFGS fit, for Chinchilla and Skaling on every dataset and grid. Each “+dom

Vocabulary128,256 Sequence len.2048 Optimization Optimizer AdamW (β1, β2) (0.9,0.95) Weight decay0.1 Grad. clip0.1 LR schedule cosine Warmup10% Final LR1×10 −6 18 Table 9Dominated-pair (“+dom”) fitting versus the default joint L-BFGS fit, for Chinchilla and Skaling on every dataset and grid. Each “+dom” row is the dominated-pair fit (Equation (10)) of the...

work page

[5] [7]

For instance, a broadly mistuned grid might artificially dampen the measured interaction or skew the optimal ratio

E.2 Hyperparameters The two run-specific hyperparameters, the global batch sizeB (tokens) and the peak learning rateη, follow the StepLaw prescription (Li et al., 2025b) as power laws in the per-token computeF (FLOPs per token, a monotone proxy for model sizeN) and the token budgetD, with coefficients refit to our own setup: B= 896.07F 0.231, η= 0.0709F −...

work page 2024

[6] [10]

Resolving discrepancies in compute-optimal scaling of language models.arXiv preprint arXiv:2406.19146,

Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, and Yair Carmon. Resolving discrepancies in compute-optimal scaling of language models.arXiv preprint arXiv:2406.19146,

arXiv

[7] [11]

A constructive prediction of the generalization error across scales.arXiv preprint arXiv:1909.12673,

Jonathan S Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales.arXiv preprint arXiv:1909.12673,

arXiv 1909

[8] [13]

At each grid point, we estimate ∂z/∂x j using two mesh-free procedures: a local moving least-squares estimator (MLS) and a global Gaussian- process estimator (GP)

11 Appendix A Differential analyses A.1 Numerical gradients We estimate derivatives of the loss surface directly from the measured runs. At each grid point, we estimate ∂z/∂x j using two mesh-free procedures: a local moving least-squares estimator (MLS) and a global Gaussian- process estimator (GP). Moving least squares (MLS).The local estimator fits a tr...

work page 1981