Pith. sign in

REVIEW 4 major objections 5 minor 46 references

Sub-Scaling Laws: On the Role of Data Density and Training Strategies in LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that standard scaling laws under-predict loss for over-trained models and dense data, and it offers a sub-optimal scaling law that bends with the over-training ratio.

desk verdict A plausible empirical extension of scaling laws to over-trained and redundant-data regimes, with a new OTR-based correction that deserves referee time, but the data-density pillar is underspecified and the law is a high-parameter fit. read the letter →

arxiv 2507.10613 v1 pith:QOWDQJIF submitted 2025-07-13 cs.LG cs.AI

Zhengyu Chen , Siqi Wang , Teng Xiao , Yudong Wang , Shiqi Chen , Xunliang Cai , Junxian He , Jingang Wang This is my paper
classification cs.LGcs.AI MSC 68T0768T50
keywords scalinglawssub-scalingdatadensityover-trainingratioChinchillalawresourceallocationredundancy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the usual scaling-law story, in which loss falls as a power of model size and data alone, is incomplete. It identifies two conditions that break that story: datasets whose samples are very similar to one another, which the paper calls high density, and training runs that use many more tokens than parameters than the compute-optimal allocation, which it calls over-training. Through experiments on over 400 models ranging from 20 million to 7 billion parameters, it shows that in these regimes the observed loss curve bends away from the standard Chinchilla prediction. The paper proposes a sub-optimal scaling law with logistic correction terms that grow with the over-training ratio, and reports that this law fits and predicts the loss in sub-scaling regimes more accurately than the standard law. If the law is right, anyone extrapolating loss for over-trained models or redundancy-heavy datasets has a practical correction that predicts the bend rather than missing it.

What carries the argument

The paper's central objects are a data-density metric and a corrected scaling law. The density of a cluster is defined as the number of samples divided by the volume of an $n$-dimensional ball with the cluster's average radius, and the dataset density weights cluster-level densities by centroid separation, following Eqs. (1)-(3); the paper uses this metric to construct low-density datasets from The Pile, reporting densities of 0.64 for the raw Pile, 0.56 after deduplication, and 0.47 after density-based selection. The second engine is the sub-optimal scaling law, which keeps the Chinchilla power-law form but multiplies the parameter and data terms by logistic repetition factors $R_N$ and $R_D$ that increase with $\mathrm{OTR}=D/N$, thereby encoding diminishing returns from redundant data and over-training.

What would settle it

Train matched models on datasets whose densities are computed with two different clustering or embedding choices, and compare density-based selection against exact n-gram deduplication; if the density ranking flips, or if the low-density advantage in the loss curve disappears, the claimed causal role of density fails. Alternatively, fit the sub-optimal law on runs with OTR at most 750 and extrapolate to OTR above 1700: if prediction error grows monotonically with OTR, the logistic saturation form is wrong.

Watch

Extended reading notes

Core claim

The central claim is that sub-scaling, meaning performance improvements that decelerate faster than power-law extrapolation predicts, is driven by high data density and non-optimal model-to-data allocation, and that this deceleration can be quantitatively captured by extending the Chinchilla law. The paper proposes the sub-optimal scaling law $L(N,D) = E + \lambda_N R_N/N^{\alpha_N} + \lambda_D R_D/D^{\alpha_D}$, where $R_N$ and $R_D$ are logistic functions of the over-training ratio $\mathrm{OTR}=D/N$, so that the effective contribution of extra parameters or extra tokens saturates as the allocation becomes increasingly imbalanced. Fitted across model sizes from 20M to 7B parameters, this law reports lower fitting and prediction error than the standard Chinchilla and OpenAI scaling laws in over-trained regimes, for example reducing prediction MAPE from 0.0300 to 0.0013 on a fixed-compute allocation extrapolation. The paper treats this as evidence that diversity and allocation, not only raw scale, govern scaling behavior.

Load-bearing premise

The load-bearing premise is that the clustering-based density metric in Eqs. (1)-(3) genuinely measures data redundancy; the paper never specifies the clustering algorithm, embeddings, dimension $n$, or cluster count $K$, and its own limitations section concedes the results may not transfer across architectures or training regimes.

Editorial extensions

If this is right

  • For training runs whose token-to-parameter ratio exceeds the compute-optimal value, the corrected law predicts higher loss than standard scaling laws, so compute budgets and extrapolation curves can be set more honestly.
  • Density-based data selection can restore more linear performance gains: the lower-density dataset in the paper shows less sub-scaling than the raw Pile, suggesting redundancy removal is a concrete lever on scaling behavior.
  • At a fixed over-training ratio, loss still follows a power law in compute, and the exponent stabilizes for OTR between roughly 50 and 1700, defining a stable over-training regime in which extra tokens have little effect on the scaling exponent.
  • The sub-optimal scaling law gives over-trained small models a quantitative way to predict when additional training tokens stop paying off, which is useful for deciding how long a fixed model should be trained before a larger model becomes worthwhile.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit is that if density truly drives sub-scaling, then exact and fuzzy deduplication pipelines should produce the same ranking of datasets as Eqs. (1)-(3); testing that equivalence is a cheap external check on the density metric.
  • The same logistic-correction device could be fitted to downstream task losses rather than pretraining loss, and because tasks differ in reasoning depth, the shape parameters $k_1$ and $k_2$ would likely differ across tasks, making the correction task-specific.
  • A testable extension suggested by the paper's logic is that, for a fixed FLOP budget, low-density data plus over-training should reach a target loss at lower cost than high-density data plus compute-optimal allocation, since density controls the marginal gain of additional tokens.
  • If the density metric survives scrutiny, it could be used prospectively to rank candidate pretraining corpora before training, treating density as a measurable predictor of how fast a corpus will stop yielding improvements.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper argues that traditional scaling laws fail in two sub-scaling regimes: high data density (redundant training data) and non-optimal resource allocation (over-training). It introduces a cluster-based data density metric (Eqs. 1–3), reports density values for The Pile, Deduplicated Pile, and Density-Based Pile, and proposes a sub-optimal scaling law L(N,D) = E + λ_N R_N / N^{α_N} + λ_D R_D / D^{α_D}, where R_N and R_D are logistic functions of the over-training ratio OTR = D/N. The law is fit to training-loss curves from models between 20M and 7B parameters and evaluated by MAPE against Hoffmann and OpenAI scaling laws. The paper also reports experiments on batch-size and learning-rate scaling under over-training. The central abstract claim is that high data density and non-optimal allocation cause sub-scaling, and that the proposed law predicts loss better than Chinchilla in sub-scaling regimes.

Significance. If the density pillar were established, the paper would provide a practical data-quality metric and a correction to Chinchilla-style extrapolation for over-trained models. The empirical sweep is large (over 400 models), the code is released, and the prediction protocol in Appendix F is a genuine extrapolation for sub-1B models: the first quarter of each training run is used to predict later loss, and for larger models the fit uses smaller models plus the first quarter of the target run. These are real strengths. However, the significance is currently conditional: the density metric is underspecified and scale-dependent, the data-density comparison in Section 3.1 uses a per-dataset free parameter, and the sub-optimal law is validated only with point MAPE values, without seeds or uncertainty estimates. The non-optimal/OTR pillar is more defensible than the density pillar, but the paper's headline claim requires both.

major comments (4)
  1. [§2.1, Eqs. (1)–(3)] The density metric is not sufficiently specified to support the central claim that high density causes sub-scaling. The manuscript does not state the embedding model, embedding normalization, dimension n, clustering algorithm, number of clusters K, or the procedure used to construct Density-Based Pile. The metric is also not scale-invariant: if all embedding coordinates are multiplied by a constant s, then r_i and R scale by s and ρ_i and ρ scale by s^{-n}, so the reported values 0.64, 0.56, and 0.47, and any density-based selection threshold, depend on an arbitrary coordinate scale. Without this specification, the Figure 4 high- versus low-density comparison and the Appendix A.1 density table do not establish that high data density, rather than the authors' own construction, drives sub-scaling.
  2. [§3.1, Eq. (8) and Figure 8] The comparison in Section 3.1 is circular for the density claim. Equation (8), P = R_D · λ · C^α, treats R_D as a per-dataset fitted decay factor ('we could fit different density datasets with varying R_D values'), so the sub-optimal curve has a free parameter for every dataset and is expected to fit better than a fixed power law. Moreover, Eq. (8) is not the functional form derived in Eqs. (4)–(5): the information-theoretic model P(n) = P0(1 − e^{−β I0 n^{−α}}) is never directly tested. To validate the density pillar, the authors should either fix R_D as an explicit function of the density metric and test out-of-sample, or compare models of equal flexibility on held-out data.
  3. [§3.2, Eq. (9), Appendix F, Table 4, Figure 10] The sub-optimal scaling law has seven free parameters (E, λ_D, α_D, λ_N, α_N, k1, k2), and the paper reports only point MAPE values with no error bars, no seeds, and no parameter uncertainties. The 'held-out' evaluation for larger models uses all smaller models plus the first quarter of the target model's training curve, so the prediction is not fully independent of the target run. The improvement over Hoffmann/OpenAI laws could partly reflect the extra flexibility of the logistic factors. Please report repeated-seed means and standard deviations, parameter covariance, and comparisons against equally flexible baselines (for example, a power law whose exponents are allowed to depend on OTR) to show that the specific logistic form in Eqs. (12)–(13) is the reason for the improved MAPE.
  4. [Table 1, §3.2] The claim that the sub-optimal law outperforms the traditional scaling law is not uniformly supported by the paper's own table. In the 5-billion-token row, the prediction MAPE for the proposed sub-optimal law is 0.00887, which is worse than the traditional scaling law's 0.00757. This is a direct contradiction of the unqualified comparison in Section 3.2. The authors should either qualify the claim (e.g., the benefit appears only at large token counts) or explain why the 5B case is not representative.
minor comments (5)
  1. [Appendix A.1] The construction of Density-Based Pile is described only as 'we use metric density to select data from The Pile dataset,' with the details said to be in the Appendix, but no selection algorithm, density threshold, or cluster configuration is actually given; please provide the full procedure so the datasets are reproducible.
  2. [Appendix F] The final fitted constants (455.345, 61.929, 1.372, k1 = 0.00810, k2 = 0.00114) are reported without uncertainties or fitting details; please state the optimizer, loss function, initialization, and parameter covariance, and report the implied confidence bands for the Figure 10 predictions.
  3. [§2.2, Figure 7] The statement that the OTR threshold is 'positively correlated with the model size N' is asserted without supporting data or a statistical test; if this claim is retained, please provide the analysis.
  4. [Appendix D and E] There are several typos and inconsistent names: 'explaination' in Appendix D, 'raning' in Appendix E, 'Educational Valu' in Appendix G, and 'Hoffman Scaling Law' versus 'Hoffmann Scaling Law' in Table 4; please correct these throughout.
  5. [General] The manuscript says 'over 400 models' were trained, but no run log, seed count, or per-configuration repetition count is provided; adding this information would make the empirical sweep auditable and would support the error-bar analysis requested in the major comments.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the sub-optimal scaling law is an empirical parametric extension validated by held-out extrapolation, and the density metric concerns are correctness risks rather than circular reductions.

full rationale

The paper's central derivation is not circular by construction. Equation (9) is a parametric generalization of the Chinchilla form with logistic OTR-dependent factors R_N and R_D; these factors are fitted to training runs, but validation is genuinely out-of-sample: Appendix F states 'we use the first quarter of the training data to predict subsequent loss values,' and Figure 16 extrapolates to OTR>750 from fits on OTR<=750. The comparison against Llama3 8B-15T in Figure 10 also tests against external published models. Section 3.1's R_D is admittedly a per-dataset fit parameter ('we could fit different density datasets with varying RD values'), and Figure 8 is described as fitting rather than prediction; the paper even clarifies in Appendix C that 'The primary goal of Figure 8 was to illustrate the relationship ... rather than to validate the traditional scaling laws directly.' That weakens the evidentiary value of the 'better fit' claim but does not make it circular. The density metric in Eqs. (1)-(3) is underspecified and scale-dependent, which is a reproducibility and validity concern, not a self-justifying reduction: the density values are outputs of a stated formula, and the claim that lower-density datasets exhibit less sub-scaling is an empirical observation, even if the metric itself is questionable. The self-citations (e.g., Chen et al. 2025 for density-based data selection, Wang et al. 2024 for compute scaling) are related-work attributions and are not load-bearing for the new law. Overall, the manuscript does not exhibit a step where a prediction is equivalent to its fitted input by definition; the main risks are overfitting and metric validity rather than circularity.

Assumptions & free parameters 9 free parameters · 6 assumptions · 1 invented entities

The central claims rest on two families of assumptions: a toy information-theory model (P(n) as an exponential saturation of power-law information gain) and an empirical parametric loss law with seven fitted coefficients and logistic correction factors. The density metric is an invented measure whose validity is assumed. The paper contributes a large but underspecified empirical study, so the ledger is long relative to the theoretical content.

free parameters (9)
  • k1 (logistic steepness for R_D) = 0.00810
    Fitted in Appendix F to control how quickly the data repetition factor saturates with over-training ratio.
  • k2 (logistic steepness for R_N) = 0.00114
    Fitted in Appendix F for the model-size repetition factor.
  • lambda_D = 455.345
    Coefficient in the sub-optimal scaling law, fitted in Appendix F.
  • alpha_D = 0.289
    Exponent on training tokens D in the fitted law.
  • lambda_N = 61.929
    Coefficient for the model-size term in the fitted law.
  • alpha_N = 0.272
    Exponent on model size N in the fitted law.
  • E (irreducible loss) = 1.372
    Baseline loss term in the fitted sub-optimal law.
  • alpha (density information exponent)
    Exponent in I(n) = I0 n^{-alpha} for high-density data; the paper does not report its fitted value, but it is free and estimated from performance curves.
  • R_D per dataset in Eq. 8
    Decay factor fitted separately for density=0.64 and 0.47; values not reported.
assumptions (6)
  • ad hoc to paper P(n) = P0 (1 - e^{-beta I(n)}) relates information gain to performance.
    Assumed in Section 2.1 and Appendix B without derivation or external justification.
  • ad hoc to paper I(n) = I0 n^{-alpha} for high-density data and I(n) = I0 n for low-density data.
    Equations (4)-(5), modeling assumptions not derived from information theory.
  • domain assumption L(C) = lambda_C C^{-alpha_C} for loss versus compute.
    Equation (6), from Kaplan et al. (2020), accepted as a standard empirical scaling law.
  • ad hoc to paper The density metric in Eqs. (1)-(3) measures redundancy relevant to learning.
    No independent validation; the metric is defined and used to construct the datasets that are then used to test the hypothesis.
  • ad hoc to paper Logistic functions R_D and R_N capture over-training effects.
    Equations (12)-(13), chosen empirically, no derivation.
  • domain assumption The Pile and its derived subsets approximate web-scale pretraining data.
    All experiments use The Pile; generalization to other corpora is untested.
invented entities (1)
  • Dual intra/inter-cluster data density metric
    purpose: Quantify data redundancy and diversity to select low-density training data.
    Introduced in Eqs. (1)-(3) and used to construct Density-Based Pile; no external benchmark demonstrates it predicts training performance beyond this paper's own fits.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sub-Scaling Laws: On the Role of Data Density and Training Strategies in LLMs." pith.science (2026). https://pith.science/paper/QOWDQJIF

@misc{pith2026250710613,
  author       = {Pith},
  title        = {Pith review of: Sub-Scaling Laws: On the Role of Data Density and Training Strategies in LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QOWDQJIF}},
  note         = {Machine review of arXiv:2507.10613}
}
read the original abstract

Traditional scaling laws in natural language processing suggest that increasing model size and training data enhances performance. However, recent studies reveal deviations, particularly in large language models, where performance improvements decelerate, which is a phenomenon known as sub-scaling. This paper revisits these scaling laws by examining the impact of data quality and training strategies on model performance. Through extensive empirical analysis of over 400 models, we identify high data density and non-optimal resource allocation as key factors contributing to sub-scaling. High data density leads to diminishing returns due to redundant information, while optimal resource allocation is crucial for sustained performance improvements. We propose a sub-optimal scaling law that better predicts performance in sub-scaling regimes, highlighting the importance of data quality and diversity.

Figures

Figures reproduced from arXiv: 2507.10613 by the authors.

Figure 1
Figure 1. Sub-scaling phenomenon in loss. Scaling law fits well with 5B training tokens, but as tokens increase, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. (left) LLaMA 2’s scaling curve outperforms LLaMA 3’s, despite LLaMA 3’s advanced strategies. (right) Higher-density datasets lead to sub-scaling. We propose metric density to measure redundancy and diversity: higher density indicates more redundancy and less diversity, leading to sub-scaling (see Section 2.1). 10 7 10 8 10 9 10 10 Model Size 2.5 3.0 3.5 4.0 Loss 1e+18 3e+18 6e+18 1e+19 3e+19 6e+19 1e+20 3e+20 Optima… view at source ↗
Figure 3
Figure 3. (left) With a fixed total compute budget, we adjust the model-to-data allocation ratio and plot the training loss against model size. A black curve connects the minimum points of each curve, illustrating the optimal Chinchilla law. (right) However, current large language models, such as Llama3 8B, are trained on 15T tokens, with a model-to-data allocation strategy that significantly deviates from the optimal Chinchi… view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: We compare models with (left) 100M param￾eters and (right) 800M parameters trained on high and low density dataset, which demonstrates that higher den￾sity results in a degressive performance increase. of hyper-parameters, to investigate the sub-scaling law phenomenon.…
Figure 5
Figure 5. Figure 5: The yellow and blue circle represents a cluster with a larger number of samples and higher density, where the samples are more similar to each other. In contrast, the gray circle, containing fewer samples and lower density, signifies a cluster where the samples are les…
Figure 6
Figure 6. Figure 6: The relationship between number of samples [PITH_FULL_IMAGE:figures/full_fig_p004_6.png]
Figure 7
Figure 7. Figure 7: (left) Loss vs. Compute Budget Across Model Sizes. (middle) Power-Law Relationship and Convergence Point. (right) For L = λC CαC , when OT R ≤ 50, it is observed that αC decreases when increasing OT R. And when OT R > 50 and OT R ≤ 1700, αC follows a normal distributio…
Figure 8
Figure 8. Figure 8: The figure illustrates the performance of language models as a function of the number of samples, [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: We compare the sub-optimal scaling law L = λD·RD DαD + λN ·RN NαN +E with traditional scaling law (Hoffmann et al., 2022) L = λD DαD + λN NαN + E across the same range of training tokens and model sizes. With the number of training tokens increasing, the RD, RN becomes…
Figure 10
Figure 10. Figure 10: (left) With the total amount of compute budget fixed, we change the model/data allocation ratio, then plot the training loss vs model size curve. We use a black curve to connect the training data of smaller models and do the extrapolation. However, it could be observe…
Figure 11
Figure 11. Figure 11: The number of processed tokens relative to different batch sizes needed to reach a specific training loss, [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: The empirical data points show a scattered distribution around the fitted power-law line for loss and the [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: The relationship between tokens and learning rate required to achieve a specific training loss across four [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 14
Figure 14. Figure 14: The empirical data points exhibit a scattered distribution around the fitted power-law line for the loss and [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]
Figure 16
Figure 16. Figure 16: The training data includes the loss values and [PITH_FULL_IMAGE:figures/full_fig_p019_16.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

46 extracted references · 17 canonical work pages

  1. [1]

    Amro Abbas, Evgenia Rusak, Kushal Tirumala, Wieland Brendel, Kamalika Chaudhuri, and Ari S Morcos. 2024. Effective pruning of web-scale datasets based on complexity of concept clusters. arXiv preprint arXiv:2401.04578

    arXiv 2024

  2. [2]

    Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. 2021. Explaining neural scaling laws. arXiv preprint arXiv:2102.06701

    arXiv 2021

  3. [3]

    Zhengyu Chen, Jixie Ge, Heshen Zhan, Siteng Huang, and Donglin Wang. 2021 a . Pareto self-supervised training for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13663--13672

    work page 2021

  4. [4]

    Zhengyu Chen and Donglin Wang. 2021. Multi-initialization meta-learning with domain adaptation. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1390--1394. IEEE

    work page 2021

  5. [5]

    Zhengyu Chen, Yudong Wang, Teng Xiao, Ruochen Zhou, Xuesheng Yang, Wei Wang, Zhifang Sui, and Jingang Wang. 2025. From mathematical reasoning to code: Generalization of process reward models in test-time scaling. arXiv preprint arXiv:2506.00027

    work page Pith review arXiv 2025

  6. [6]

    Zhengyu Chen, Teng Xiao, and Kun Kuang. 2022. Ba-gnn: On learning bias-aware graph neural network. In 2022 IEEE 38th International Conference on Data Engineering (ICDE), pages 3012--3024. IEEE

    work page 2022

  7. [7]

    Zhengyu Chen, Teng Xiao, Kun Kuang, Zheqi Lv, Min Zhang, Jinluan Yang, Chengqiang Lu, Hongxia Yang, and Fei Wu. 2024 a . Learning to reweight for generalizable graph neural network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 8320--8328

    work page 2024

  8. [8]

    Zhengyu Chen, Teng Xiao, Donglin Wang, and Min Zhang. 2024 b . Pareto graph self-supervised learning. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6630--6634. IEEE

    work page 2024

Show all 46 references
  1. [9]

    Zhengyu Chen, Ziqing Xu, and Donglin Wang. 2021 b . Deep transfer tensor decomposition with orthogonal constraint for recommender systems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 4010--4018

    2021

  2. [10]

    Common Crawl . 2024. Common crawl. http://commoncrawl.org

    2024

  3. [11]

    DeepSeek-AI, :, Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, Ying He, Wenjie Hu, Panpan Huang, Er...

    2024 arXiv

  4. [12]

    Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. 2022. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, pages 5547--5569. PMLR

    2022

  5. [13]

    Dimakis, Gabriel Ilharco, Shuran Song, Thomas Kollar, Yair Carmon, Achal Dave, Reinhard Heckel, Niklas Muennighoff, and Ludwig Schmidt

    Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean-Pierre Mercat, Alex Fang, Jeffrey Li, Sedrick Scott Keh, Rui Xin, Marianna Nezhurina, Igor Vasiljevic, Jenia Jitsev, Alexandros G. Dimakis, Gabriel Ilharco, Shuran S...

    2024 arXiv

  6. [14]

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv

    2020

  7. [15]

    Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garc \'i a, Ciprian Chelba, and Colin Cherry

    B. Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garc \'i a, Ciprian Chelba, and Colin Cherry. 2021. https://api.semanticscholar.org/CorpusID:237532682 Scaling laws for neural machine translation . ArXiv, abs/2109.07740

    2021 arXiv

  8. [16]

    Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M

    Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish...

    2020 arXiv

  9. [18]

    Danny Hernandez, Tom B. Brown, Tom Conerly, Nova Dassarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, Scott Johnston, Benjamin Mann, Christopher Olah, Catherine Olsson, Dario Amodei, Nicholas Joseph, Jared Kaplan, and Sam McCandl...

    2022 arXiv

  10. [19]

    Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. 2021. Scaling laws for transfer. arXiv preprint arXiv:2102.01293

    2021 arXiv

  11. [20]

    Rae, Oriol Vinyals, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osin...

    2022 arXiv

  12. [21]

    Shengding Hu, Xin Liu, Xu Han, Xinrong Zhang, Chaoqun He, Weilin Zhao, Yankai Lin, Ning Ding, Zebin Ou, Guoyang Zeng, et al. 2023. Predicting emergent abilities with infinite resolution evaluation. In The Twelfth International Conference on Learning Representations

    2023

  13. [22]

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai L...

    2024 arXiv

  14. [23]

    Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, and Sanmi Koyejo. 2024. Scaling laws for downstream task performance of large language models. arXiv preprint arXiv:2402.04177

    2024

  15. [24]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  16. [25]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. https://arxiv.org/abs/2001.08361 Scaling laws for neural language models . Preprint, arXiv:2001.08361

    2020 arXiv

  17. [26]

    Aran Komatsuzaki. 2019. https://api.semanticscholar.org/CorpusID:189928090 One epoch is all you need . ArXiv, abs/1906.06669

    2019 arXiv

  18. [27]

    Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. 2018. https://arxiv.org/abs/1812.06162 An empirical model of large-batch training . Preprint, arXiv:1812.06162

    2018 arXiv

  19. [28]

    Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. 2024. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36

    2024

  20. [29]

    Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel

    Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. 2023. https://api.semanticscholar.org/CorpusID:258888192 Scaling data-constrained language models . ArXiv, abs/2305.16264

    2023 arXiv

  21. [30]

    Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, and Yair Carmon. 2024. Resolving discrepancies in compute-optimal scaling of language models. arXiv preprint arXiv:2406.19146

    2024 arXiv

  22. [31]

    Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446

    2021 arXiv

  23. [32]

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arX...

    2024 arXiv

  24. [33]

    Noveen Sachdeva, Benjamin Coleman, Wang-Cheng Kang, Jianmo Ni, Lichan Hong, Ed H Chi, James Caverlee, Julian McAuley, and Derek Zhiyuan Cheng. 2024. How to train data-efficient llms. arXiv preprint arXiv:2402.09668

    2024 arXiv

  25. [34]

    Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. 2022. Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language m...

    2022 arXiv

  26. [35]

    Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. 2022. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35:19523--19536

    2022

  27. [36]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    2023 arXiv

  28. [37]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  29. [38]

    Siqi Wang, Zhengyu Chen, Bei Li, Keqing He, Min Zhang, and Jingang Wang. 2024. Scaling laws across model architectures: A comparative analysis of dense and moe models in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Proce...

    2024

  30. [39]

    Chuhan Wu and Ruiming Tang. 2024. Performance law of large language models. arXiv preprint arXiv:2408.09895

    2024 arXiv

  31. [40]

    Teng Xiao, Yige Yuan, Zhengyu Chen, Mingxiao Li, Shangsong Liang, Zhaochun Ren, and Vasant G. Honavar. 2025 a . https://openreview.net/forum?id=jfwe9qNqRi Simper: A minimalist approach to preference alignment without hyperparameters . In The Thirteenth International Conference...

    2025

  32. [41]

    Teng Xiao, Yige Yuan, Mingxiao Li, Zhengyu Chen, and Vasant G. Honavar. 2025 b . https://openreview.net/forum?id=2QdsjiNXgj On a connection between imitation learning and RLHF . In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April...

    2025

  33. [42]

    Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. 2024. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36

    2024

  34. [43]

    Chen Yang, Junzhuo Li, Xinyao Niu, Xinrun Du, Songyang Gao, Haoran Zhang, Zhaoliang Chen, Xingwei Qu, Ruibin Yuan, Yizhi Li, et al. 2024. The fine line: Navigating large language model pretraining with down-streaming capability analysis. arXiv preprint arXiv:2404.01204

    2024 arXiv

  35. [44]

    Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. 2022. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.03466

    2022 arXiv

  36. [45]

    Greg Yang, Dingli Yu, Chen Zhu, and Soufiane Hayou. 2023. Tensor programs vi: Feature learning in infinite-depth neural networks. arXiv preprint arXiv:2310.02244

    2023 arXiv

  37. [46]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  38. [47]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.