arxiv: 2605.12715 · v1 · submitted 2026-05-12 · 💻 cs.LG · cs.CL

Recognition: no theorem link

Scaling Laws for Mixture Pretraining Under Data Constraints

Anastasiia Sedova, Natalie Schluter, Pierre Ablin, Skyler Seto

Authors on Pith no claims yet

Pith reviewed 2026-05-14 21:41 UTC · model grok-4.3

classification 💻 cs.LG cs.CL

keywords scaling lawsmixture trainingdata constraintsrepetitionlanguage model pretrainingtarget domain performance

0 comments

The pith

Mixture pretraining tolerates repeating scarce target data 15-20 times, far more than single-source training.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper examines how to mix limited high-value target data with abundant generic data when pretraining language models. It shows that repetition of the target data is the key factor affecting performance in the target domain. Mixtures allow much higher repetition rates than training only on target data. The authors derive a repetition-aware scaling law that models the diminishing returns from repeats and the regularizing effect of generic data. This law enables calculating optimal mixture ratios for given data sizes and compute budgets.

Core claim

Across more than 2000 training runs, repetition drives target-domain performance, and mixture training tolerates reusing scarce target corpora 15-20 times, with the optimum depending on target data size, compute budget, and model scale. A repetition-aware mixture scaling law accounts for the decreasing value of repeated target tokens and the regularizing role of generic data, allowing principled optimization of mixture configurations.

What carries the argument

The repetition-aware mixture scaling law that adjusts for the reduced value of repeated target tokens while incorporating the regularization provided by generic data.

If this is right

Optimal number of target data repetitions varies with target corpus size, total compute, and model scale.
Mixture training yields better target performance than single-source training under data scarcity.
Practical recommendations for mixture ratios can be computed directly from the scaling law.
Generic data serves a regularizing role that permits higher target repetition without overfitting.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

These optimal repetition counts could guide data collection priorities for low-resource domains.
Extending the law to even larger models might reveal whether the 15-20 repetition tolerance scales further.
Similar principles may apply to other modalities like vision or multimodal pretraining where target data is scarce.

Load-bearing premise

The observed repetition tolerances and scaling law parameters remain valid beyond the tested model sizes, data types, and compute budgets.

What would settle it

Train models at a scale outside the tested range using the predicted optimal mixture and check whether target-domain performance matches the scaling law prediction.

read the original abstract

As language models scale, the amount of data they require grows -- yet many target data sources, such as low-resource languages or specialized domains, are inherently limited in size. A common strategy is to mix this scarce but valuable target data with abundant generic data, which presents a fundamental trade-off: too little target data in the mixture underexposes the model to the target domain, while too much target data repeats the same examples excessively, yielding diminishing returns and eventual overfitting. We study this trade-off across more than 2,000 language-model training runs spanning multiple model and target dataset sizes, as well as several data types, including multilingual, domain-specific, and quality-filtered mixtures. Across all settings, we find that repetition is a central driver of target-domain performance, and that mixture training tolerates much higher repetition than single-source training: scarce target corpora can be reused 15-20 times, with the optimal number of repetitions depending on the target data size, compute budget, and model scale. Next, we introduce a repetition-aware mixture scaling law that accounts for the decreasing value of repeated target tokens and the regularizing role of generic data. Optimizing the scaling law provides a principled way to compute effective mixture configurations, yielding practical mixture recommendations for pretraining under data constraints.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript reports results from over 2,000 language-model pretraining runs on mixtures of scarce target data (multilingual, domain-specific, quality-filtered) with abundant generic data. It concludes that repetition is the dominant driver of target-domain performance, that mixtures tolerate 15-20 repetitions of target tokens (far more than single-source training), and that the optimal repetition count depends on target size, compute budget, and model scale. A repetition-aware scaling law is introduced that incorporates decreasing marginal value of repeated target tokens plus regularization from generic data; optimizing this law yields mixture recommendations.

Significance. If the scaling law generalizes, the work supplies a concrete, empirically grounded method for choosing mixtures under data constraints, addressing a practical bottleneck in low-resource and domain-adaptation pretraining. The breadth of the experimental sweep (multiple model sizes, data types, and budgets) is a clear strength and supports the central empirical observation that repetition tolerance is substantially higher in mixtures than in single-source regimes.

major comments (2)

[Scaling-law section (following the empirical results)] The repetition-aware scaling law is fitted directly to the same >2,000 runs that produce the reported 15-20× repetition tolerance and optimal counts. Because the functional form and coefficients (including any repetition-decay term) are determined post-hoc from these data, the law’s predictions for optimal mixtures are not independent of the observations used to fit it; an explicit hold-out validation on unseen model scales, compute budgets, or data distributions is required to substantiate the claim that the law provides “principled” recommendations beyond the tested regimes.
[Empirical results and abstract] The headline claim that “mixture training tolerates much higher repetition than single-source training” is load-bearing for the paper’s contribution. While the abstract states the 15-20× figure, the manuscript does not present a direct, quantitative side-by-side comparison (e.g., a table of repetition thresholds at which validation loss diverges for mixture vs. single-source runs at matched compute and model size).

minor comments (2)

[Abstract] The abstract does not state the explicit functional form of the repetition-aware scaling law (e.g., the precise dependence on repetition count, generic-data fraction, or model scale), which would allow readers to assess the modeling assumptions immediately.
[Scaling-law section] Notation for the repetition-decay coefficient and any other free parameters should be introduced consistently in the scaling-law section and reused in all subsequent figures and tables.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the insightful comments on our manuscript. We appreciate the recognition of the experimental breadth and the practical implications of our findings. Below, we provide point-by-point responses to the major comments and describe the revisions we will implement.

read point-by-point responses

Referee: [Scaling-law section (following the empirical results)] The repetition-aware scaling law is fitted directly to the same >2,000 runs that produce the reported 15-20× repetition tolerance and optimal counts. Because the functional form and coefficients (including any repetition-decay term) are determined post-hoc from these data, the law’s predictions for optimal mixtures are not independent of the observations used to fit it; an explicit hold-out validation on unseen model scales, compute budgets, or data distributions is required to substantiate the claim that the law provides “principled” recommendations beyond the tested regimes.

Authors: We agree that an explicit hold-out validation would strengthen the claims regarding the generalization of the scaling law. While the functional form is motivated by theoretical considerations of diminishing marginal returns on repeated tokens and the regularizing effect of generic data (drawing from established scaling law literature), the coefficients were indeed fitted to the full experimental set. In the revision, we will partition the experimental data into training and hold-out sets, refit the law on the training portion, and evaluate its predictive accuracy on unseen model scales, compute budgets, and data distributions. We will report the hold-out performance metrics and update the manuscript accordingly to substantiate the principled nature of the recommendations. revision: yes
Referee: [Empirical results and abstract] The headline claim that “mixture training tolerates much higher repetition than single-source training” is load-bearing for the paper’s contribution. While the abstract states the 15-20× figure, the manuscript does not present a direct, quantitative side-by-side comparison (e.g., a table of repetition thresholds at which validation loss diverges for mixture vs. single-source runs at matched compute and model size).

Authors: We acknowledge that a direct side-by-side comparison would make the central claim more robust and easier to verify. Although our experiments included single-source training runs for baseline comparison (which informed the 15-20× tolerance figure), these were not presented in a consolidated quantitative format. In the revised manuscript, we will add a new table and accompanying figure that directly compares the repetition thresholds at which validation loss begins to diverge for mixture versus single-source regimes, at matched compute budgets and model sizes. This will include the specific repetition counts where performance plateaus or degrades, providing the quantitative evidence requested. revision: yes

Circularity Check

1 steps flagged

Repetition-aware scaling law fitted to the same 2000+ runs; optimal repetition counts and mixture recommendations are direct outputs of that fit

specific steps

fitted input called prediction [Abstract (scaling law introduction) and subsequent optimization section]
"Next, we introduce a repetition-aware mixture scaling law that accounts for the decreasing value of repeated target tokens and the regularizing role of generic data. Optimizing the scaling law provides a principled way to compute effective mixture configurations, yielding practical mixture recommendations for pretraining under data constraints."

The scaling law is constructed by fitting to the same experimental runs that measured repetition effects. The 'optimal' repetition counts and mixture ratios are then obtained by optimizing this fitted law, so the headline claims (15-20x tolerance, dependence on target size/compute/scale) are statistically forced by the input data rather than independently derived or validated.

full rationale

The paper's central result (15-20x repetition tolerance and optimal counts depending on size/compute/scale) is obtained by fitting a repetition-aware scaling law to the identical set of >2000 training runs that first observed the repetition effects. The law's functional form and coefficients are empirically determined within the tested regimes; optimizing it then 'predicts' the very mixture configurations that were already measured. No independent derivation, closed-form proof, or held-out validation at larger scales is provided, so the recommendations reduce to a re-expression of the fitted inputs.

Axiom & Free-Parameter Ledger

2 free parameters · 1 axioms · 0 invented entities

The central claim rests on empirical scaling observations from the 2000 runs and a fitted functional form that incorporates repetition decay and generic-data regularization; no new physical entities are introduced.

free parameters (2)

repetition decay coefficient
Parameter in the scaling law that models decreasing value of repeated target tokens, fitted to experimental results.
optimal repetition count
15-20 range reported as depending on target size, compute, and model scale; derived from fits to the training runs.

axioms (1)

domain assumption Scaling behavior observed in tested model and data regimes generalizes to larger scales and unseen data types
Invoked when extending the fitted law to practical mixture recommendations beyond the experimental grid.

pith-pipeline@v0.9.0 · 5529 in / 1458 out tokens · 36674 ms · 2026-05-14T21:41:19.661355+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

58 extracted references · 58 canonical work pages · 9 internal anchors

[1]

2025 , eprint=

Scaling Laws for Optimal Data Mixtures , author=. 2025 , eprint=

work page 2025
[2]

Tensor Programs

Yang, Greg and Hu, Edward J and Babuschkin, Igor and Sidor, Szymon and Liu, Xiaodong and Farhi, David and Ryder, Nick and Pachocki, Jakub and Chen, Weizhu and Gao, Jianfeng , booktitle=. Tensor Programs

work page
[3]

Scaling Laws for Neural Language Models

Scaling laws for neural language models , author=. arXiv preprint arXiv:2001.08361 , year=

work page internal anchor Pith review Pith/arXiv arXiv 2001
[4]

arXiv preprint arXiv:2402.07871 , year=

Scaling laws for fine-grained mixture of experts , author=. arXiv preprint arXiv:2402.07871 , year=

work page arXiv
[5]

EMNLP , year=

Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Models , author=. EMNLP , year=

work page
[6]

Advances in Neural Information Processing Systems , volume=

D-cpt law: Domain-specific continual pre-training scaling law for large language models , author=. Advances in Neural Information Processing Systems , volume=

work page
[7]

arXiv preprint arXiv:2510.06548 , year=

From Acceleration to Saturation: Scaling Behavior of Bootstrapped Language Model Pretraining , author=. arXiv preprint arXiv:2510.06548 , year=

work page arXiv
[8]

arXiv preprint arXiv:2603.19149 , year=

Optimal Splitting of Language Models from Mixtures to Specialized Domains , author=. arXiv preprint arXiv:2603.19149 , year=

work page arXiv
[9]

arXiv preprint arXiv:2502.06042 , year=

Scaling laws for forgetting during finetuning with pretraining data injection , author=. arXiv preprint arXiv:2502.06042 , year=

work page arXiv
[10]

ICLR , year=

When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method , author=. ICLR , year=

work page
[11]

Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track , pages=

Scaling Parameter-Constrained Language Models with Quality Data , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track , pages=

work page 2024
[12]

Soboleva, Daria and Al-Khateeb, Faisal and Myers, Robert and Steeves, Jacob R and Hestness, Joel and Dey, Nolan , title =

work page
[13]

The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track , year=

Nemotron-CLIMB: Clustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track , year=

work page
[14]

ICLR , year=

Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling , author=. ICLR , year=

work page
[15]

Antropic Annoucements , url =

Introducing Claude Opus 4.6 , author=. Antropic Annoucements , url =

work page
[16]

0: 24T tokens of organized web data , author=

Essential-Web v1. 0: 24T tokens of organized web data , author=. arXiv preprint arXiv:2506.14111 , year=

work page arXiv
[17]

Google DeepMind Blog , volume=

Advanced version of gemini with deep think officially achieves gold-medal standard at the international mathematical olympiad , author=. Google DeepMind Blog , volume=

work page
[18]

OpenAI GPT-5 System Card

Openai gpt-5 system card , author=. arXiv preprint arXiv:2601.03267 , year=

work page internal anchor Pith review Pith/arXiv arXiv
[19]

arXiv preprint arXiv:2602.03837 , year=

Accelerating Scientific Research with Gemini: Case Studies and Common Techniques , author=. arXiv preprint arXiv:2602.03837 , year=

work page arXiv
[20]

Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: long papers) , pages=

Dolma: An open corpus of three trillion tokens for language model pretraining research , author=. Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: long papers) , pages=

work page
[21]

Findings of the Association for Computational Linguistics: ACL 2025 , pages=

Training bilingual lms with data constraints in the targeted language , author=. Findings of the Association for Computational Linguistics: ACL 2025 , pages=

work page 2025
[22]

Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

Rephrasing the web: A recipe for compute and data-efficient language modeling , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

work page
[23]

Textbooks Are All You Need

Textbooks are all you need , author=. arXiv preprint arXiv:2306.11644 , year=

work page internal anchor Pith review Pith/arXiv arXiv
[24]

Bakouch, Elie and Ben Allal, Loubna and Lozhkov, Anton and Tazi, Nouamane and Tunstall, Lewis and Patiño, Carlos Miguel and Beeching, Edward and Roucher, Aymeric and Reedi, Aksel Joonas and Gallouédec, Quentin and Rasul, Kashif and Habib, Nathan and Fourrier, Clémentine and Kydlicek, Hynek and Penedo, Guilherme and Larcher, Hugo and Morlon, Mathieu and Sr...

work page
[25]

Olmo 3

Olmo 3 , author=. arXiv preprint arXiv:2512.13961 , year=

work page internal anchor Pith review Pith/arXiv arXiv
[26]

Advances in Neural Information Processing Systems , volume=

Training compute-optimal large language models , author=. Advances in Neural Information Processing Systems , volume=

work page
[27]

International Conference on Machine Learning , pages=

Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models , author=. International Conference on Machine Learning , pages=. 2025 , organization=

work page 2025
[28]

Advances in Neural Information Processing Systems , volume=

Datacomp-lm: In search of the next generation of training sets for language models , author=. Advances in Neural Information Processing Systems , volume=

work page
[29]

Submitted to NeurIPS 2026 , year=

Mix, Don't Tune: Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settings , author=. Submitted to NeurIPS 2026 , year=

work page 2026
[30]

Advances in Neural Information Processing Systems , volume=

Scaling data-constrained language models , author=. Advances in Neural Information Processing Systems , volume=

work page
[31]

arXiv preprint arXiv:2205.10487 , year=

Scaling laws and interpretability of learning from repeated data , author=. arXiv preprint arXiv:2205.10487 , year=

work page arXiv
[32]

arXiv preprint arXiv:2403.16952 , year=

Data mixing laws: Optimizing data mixtures by predicting language modeling performance , author=. arXiv preprint arXiv:2403.16952 , year=

work page arXiv
[33]

Xie, Sang Michael and Pham, Hieu and Dong, Xuanyi and Du, Nan and Liu, Hanxiao and Lu, Yifeng and Liang, Percy and Le, Quoc V and Ma, Tengyu and Yu, Adams Wei , journal=

work page
[34]

Fan, Simin and Pagliardini, Matteo and Jaggi, Martin , journal=

work page
[35]

Language models on a diet: Cost-effective development of domain-specific models with

Gadre, Samir Yitzhak and Ilharco, Gabriel and Fang, Alex and Hayase, Jonathan and Smber, Georgios and Maini, Pratyush and Thrush, Tristan and Raber, Florian and Nguyen, Thao and others , journal=. Language models on a diet: Cost-effective development of domain-specific models with

work page
[36]

Transactions on Machine Learning Research , year=

Emergent abilities of large language models , author=. Transactions on Machine Learning Research , year=

work page
[37]

The state and fate of linguistic diversity and inclusion in the

Joshi, Pratik and Santy, Sebastin and Buber, Amar and Bali, Kalika and Choudhury, Monojit , journal=. The state and fate of linguistic diversity and inclusion in the

work page
[38]

Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and others , journal=. The

work page
[39]

Paster, Keiran and Santos, Marco Dos and Azerbayev, Zhangir and Ba, Jimmy , journal=

work page
[40]

Yu, Longhui and Jiang, Weisen and Shi, Han and Yu, Jincheng and Liu, Zhengying and Zhang, Yu and Kwok, James T and Li, Zhenguo and Weller, Adrian and Liu, Weiyang , journal=

work page
[41]

arXiv preprint arXiv:2506.20920 , year=

Penedo, Guilherme and Kydl. arXiv preprint arXiv:2506.20920 , year=

work page arXiv
[42]

Penedo, Guilherme and Kydl. The. Advances in Neural Information Processing Systems , volume=

work page
[43]

Soldaini, Luca and Lo, Kyle , institution=

work page
[44]

Nemotron- CC : Transforming C ommon C rawl into a Refined Long-Horizon Pretraining Dataset

Su, Dan and Kong, Kezhi and Lin, Ying and Jennings, Joseph and Norick, Brandon and Kliegl, Markus and Patwary, Mostofa and Shoeybi, Mohammad and Catanzaro, Bryan. Nemotron- CC : Transforming C ommon C rawl into a Refined Long-Horizon Pretraining Dataset. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon...

work page doi:10.18653/v1/2025.acl-long.123 2025
[45]

Walsh, Pete and Soldaini, Luca and Groeneveld, Dirk and Lo, Kyle and Arora, Shane and Bhagia, Akshita and Gu, Yuling and Huang, Shengyi and Jordan, Matt and Lambert, Nathan and others , journal=. 2

work page
[46]

arXiv preprint arXiv:2404.07177 , year=

Scaling Laws for Data Filtering--Data Curation cannot be Compute Agnostic , author=. arXiv preprint arXiv:2404.07177 , year=

work page arXiv
[47]

Adam: A Method for Stochastic Optimization

Adam: A Method for Stochastic Optimization , author=. arXiv preprint arXiv:1412.6980 , year=

work page internal anchor Pith review Pith/arXiv arXiv
[48]

arXiv preprint arXiv:2405.18392 , year=

Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations , author=. arXiv preprint arXiv:2405.18392 , year=

work page arXiv
[49]

arXiv preprint arXiv:2406.19146 , year=

Resolving Discrepancies in Compute-Optimal Scaling of Language Models , author=. arXiv preprint arXiv:2406.19146 , year=

work page arXiv
[50]

OpenAI blog , year=

Language Models are Unsupervised Multitask Learners , author=. OpenAI blog , year=

work page
[51]

Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages=

SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing , author=. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages=

work page 2018
[52]

Journal of Machine Learning Research , volume=

Exploring the limits of transfer learning with a unified text-to-text transformer , author=. Journal of Machine Learning Research , volume=

work page
[53]

Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge , author=. arXiv preprint arXiv:1803.05457 , year=

work page internal anchor Pith review Pith/arXiv arXiv
[54]

HellaSwag: Can a Machine Really Finish Your Sentence?

HellaSwag: Can a Machine Really Finish Your Sentence? , author=. arXiv preprint arXiv:1905.07830 , year=

work page internal anchor Pith review Pith/arXiv arXiv 1905
[55]

The LAMBADA dataset: Word prediction requiring a broad discourse context

The LAMBADA dataset: Word prediction requiring a broad discourse context , author=. arXiv preprint arXiv:1606.06031 , year=

work page internal anchor Pith review Pith/arXiv arXiv
[56]

Proceedings of AAAI , year=

PIQA: Reasoning about Physical Intuition in Natural Language , author=. Proceedings of AAAI , year=

work page
[57]

Crowdsourcing Multiple Choice Science Questions

Crowdsourcing Multiple Choice Science Questions , author=. arXiv preprint arXiv:1707.06209 , year=

work page internal anchor Pith review Pith/arXiv arXiv
[58]

Proceedings of AAAI , year=

WinoGrande: An Adversarial Winograd Schema Challenge at Scale , author=. Proceedings of AAAI , year=

work page