Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

R&B: Domain Regrouping and Data Mixture Balancing for Efficient Foundation Model Training

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read R&B claims that repartitioning training data by semantic similarity and rebalancing domain weights from already-computed gradients delivers state-of-the-art data mixing at 0.01% extra compute.

desk verdict R&B is a genuinely new and efficient data-mixing framework, but the cluster-count selection appears tuned on the eval set and the theory doesn't quite cover the implemented update; worth peer review with major revisions. read the letter →

arxiv 2505.00358 v1 pith:FRBFVOKB submitted 2025-05-01 cs.LG cs.AIcs.CL

Albert Ge , Tzu-Heng Huang , John Cooper , Avi Trost , Ziyi Chu , Satya Sai Srinath Namburi GNVV , Ziyang Cai , Kendall Park
show 2 more authors
Nicholas Roberts Frederic Sala
This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords datamixingdomainregroupingsemanticclusteringgradientbalancingGrammatrixonlineselectionefficienttrainingfoundationmodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

R&B is a claim about two separate bottlenecks in data mixing: what counts as a domain, and how much it costs to reweight domains. The paper argues that human-defined categories like 'open QA' or 'summarization' leave useful structure on the table, and that repartitioning data by embedding similarity into finer-grained clusters improves evaluation loss. It then argues that reweighting can be made nearly free by reading off domain gradients accumulated during ordinary training, forming their Gram matrix, and applying a softmax update, instead of paying for evaluation passes. The empirical claim is that this combination matches or beats existing data-mixing strategies on five datasets while adding about 0.01% extra compute, more than 99% less than the baselines. If true, data mixing stops being a separate, expensive optimization and becomes a byproduct of training itself.

What carries the argument

The load-bearing object is the Gram matrix $G_{ij}(t) = \nabla L(\theta_t; D_i)^\top \nabla L(\theta_t; D_j)$, the matrix of inner products between the gradients of each domain on the current model. Each round, R&B accumulates final-layer per-domain gradients from ordinary backward passes, forms $G$, and sets the next sampling distribution to $\mathrm{softmax}(\lambda G p / \|G p\|_2)$, where $p$ holds the evaluation-domain proportions. The derivation treats the loss as locally linear in the SGD step, so the per-step loss decrease is $\eta p^\top G p'$; maximizing this with a cross-entropy regularizer yields the softmax update. The other half, Regroup, uses k-means on ModernBERT embeddings plus a silhouette-score sweep to choose the number of clusters, and a regret bound (Lemma 1) says that low-radius, well-separated clusters make the mixing objective stable.

What would settle it

A checkpoint experiment: fix a trained model and the learned clusters, compute the Gram matrix $G$, and measure the one-step loss decrease of the softmax-weighted mixture versus uniform sampling and versus the best single cluster. R&B's derivation requires the softmax-weighted step to decrease loss at least as much as any fixed proportion for a small learning rate; a measured violation would falsify the balancing mechanism.

Watch

Extended reading notes

Core claim

The paper's central claim is that predefined, human-assigned data categories are the wrong units for data mixing, and that the right units are finer-grained domains obtained by semantically clustering the training data. On top of that, it claims that the optimal mixture over those domains can be tracked online at almost no extra cost: instead of running evaluation forward passes or separate gradient computations, R&B reuses per-example gradients that normal backprop already produces, aggregates them per domain, forms the Gram matrix $G$, and reweights via $\mathrm{softmax}(\lambda G p / \|Gp\|_2)$. The paper reports that this two-stage recipe matches or improves on Skill-It, Aioli, and DGA across Dolly-15k, Super-NaturalInstructions, S1-reasoning, and DataComp CLIP training while adding only about 0.01% compute overhead. It also proves a regret bound showing that compact, well-separated clusters reduce the worst-case penalty for bundling data into skills, which is the theoretical reason regrouping helps.

Load-bearing premise

The method depends on the assumption that data points grouped by embedding similarity will also affect the model in similar ways during training, so that reweighting those groups actually reweights what the model learns.

Editorial extensions

If this is right

  • Because R&B's overhead scales with the square of the number of domains rather than with evaluation-set size, models can be mixed over hundreds of fine-grained clusters, exactly the regime where prior methods become prohibitively expensive.
  • Regrouping alone, before any balancing, improves stratified sampling on three of the four language datasets tested, so the definition of a domain matters as much as the weighting.
  • The full R&B pipeline reaches lower evaluation loss than the strongest baselines on both NaturalInstructions benchmarks, and on the 40,000-step Dolly run it keeps a consistent advantage through training.
  • On multimodal CLIP training the advantage grows with domain count: with 50 domains R&B beats stratified sampling by 3.27% relative average score, suggesting the method pays off when semantic structure is fine-grained.
  • R&B reaches convergence with roughly 20% of the training steps needed by comparison methods, so efficient mixing also means faster training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Not tested here: R&B's update could be run over sliding windows of a single corpus, turning mixture optimization into a fully local, online data-selection rule rather than a global one.
  • If the silhouette-score heuristic generalizes, practitioners could choose the number of clusters without training any model, making R&B a preprocessing step plus a free training-time add-on.
  • The paper's evidence that topic-based clusters beat task-based categories on instruction data suggests a transfer claim worth testing: topic groupings should be more stable across model sizes and architectures than task groupings, because they track content rather than annotation labels.
  • Because the Gram matrix is a per-domain neural tangent kernel, it could also flag redundant or conflicting domains mid-training: domains with persistently low alignment to the evaluation mixture are candidates for downweighting or removal.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes R&B, a two-stage data-mixing method: Regroup repartitions training data by k-means on ModernBERT embeddings, and Balance reweights the resulting domains online using a Gram matrix of per-domain gradient inner products accumulated during ordinary training, with a softmax update driven by evaluation-set proportions. The authors give a regret bound for stable clusterings, a first-order derivation of the update rule, a FLOP cost model, and experiments on Dolly-15k, Super-NaturalInstructions (ID/OOD), S1-Reasoning, and DataComp CLIP training. The central claims are that semantic clusters outperform human-defined skill categories and that R&B matches or beats prior mixing methods with 0.01%-scale overhead.

Significance. If the empirical claims survive scrutiny, R&B is a useful contribution: the gradient-accumulation trick avoids separate evaluation passes, the m^2 scaling is much cheaper than methods that evaluate on large evaluation sets, and the theoretical framing (regret bound, NTK-style Gram matrix) is a reasonable scaffold. The experiments span natural language, reasoning, and multimodal training, and the ablations that separate Regroup from Balance are a strength. At present, however, the m* selection protocol and the headline table's domain confound are load-bearing gaps.

major comments (3)
  1. [Section 3.2, Figure 2, Tables 1-2] Figure 2's top row sweeps k and marks a 'sweet spot' (green star) on evaluation-loss curves, and Tables 1 and 2 then use m* = 30, 100, 7, and 10 for the reported head-to-head results. The text does not describe a held-out procedure for choosing these values, so the Regroup comparisons may be selected on the same evaluation loss that is later reported, which would inflate R&B's advantage; baselines are not given equivalent k-selection freedom. Algorithm 2 proposes selecting m* with a clustering metric, but that metric is not used in the reported protocol, and the silhouette-to-loss correlations in Figure 2's bottom row (R^2 = 0.047, 0.173, 0.482, 0.895) are too weak on three of the four datasets to serve as a validated substitute. Please either select m* on a held-out portion of the evaluation set, use and validate Algorithm 2's metric, or report sensitivity to m* and show that the headline gains persist.
  2. [Table 1] Table 1 compares R&B on regrouped domains (m* = 30, 100, 7) against Skill-It, Aioli, and DGA on their original domains (m = 38, 60, 8), without stating this in the table or its caption. The performance gap therefore conflates the Regroup component with the Balance component; the ablation table in Figure 3 does control for this, but the paper's headline table does not. Please make Table 1 a same-domain comparison (or explicitly present the Figure 3 ablation as the performance evidence and keep Table 1 for overhead), and state in the caption which domain partition each row uses.
  3. [Section 3.2 / Appendix B.1.2] The theoretical mechanism assumes that ModernBERT embedding clusters 'mimic the gradients' of the model being trained (Appendix B.1.2), but the only empirical support offered is the silhouette-to-loss correlation, which is negligible for Dolly-15k (R^2 = 0.047), weak for Super-NatInst (R^2 = 0.173), and moderate for Super-NatInst Test (R^2 = 0.482), and the paper does not use silhouette to select m*. As written, the claim that semantic clustering is generally superior to human-defined domains rests on a small number of training runs whose cluster counts were selected from the evaluation curves in Figure 2. Please add a direct check (e.g., compare k-means cluster alignment with per-cluster gradient inner products during a probe run) or substantially soften the mechanism claim and restrict it to the datasets where the effect is replicated.
minor comments (5)
  1. [Title and Section 4.1] The title contains a typo ('B alancing') and Section 4.1 says 'three distinct three natural-language data settings'; please fix both.
  2. [Table 1] Table 1's overhead entries for Skill-It and Aioli on Sup-NatInst test are formatted as '6x107%' and '7x106%'; use proper superscripts and state whether these are relative overhead percentages or multiplicative factors.
  3. [Algorithm 3] Algorithm 3 ends with an empty 'Return' statement; return the trained model or remove the line.
  4. [Table 6] In Table 6, 'num layersto track=1, lamda=3' has typos and should name the layer-selection procedure; also clarify why R&B on original NI-OOD produces a NaN in Gp, since the footnote says so but the main text should state this limitation.
  5. [Figure 3] The text claims R&B reaches convergence with only 20% of the training steps needed by other methods, but Figure 3 does not show a convergence threshold or quantify how this percentage was computed; please add the threshold or revise the claim.

Circularity Check

1 steps flagged · score 6.0 of 10

Regroup's cluster count m* is selected on the evaluation-loss curves that are then reported as the headline result; the empirical claim is partly a fitted input, while the Balance update itself is not circular.

  1. fitted input called prediction [Section 3.2, Fig. 2; Tables 1–2; Eq. (2)–(3); Algorithm 2]
    "To keep our investigation tractable, we focus on k-means clustering, and sweep over k. ... The top row of Figure 2 shows that training on the resulting clusters often results in significantly better performance compared to pre-determined partitions. ... there is a 'sweet spot' in the number of domains used for data mixing, indicated by the green star."

    The 'sweet spot' is read off the evaluation-loss curves in Fig. 2, and the same evaluation loss is then reported as R&B's result: Tables 1–2 use m* = 30, 100, 7, and 10, matching the green-star minima. This makes the claimed Regroup advantage over fixed human domains depend on choosing k to minimize the very benchmark that is later reported, while baselines are not given the same selection freedom. Algorithm 2 proposes a silhouette-based metric for choosing m*, but the paper does not use it for the reported numbers, and the bottom-row R² values (0.047, 0.173, 0.482) are too weak to support that alternative. Hence the headline gain is partly fitted, not predicted.

full rationale

The core Balance derivation is self-contained: it starts from a one-step evaluation-loss decrease L(θ_{t+1};D_p) ≈ L(θ_t;D_p) − η pᵀGp′, maximizes pᵀGp′ over the simplex with entropy regularization, and obtains the softmax update p′ = softmax(λGp/‖Gp‖). Here the evaluation proportions p are an input, not a fitted output, and Lemma 2 is a valid inequality showing the greedy argmax achieves the largest one-step decrease under smoothness; that is not circular. There is no load-bearing self-citation chain: Skill-It, Aioli, and DGA are external baselines, and the same-group citation [22] is related work, not a premise. The main circularity is in the Regroup experiment: m* is selected by sweeping k and reading the minimum of the evaluation-loss curves (Fig. 2, green stars), then Tables 1–2 report results at exactly those m* values (30, 100, 7, 10) as evidence that semantic clustering beats human domains. This is selection on the test/evaluation set, so the empirical gain is partly forced by construction; the silhouette-based selection in Algorithm 2 is not actually used, and its own reported correlations (R² = 0.047, 0.173, 0.482) are weak on three of four datasets. The paper also admits an additional limitation that Balance cannot be applied to the original Sup-NatInst test split because train and validation groups must coincide, which narrows but does not circularize the claim. Overall, the theoretical derivation is independent, but the central empirical comparison for Regroup reduces in part to a fitted hyperparameter, warranting a partial-circularity score of 6.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on four assumptions: smoothness and small learning rate, embedding-gradient mimicry, final-layer gradients approximating full gradients, and shared training-evaluation partitions. The most fragile is embedding-gradient mimicry, which the paper validates only weakly. In addition, four free parameters (lambda, m*, K, gradient layer) are set by hand or by evaluation-based selection, and they all influence the reported numbers.

free parameters (4)
  • lambda = 3 (S1); unspecified elsewhere
    Regularization strength in the softmax update. It is a hyperparameter selected per dataset without a stated tuning protocol.
  • number of clusters m* = 7 (Dolly), 30 (Sup-NatInst), 100 (Sup-NatInst test), 10 (S1), 10-150 (DataComp)
    Chosen from silhouette or evaluation-loss sweeps; the choice affects all downstream results and is not independently validated.
  • steps per round K = not reported
    Controls reweighting frequency and the claimed compute overhead. No value is given in the paper, making the overhead figures hard to verify.
  • gradient layer choice = 1 (S1 only)
    Algorithm 1 uses 'final-layer gradients'; only the S1 experiment specifies which layer is tracked. The theory assumes full-model gradients.
assumptions (4)
  • domain assumption The loss is L-smooth and the learning rate is small enough that the first-order Taylor expansion of L around theta_t is valid.
    Used in Appendix B.1 to derive the Gp update. Not verified for the trained models.
  • ad hoc to paper Embedding similarity from ModernBERT mimics the gradient similarity of the trained model.
    Central premise for Regroup. Appendix B.1.2 states it as an assumption; empirical support from silhouette correlations is weak on three of four datasets.
  • domain assumption Per-skill gradients can be computed from a single backward pass using final-layer gradient decomposition, and these approximate the full gradients used in the theory.
    Practical implementation uses final-layer gradients, while the theoretical Gram matrix is defined with full model gradients.
  • domain assumption Training and evaluation data share the same partition structure induced by the clustering function S.
    Required for Balance to compute Gp. The authors acknowledge this fails for out-of-domain evaluation and omit R&B on the original NI-OOD partition.

how reviews work

0 comments
Cite this review

Pith. "Pith review of R&B: Domain Regrouping and Data Mixture Balancing for Efficient Foundation Model Training." pith.science (2026). https://pith.science/paper/FRBFVOKB

@misc{pith2026250500358,
  author       = {Pith},
  title        = {Pith review of: R&B: Domain Regrouping and Data Mixture Balancing for Efficient Foundation Model Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FRBFVOKB}},
  note         = {Machine review of arXiv:2505.00358}
}
read the original abstract

Data mixing strategies have successfully reduced the costs involved in training language models. While promising, such methods suffer from two flaws. First, they rely on predetermined data domains (e.g., data sources, task types), which may fail to capture critical semantic nuances, leaving performance on the table. Second, these methods scale with the number of domains in a computationally prohibitive way. We address these challenges via R&B, a framework that re-partitions training data based on semantic similarity (Regroup) to create finer-grained domains, and efficiently optimizes the data composition (Balance) by leveraging a Gram matrix induced by domain gradients obtained throughout training. Unlike prior works, it removes the need for additional compute to obtain evaluation information such as losses or gradients. We analyze this technique under standard regularity conditions and provide theoretical insights that justify R&B's effectiveness compared to non-adaptive mixing approaches. Empirically, we demonstrate the effectiveness of R&B on five diverse datasets ranging from natural language to reasoning and multimodal tasks. With as little as 0.01% additional compute overhead, R&B matches or exceeds the performance of state-of-the-art data mixing strategies.

Figures

Figures reproduced from arXiv: 2505.00358 by the authors.

Figure 1
Figure 1. Instead of using pre-determined domains (e.g., by task type), we find that it is often better to first repartition the data into finer-grained, semantically related domains. Optimizing the proportions of these new semantic domains can significantly improve training performance. improve training performance over that of the general predefined domains. These improvements are even more pronounced when the number of ski… view at source ↗
Figure 2
Figure 2. Top Row: Across various data settings, we find that there is a “sweet spot” in the number of domains used for data mixing, indicated by the green star. The optimal number of groups varies significantly with the dataset, which motivates the need for compute-efficient data mixing. Bottom Row: We find that silhouette score often correlates with model performance, suggesting that it is possible to predict data mixing pe… view at source ↗
Figure 3
Figure 3. Left: Regrouping skills before applying data mixing strategies can yield substantial improvements. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Left: Training loss curves for Dolly-15k trained for 40,000 steps with different data mixing [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]
Figure 5
Figure 5. Figure 5: Domain weight evolution during training. Our method dynamically adjusts the importance of [PITH_FULL_IMAGE:figures/full_fig_p021_5.png]
Figure 6
Figure 6. Figure 6: Comparison between domain proportions in training versus evaluation data (KL Divergence: [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Evaluating Sample Utility for Efficient Data Selection by Mimicking Model Weights

    cs.LG 2025-01 conditional novelty 6.0 of 10

    A gradient alignment score against a reference model's weights selects and reweights training samples, improving data efficiency in image classification and CLIP pretraining.

Reference graph

Works this paper leans on

36 extracted references · 7 canonical work pages · cited by 1 Pith paper

  1. [1]

    DoGE: Domain Reweighting with Generalization Estimation

    Fan, S.; Pagliardini, M.; Jaggi, M. DoGE: Domain Reweighting with Generalization Estimation. 2024; http://arxiv.org/abs/2310.15393, arXiv:2310.15393

    arXiv 2024

  2. [2]

    M.; Pham, H.; Dong, X.; Du, N.; Liu, H.; Lu, Y.; Liang, P.; Le, Q

    Xie, S. M.; Pham, H.; Dong, X.; Du, N.; Liu, H.; Lu, Y.; Liang, P.; Le, Q. V.; Ma, T.; Yu, A. W. DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining. 2023; http://arxiv. org/abs/2305.10429, arXiv:2305.10429 [cs]

    arXiv 2023

  3. [3]

    F.; Roberts, N.; Bhatia, K.; Wang, J.; Zhang, C.; Sala, F.; R´ e, C

    Chen, M. F.; Roberts, N.; Bhatia, K.; Wang, J.; Zhang, C.; Sala, F.; R´ e, C. Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models. 2023; http://arxiv.org/ abs/2307.14430, arXiv:2307.14430 [cs]

    arXiv 2023

  4. [4]

    F.; Hu, M

    Chen, M. F.; Hu, M. Y.; Lourie, N.; Cho, K.; R´ e, C. Aioli: A Unified Optimization Framework for Language Model Data Mixing. 2024; http://arxiv.org/abs/2411.05735, arXiv:2411.05735

    arXiv 2024

  5. [5]

    Jiang, Y.; Zhou, A.; Feng, Z.; Malladi, S.; Kolter, J. Z. Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws. 2024; http://arxiv.org/abs/2410.11820, arXiv:2410.11820

    arXiv 2024

  6. [6]

    Organize the Web: Constructing Domains Enhances Pre-Training Data Curation

    Wettig, A.; Lo, K.; Min, S.; Hajishirzi, H.; Chen, D.; Soldaini, L. Organize the Web: Constructing Domains Enhances Pre-Training Data Curation. 2025; http://arxiv.org/abs/2502.10341, arXiv:2502.10341 [cs]. 9

    arXiv 2025

  7. [7]

    Free Dolly: Introducing the World’s First Truly Open Instruction-Tuned LLM

    Conover, M.; Hayes, M.; Mathur, A.; Xie, J.; Wan, J.; Shah, S.; Ghodsi, A.; Wendell, P.; Zaharia, M.; Xin, R. Free Dolly: Introducing the World’s First Truly Open Instruction-Tuned LLM. 2023; https://www.databricks.com/blog/2023/04/12/ dolly-first-open-commercially-viable-instruction-tuned-llm

    work page 2023

  8. [8]

    DsDm: Model-Aware Dataset Selection with Datamodels

    Engstrom, L.; Feldmann, A.; Madry, A. DsDm: Model-Aware Dataset Selection with Datamodels. 2024; http://arxiv.org/abs/2401.12926, arXiv:2401.12926 [cs, stat]

    arXiv 2024

Show all 36 references
  1. [9]

    LESS: Selecting Influential Data for Targeted Instruction Tuning

    Xia, M.; Malladi, S.; Gururangan, S.; Arora, S.; Chen, D. LESS: Selecting Influential Data for Targeted Instruction Tuning. 2024; http://arxiv.org/abs/2402.04333, arXiv:2402.04333 [cs]

    2024 arXiv

  2. [10]

    Grad-match: Gradient matching based data subset selection for efficient deep model training

    Killamsetty, K.; Durga, S.; Ramakrishnan, G.; De, A.; Iyer, R. Grad-match: Gradient matching based data subset selection for efficient deep model training. International Conference on Machine Learning. 2021; pp 5464–5474

    2021

  3. [11]

    Evaluating Sample Utility for Data Selection by Mimicking Model Weights

    Huang, T.-H.; Bilkhu, M.; Sala, F.; Movellan, J. Evaluating Sample Utility for Data Selection by Mimicking Model Weights. arXiv preprint arXiv:2501.06708 2025,

    2025 arXiv

  4. [12]

    Mixture-of-Skills: Learning to Optimize Data Usage for Fine- Tuning Large Language Models

    Wu, M.; Vu, T.-T.; Qu, L.; Haffari, G. Mixture-of-Skills: Learning to Optimize Data Usage for Fine- Tuning Large Language Models. 2024; http://arxiv.org/abs/2406.08811, arXiv:2406.08811 [cs]

    2024 arXiv

  5. [13]

    J.; Adila, D.; Sala, F

    Huang, T.-H.; Shin, C.; Tay, S. J.; Adila, D.; Sala, F. Multimodal data curation via object detection and filter ensembles. arXiv preprint arXiv:2401.12225 2024,

    2024 arXiv

  6. [14]

    M.; Santurkar, S.; Ma, T.; Liang, P

    Xie, S. M.; Santurkar, S.; Ma, T.; Liang, P. Data Selection for Language Models via Importance Resampling. 2023; http://arxiv.org/abs/2302.03169, arXiv:2302.03169 [cs]

    2023 arXiv

  7. [15]

    Abbas, A.; Tirumala, K.; Simig, D.; Ganguli, S.; Morcos, A. S. SemDeDup: Data-efficient learn- ing at web-scale through semantic deduplication. 2023; http://arxiv.org/abs/2303.09540, arXiv:2303.09540 [cs]

    2023 arXiv

  8. [16]

    Deduplicating Training Data Makes Language Models Better

    Lee, K.; Ippolito, D.; Nystrom, A.; Zhang, C.; Eck, D.; Callison-Burch, C.; Carlini, N. Deduplicating Training Data Makes Language Models Better. 2022; http://arxiv.org/abs/2107.06499, arXiv:2107.06499

    2022 arXiv

  9. [17]

    Tirumala, K.; Simig, D.; Aghajanyan, A.; Morcos, A. S. D4: Improving LLM Pretraining via Document De-Duplication and Diversification. 2023; http://arxiv.org/abs/2308.12284, arXiv:2308.12284 [cs]

    2023 arXiv

  10. [18]

    BiMix: Bivariate Data Mixing Law for Language Model Pretraining

    Ge, C.; Ma, Z.; Chen, D.; Li, Y.; Ding, B. BiMix: Bivariate Data Mixing Law for Language Model Pretraining. 2024; http://arxiv.org/abs/2405.14908, arXiv:2405.14908 [cs]

    2024 arXiv

  11. [19]

    Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance

    Ye, J.; Liu, P.; Sun, T.; Zhou, Y.; Zhan, J.; Qiu, X. Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance. 2024; http://arxiv.org/abs/2403.16952, arXiv:2403.16952

    2024 arXiv

  12. [20]

    RegMix: Data Mixture as Regression for Language Model Pre-training

    Liu, Q.; Zheng, X.; Muennighoff, N.; Zeng, G.; Dou, L.; Pang, T.; Jiang, J.; Lin, M. RegMix: Data Mixture as Regression for Language Model Pre-training. 2024; http://arxiv.org/abs/2407. 01492, arXiv:2407.01492 [cs]

    2024 arXiv

  13. [21]

    AutoScale: Automatic Prediction of Compute-optimal Data Composition for Training LLMs

    Kang, F.; Sun, Y.; Wen, B.; Chen, S.; Song, D.; Mahmood, R.; Jia, R. AutoScale: Automatic Prediction of Compute-optimal Data Composition for Training LLMs. 2024; http://arxiv.org/ abs/2407.20177, arXiv:2407.20177 [cs, stat]

    2024

  14. [22]

    Compute Optimal Scaling of Skills: Knowledge vs Reasoning

    Roberts, N.; Chatterji, N.; Narang, S.; Lewis, M.; Hupkes, D. Compute Optimal Scaling of Skills: Knowledge vs Reasoning. 2025; https://arxiv.org/abs/2503.10061, eprint: 2503.10061

    2025 arXiv

  15. [23]

    X.; Duderstadt, B.; Mulyar, A

    Nussbaum, Z.; Morris, J. X.; Duderstadt, B.; Mulyar, A. Nomic Embed: Training a Reproducible Long Context Text Embedder. 2024; eprint: 2402.01613

    2024 arXiv

  16. [24]

    Wang, Y. et al. Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks. 2022; http://arxiv.org/abs/2204.07705, arXiv:2204.07705 [cs]. 10

    2022 arXiv

  17. [25]

    L.; Fei-Fei, L.; Hajishirzi, H.; Zettlemoyer, L.; Liang, P.; Cand` es, E.; Hashimoto, T

    Muennighoff, N.; Yang, Z.; Shi, W.; Li, X. L.; Fei-Fei, L.; Hajishirzi, H.; Zettlemoyer, L.; Liang, P.; Cand` es, E.; Hashimoto, T. s1: Simple test-time scaling. 2025; https://arxiv.org/abs/2501. 19393, eprint: 2501.19393

    2025 arXiv

  18. [26]

    Rousseeuw, P. J. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics 1987, 20, 53–65

    1987

  19. [27]

    Dynamic Gradient Alignment for Online Data Mixing

    Fan, S.; Grangier, D.; Ablin, P. Dynamic Gradient Alignment for Online Data Mixing. 2024; http://arxiv.org/abs/2410.02498, arXiv:2410.02498 [cs]

    2024 arXiv

  20. [28]

    GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow

    Black, S.; Gao, L.; Wang, P.; Leahy, C.; Biderman, S. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow. 2021; https://doi.org/10.5281/zenodo.5297715

    2021 doi

  21. [29]

    Yang, A. et al. Qwen2 Technical Report. arXiv preprint arXiv:2407.10671 2024,

    2024 arXiv

  22. [30]

    W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; others Learning transferable visual models from natural language supervision

    Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; others Learning transferable visual models from natural language supervision. International conference on machine learning. 2021; pp 8748–8763

    2021

  23. [31]

    OpenCLIP

    Ilharco, G.; Wortsman, M.; Wightman, R.; Gordon, C.; Carlini, N.; Taori, R.; Dave, A.; Shankar, V.; Namkoong, H.; Miller, J.; Hajishirzi, H.; Farhadi, A.; Schmidt, L. OpenCLIP. 2021; https: //doi.org/10.5281/zenodo.5143773

    2021 doi

  24. [32]

    Gadre, S. Y.; Ilharco, G.; Fang, A.; Hayase, J.; Smyrnis, G.; Nguyen, T.; Marten, R.; Wortsman, M.; Ghosh, D.; Zhang, J.; others Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems 2023, 36, 27092–27112

    2023

  25. [33]

    B.; Chess, B.; Child, R.; Gray, S.; Radford, A.; Wu, J.; Amodei, D

    Kaplan, J.; McCandlish, S.; Henighan, T.; Brown, T. B.; Chess, B.; Child, R.; Gray, S.; Radford, A.; Wu, J.; Amodei, D. Scaling Laws for Neural Language Models. 2020; http://arxiv.org/abs/ 2001.08361, arXiv:2001.08361 [cs]

    2020 arXiv

  26. [34]

    What’s the Backward-Forward FLOP Ratio for Neural Networks? 2021; https: //epoch.ai/blog/backward-forward-FLOP-ratio

    Hobbhahn, M. What’s the Backward-Forward FLOP Ratio for Neural Networks? 2021; https: //epoch.ai/blog/backward-forward-FLOP-ratio

    2021

  27. [35]

    Efficient Per-Example Gradient Computations

    Goodfellow, I. Efficient Per-Example Gradient Computations. 2015; http://arxiv.org/abs/ 1510.01799, arXiv:1510.01799 [stat]

    2015 arXiv

  28. [36]

    T.; Wu, T.; Song, D.; Mittal, P.; Jia, R

    Wang, J. T.; Wu, T.; Song, D.; Mittal, P.; Jia, R. GREATS: Online Selection of High-Quality Data for LLM Training in Every Iteration. 2024. 11 The appendix is structured as follows. Appendix A introduces our notation, followed by theoretical insights and proofs in Appendix B. ...

    2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.