Pith. sign in

REVIEW 5 major objections 6 minor 55 references

Domain2Vec: Vectorizing Datasets to Find the Optimal Data Mixture without Training

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Domain2Vec claims that the optimal pretraining data mixture can be found by aligning a dataset's 260-dimensional domain vector to the validation set, with no proxy model training.

desk verdict Novel dataset-vectorization idea for data-mixture search, but the training-free DA2 claim is under-validated; worth a serious referee. read the letter →

arxiv 2506.10952 v1 pith:Z4REOJCG submitted 2025-06-12 cs.CL cs.AIcs.LG

Mozhi Zhang , Howe Tissue , Lu Wang , Xipeng Qiu This is my paper
classification cs.CLcs.AIcs.LG
keywords datamixtureoptimizationdomainvectorsmeta-domainslanguagemodelpretrainingdistributionalignmentassumptiontraining-freeselectiondatasetvectorizationThePile
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's central claim is that a dataset can be summarized by a vector—its distribution over 260 reusable "meta-domains" (semantically distinct clusters of documents)—and that the best pretraining mixture is the one whose vector most closely matches the validation set's vector. Concretely, Domain2Vec predicts the optimal mixture by solving $r^* = \arg\min_r \mathrm{Dist}(V_{\mathrm{train}} \cdot r, v_{\mathrm{valid}})$ with Huber loss, so no proxy model needs to be trained for mixture selection. The paper reports that this training-free rule ranks C4/Knowledge-Pile mixtures with Spearman correlation 0.6657 across validation sets (random ranking and kNN give 0.0497 and 0.3543), reaches the Pile-CC validation loss of the original Pile mixture using only 51.5% of the training compute, and improves downstream performance by an average of 2.83% under a fixed compute budget. If correct, this turns data-mixture search into a cheap geometric problem in domain-vector space and makes the result portable when datasets are added or removed.

What carries the argument

The central object is the domain vector: a probability distribution over 260 meta-domains, where a meta-domain is a k-means cluster of document embeddings that acts like a basis direction in dataset space. The load-bearing algebraic step is that the mixed training set's vector is the linear combination $V_{\mathrm{train}} \cdot r$, making the objective $r^* = \arg\min_r \mathrm{Dist}(V_{\mathrm{train}} \cdot r, v_{\mathrm{valid}})$ cheap to evaluate with Huber loss. The paper's pilot study also shows that mixture rankings are nearly invariant across model sizes (Spearman 0.9743 and Pearson 0.9947 between 83M and 1.6B models), which supports transferring the no-training solution to larger LMs. When integrated with RegMix, the same vector space lets the paper fit per-meta-domain loss functions $f_i(V_{\mathrm{train}} \cdot r)$ and sum them weighted by the validation vector, avoiding refits when datasets change.

What would settle it

Train a 7B-parameter model on several C4/Knowledge-Pile mixtures (or Pile component mixtures) and compare actual validation-loss rankings to Domain2Vec's distance ranking; if the Spearman correlation falls well below the reported 0.97 from the 83M-to-1.6B pilot, or if the DA2-chosen mixture is outperformed by a random mixture, the central claim fails. A cheaper check on a fixed validation set is to test whether Dist($V_{\mathrm{train}} \cdot r$, $v_{\mathrm{valid}}$) predicts validation loss across mixtures when $r$ is varied in directions that do not change document-type composition but do change content quality.

Watch

Extended reading notes

Core claim

On the paper's own terms, optimal data-mixture discovery reduces to a distance-minimization problem in a fixed 260-dimensional semantic space. Meta-domains are built by embedding a 5.2 TB corpus of English, Chinese, and code, clustering it with k-means into 260 groups, and training a classifier (a fine-tuned Qwen2-1.5B, with 74.73% accuracy) that maps each document to a distribution over the groups. Any dataset becomes the average of its sampled documents' classifier outputs, so a mixture with ratios $r$ has domain vector $V_{\mathrm{train}} \cdot r$. Under the Distribution Alignment Assumption, validation loss decreases as that vector approaches $v_{\mathrm{valid}}$, so the paper sets $r^* = \arg\min_r \mathrm{Dist}(V_{\mathrm{train}} \cdot r, v_{\mathrm{valid}})$. It validates the rule by showing that mixture rankings on 20 validation sets are strongly correlated with actual rankings, and that selected mixtures need only 51.5% of baseline compute to match Pile-CC validation loss while improving average downstream accuracy by 2.83% under equal compute.

Load-bearing premise

The method assumes that aligning the document-type composition of the training mixture with the validation set's composition lowers validation loss, and that the best mixture found at small model sizes stays the best at production scale.

Editorial extensions

If this is right

  • Pretraining budgets shrink: on the Pile-CC validation target, the selected mixture matches the original Pile mixture's validation loss at 51.5% of training steps, or cuts validation loss by roughly 4% at the same budget.
  • Downstream gains: under equal compute, the Domain2Vec+RegMix mixture improves average downstream accuracy by 2.83% over the Human Pile mixture, landing close to DoReMi and RegMix while using only 0.26% of their estimated FLOPs.
  • Dataset changes become cheap: because each dataset is a fixed vector, adding or removing a source only requires vectorizing the new source and rerunning the minimization, not retraining proxy models or refitting mixture-loss functions.
  • Prior methods become reusable: Domain2Vec+RegMix fits loss functions per meta-domain, so predictions transfer to unseen training and validation sets without collecting new fitting points.
  • A mixture can be scored by a matrix-vector product plus a distance evaluation, so comparing thousands of candidate mixtures costs almost nothing compared with training a language model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not explore using the same geometry to choose the validation distribution itself; a deliberately broad $v_{\mathrm{valid}}$ would likely select a mixture optimized for general performance rather than one benchmark.
  • DA2 is applied as a static one-shot optimization; an online version that periodically re-solves the distance minimization during pretraining would test whether alignment is the causal driver of the reported gains.
  • The domain vector summarizes only document-type composition, not within-domain ordering or quality; combining Domain2Vec with a quality filter is a natural next experiment that might yield further savings.
  • The 260 meta-domains were built for English, Chinese, and code; vectorizing multilingual or multimodal corpora would require rebuilding the vocabulary, so the paper's "any dataset" claim is untested in those regimes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. Domain2Vec proposes representing text datasets as distributions over 260 k-means-clustered "meta-domains" using a fine-tuned Qwen2-1.5B classifier, and then selecting pretraining mixture ratios by minimizing a distance between the training mixture's domain vector and the validation set's domain vector (Eq. 5, DA2). The paper also integrates domain vectors into a RegMix-style regression (Eq. 7) to reduce the cost of fitting mixture-performance models. Experiments rank six C4/Knowledge-Pile mixtures (Table 2) and evaluate downstream performance of Pile-based mixtures (Table 3), reporting validation-loss savings at 51.5% of the compute of the Human mixture and downstream performance at 0.26% of DoReMi's estimated FLOPs.

Significance. The vectorization idea is clean and the integration with RegMix is a useful modular contribution that could reduce the cost of data-mixture search. The reported compute savings are substantial if the central assumption holds. However, the validity of the DA2 surrogate is not established by the supplied evidence: the pilot study tests a different property, and the direct correlation in Table 2 is moderate. The paper is honest in labeling DA2 as an assumption and in listing unresolved choices, but the current evidence does not support the "optimal data mixture" framing without qualification.

major comments (5)
  1. [3.3, Eq. (5), Appendix A] The central objective Eq. (5) is an assumption, not a consequence of maximum likelihood. Appendix A shows only that when Dtrain and Dvalid are identical distributions, MLE on Dtrain optimizes the validation loss; it does not show that proximity of the 260-dimensional meta-domain histograms implies proximity of the document-level distributions. Two corpora can have nearly identical domain vectors while differing in token-level difficulty, style, or intra-cluster topic composition, so minimizing Dist(Vtrain·r, vvalid) need not minimize LDvalid(r). The paper should either derive a quantitative bound connecting the two or provide direct counterfactual evidence, for example mixtures with equal domain-vector distance but different measured losses.
  2. [3.2] The pilot study is structurally separate from DA2. It establishes only that the ranking of six mixtures is similar for 83M and 1.6B models (Spearman 0.9743); it does not test whether domain-vector alignment ranks mixtures correctly. Consequently, the only direct support for Eq. (5) is Table 2, and the pilot study should not be presented as validating the DA2 approach.
  3. [4.1, Table 2] The direct evidence for Eq. (5) is moderate and statistically thin. Table 2 reports aggregate Spearman 0.6657 and Pearson 0.5833 across 20 validation sets, but with only two datasets and six prespecified ratios, and without per-validation-set correlations, confidence intervals, or significance tests. Random ranking already gives Spearman 0.0497 and the kNN baseline gives 0.3543. The experiments also rank discrete mixtures rather than solving Eq. (4)/(5) over the continuous simplex, so the claim of finding "the optimal data mixture" is stronger than the setup can verify.
  4. [4.2, Table 3] The downstream results do not clearly support the "comparable performance" wording. Table 3 shows Domain2Vec+DA2 averaging 0.432 and Domain2Vec+RegMix averaging 0.436, whereas DoReMi and RegMix average 0.439 and 0.441 respectively; the reported improvement is only relative to the Human mixture (0.424). No error bars or significance tests are given, so the comparison to strong baselines is not established. The paper should report variance across seeds, provide a direct statistical comparison, and qualify the abstract's claim about downstream improvement.
  5. [2, 4.2] The meta-domain vocabulary and classifier are constructed from a 5.2TB corpus that appears to overlap with the evaluation corpora (The Pile, RedPajama) in source composition. This creates a risk of leakage: domain vectors for evaluation datasets may be easy to estimate because the same kinds of sources were used to build the 260 meta-domains and to fine-tune the classifier. A disjointness analysis or a hold-out source experiment is needed before claiming generalization to new domains. Likewise, key hyperparameters (k=260, N=1000, Huber loss, top-100 averaging, Dirichlet prior) are chosen without sensitivity analysis; only distance functions are compared in Appendix E, and Figure 2's inertia curve is not a predictive-validity check.
minor comments (6)
  1. [Throughout] The naming is inconsistent: DOMAIN2VEC, Domain2Vec, D OMAIN2VEC, and Domain2vec appear in different places; please unify the notation.
  2. [4.2] There are typos such as "stardard practice" and "REG MIX" in Figure 5; the grammar in "The computational complexity of these methods will grow non-linearly" should be corrected.
  3. [Appendix B vs. 4.2] Algorithms 2 and 3 describe top-1 selection, while Section 4.2 and the text mention averaging the top-100 predicted mixtures; make the final selection rule explicit and consistent in both places.
  4. [Eqs. (6)-(7)] The notation fi and vtrain is ambiguous: please specify explicitly that vtrain = Vtrain·r and that each fi is fitted on the 260-dimensional domain-vector space.
  5. [3.1 vs. 4.2] Section 3.1 says Dvalid is often i.i.d. with Dtrain, but Section 4.2 uses Pile-CC as validation and deliberately optimizes the mixture for Pile-CC; please clarify what "i.i.d." means in this setting and how the one-epoch restriction interacts with the data-amount constraint.
  6. [Figure 2] The inertia-versus-k curve is monotonically decreasing and does not by itself justify k=260; report ranking or downstream performance for several values of k as a sensitivity check.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DA2 is an explicit assumption and all predictions are checked against measured losses.

full rationale

The derivation chain is not circular. Equation 5 defines the training-free selector as the minimizer of a distance in domain-vector space, but this is explicitly introduced as the Distribution Alignment Assumption rather than derived from the data; Appendix A only gives a special-case MLE justification for exact distribution equality, not a construction that makes the distance objective equal to validation loss. The central evaluation in Table 2 compares the DA2 ranking against actual validation losses of trained 83M and 1.6B models, so the objective is empirically tested rather than defined to match the reported outcome. The Domain2Vec+RegMix variant fits per-meta-domain surrogate models and validates ranking accuracy on held-out mixture ratios (Figure 4), so its predictions are not forced by construction either. The Pile-CC-only baseline is a sanity check whose stated purpose is to verify the Pile-CC-validation-loss/downstream correlation, not to prove DA2. Self-citations (Fei et al. 2024; Ye et al. 2024; Wang et al. 2025) appear for datasets and related work, and none is load-bearing for the distance-minimization claim or the fitted surrogate. Concerns about the classifier training corpus overlapping evaluation sources or about hyperparameter choices (Huber loss, N=1000, top-100 averaging) are potential leakage/tuning issues, not reductions of a prediction to its own input.

Assumptions & free parameters 5 free parameters · 5 assumptions · 1 invented entities

The central method has few directly fitted scalars (k, N, distance, K, Dirichlet prior), but the entire representation rests on the constructed 260-class classifier and the 5.2TB clustering corpus, which are not released. The RegMix variant additionally fits 10,500 surrogate models. The main invented construct is the meta-domain itself.

free parameters (5)
  • Number of meta-domains (k) = 260 (120 English, 120 Chinese, 20 Code)
    Chosen by inspecting the inertia curve in Figure 2; determines the dimensionality of the domain vector and the granularity of the representation. Not derived from first principles.
  • Sample size N for domain vector estimation = 1000 documents
    Set in Section 2 as 'enough' for an accurate and stable domain vector, with no stability analysis or confidence interval reported.
  • Distance function in DA2 = Huber Loss
    Selected based on Appendix E, where it outperforms L1, L2, and JS distance on the Section 4.1 ranking task. This is a model choice tuned to the evaluation.
  • Number of candidate mixtures K and top-k averaging = 100,000 candidates, top-100 averaged (Section 4.2); top-1 used in Section 4.1
    The final mixture is the average of the best 100 sampled Dirichlet candidates. The choice of K and k affects the result and is not justified beyond stability.
  • Dirichlet prior (atrain) for sampling mixtures = token distribution of the training datasets
    Used to sample candidate mixtures in Algorithms 2 and 3. It biases the search space toward the natural token proportions and affects which mixtures are reachable.
assumptions (5)
  • domain assumption Distribution Alignment Assumption: lower validation loss when the training set's meta-domain distribution is better aligned with the validation set's distribution.
    Stated in Section 3.3 and Appendix A. It is the foundation of DA2 and is justified only by a heuristic MLE argument, not proven.
  • domain assumption Meta-domains form a sufficient basis: any dataset's LM-relevant features are captured by its distribution over 260 k-means clusters.
    Section 2 assumes spanning set and linear independence based on corpus diversity and deduplication. No evidence that cluster membership captures the properties that determine cross-domain transfer.
  • domain assumption Mixture-ratio rankings are invariant to model size.
    Section 3.2 supports this only for 83M vs 1.6B on one dataset pair; the training-free method generalizes this to larger models and other mixes.
  • domain assumption The validation set is a good proxy for downstream performance.
    Section 4.2 uses Pile-CC because RegMix found it correlates with downstream tasks; the method is not evaluated on an independent validation set in the main downstream experiment.
  • domain assumption The meta-domain classifier, trained on a 5.2TB corpus, generalizes to unseen datasets without leakage.
    Section 2 reports 74.73% accuracy on a constructed test set, but the corpus overlaps in source composition with evaluation corpora (The Pile, C4). The paper does not analyze this overlap.
invented entities (1)
  • Meta-domain
    purpose: A cluster of documents in embedding space that serves as a basis component for representing any dataset as a distribution.
    Defined operationally by k-means on bge embeddings; there is no external falsifiable prediction attached to a specific meta-domain. Its usefulness is only shown internally via t-SNE grouping and classifier accuracy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Domain2Vec: Vectorizing Datasets to Find the Optimal Data Mixture without Training." pith.science (2026). https://pith.science/paper/Z4REOJCG

@misc{pith2026250610952,
  author       = {Pith},
  title        = {Pith review of: Domain2Vec: Vectorizing Datasets to Find the Optimal Data Mixture without Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z4REOJCG}},
  note         = {Machine review of arXiv:2506.10952}
}
abstract

We introduce~\textsc{Domain2Vec}, a novel approach that decomposes any dataset into a linear combination of several \emph{meta-domains}, a new concept designed to capture the key underlying features of datasets. \textsc{Domain2Vec} maintains a vocabulary of meta-domains and uses a classifier to decompose any given dataset into a domain vector that corresponds to a distribution over this vocabulary. These domain vectors enable the identification of the optimal data mixture for language model (LM) pretraining in a training-free manner under the \emph{\textbf{D}istribution \textbf{A}lignment \textbf{A}ssumption} (DA$^{2}$), which suggests that when the data distributions of the training set and the validation set are better aligned, a lower validation loss is achieved. Moreover, \textsc{Domain2vec} can be seamlessly integrated into previous works to model the relationship between domain vectors and LM performance, greatly enhancing the efficiency and scalability of previous methods. Extensive experiments demonstrate that \textsc{Domain2Vec} helps find the data mixture that enhances downstream task performance with minimal computational overhead. Specifically, \textsc{Domain2Vec} achieves the same validation loss on Pile-CC using only $51.5\%$ of the computation required when training on the original mixture of The Pile dataset. Under equivalent compute budget, \textsc{Domain2Vec} improves downstream performance by an average of $2.83\%$.

Figures

Figures reproduced from arXiv: 2506.10952 by the authors.

Figure 1
Figure 1. The domain vector of each sub-dataset of The Pile (Gao et al., 2021), where each row corresponds to a sub-dataset and each column corresponds to a meta-domain. The higher the proportion of data belonging to a particular meta-domain, the closer the color of the corresponding cell is to blue. We display distribution on some English meta-domains for clarity. The full picture is shown in [PITH_FULL_IMAGE:figures/full_f… view at source ↗
Figure 2
Figure 2. The number of meta-domains vs. Inertia. After getting the corpora, we aim to extract the meta￾domains in the corpora, that is, to divide the corpora into some (semantically) different clusters, to serve as the meta￾domains. We employ k-means (Macqueen, 1967; Arthur & Vassilvitskii, 2006) clustering algorithm to implement the separation and utilize bge-small-en-v1.5 and bge-small-zh-v1.5 (Xiao et al., 2023) to comput… view at source ↗
Figure 3
Figure 3. The validation loss on the EuroParl (The Pile) and Stackexchange (RedPajama) of models trained using data mixture in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Relationship between the number of trained data mix￾tures and the Spearman correlation. to find the better mixture ratio than others. Moreover, the pilot study suggests that the predicted ranking holds across model sizes while the predicted loss becomes meaningless for…
Figure 5
Figure 5. Figure 5: The validation loss on the Pile-CC subset. DOMAIN2VEC achieves the comparable validation loss of Human (The model using original data mixture from The Pile), but uses only 51.5% training computational costs of Human. Using the same training cost, DOMAIN2VEC can reduce …
Figure 6
Figure 6. Figure 6: Visualization (t-SNE) of domain vectors of The Pile. use our proposed Domain2Vec to get a comparable down￾stream performance with lowest cost by mixing datasets from different sources. Notably, DOMAIN2VEC + DA2 and DOMAIN2VEC + REGMIX, using only about 0.26% of the FLO…
Figure 7
Figure 7. Figure 7: The Domain Vector of each sub-dataset of The Pile (Gao et al., 2021), where each row corresponds to a sub-dataset and each column corresponds to a meta-domain. The higher the proportion of data belonging to a particular meta-domain, the closer the color of the correspo…
Figure 8
Figure 8. Figure 8: The validation loss on different dataset of models trained using data mixture in [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: The validation loss on different dataset of models trained using data mixture in [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

55 extracted references · 24 canonical work pages

  1. [1]

    L., and Paul, M

    Ankner, Z., Blakeney, C., Sreenivasan, K., Marion, M., Leavitt, M. L., and Paul, M. Perplexed by perplexity: Perplexity-based data pruning with small reference models, 2024. URL https://arxiv.org/abs/2405.20541

    arXiv 2024

  2. [2]

    and Vassilvitskii, S

    Arthur, D. and Vassilvitskii, S. k-means++: The advantages of careful seeding. Technical report, Stanford, 2006

    work page 2006

  3. [3]

    L., Gao, J., and Choi, Y

    Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. Piqa: Reasoning about physical commonsense in natural language. In AAAI Conference on Artificial Intelligence, 2019. URL https://api.semanticscholar.org/CorpusID:208290939

    work page 2019

  4. [4]

    Cross-Table Pretraining towards a Universal Function Space for Heterogeneous Tabular Data

    Chen, J., Lin, Z., Chen, Q., and Sun, J. Cross-table pretraining towards a universal function space for heterogeneous tabular data. arXiv preprint arXiv:2406.00281, 2024

    work page Pith review arXiv 2024

  5. [5]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/1803.05457, 2018. URL https://api.semanticscholar.org/CorpusID:3922816

    arXiv 2018

  6. [6]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024

    DeepSeek-AI. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024

    2024

  7. [7]

    DOGE : Domain reweighting with generalization estimation

    Fan, S., Pagliardini, M., and Jaggi, M. DOGE : Domain reweighting with generalization estimation. In Second Agent Learning in Open-Endedness Workshop, 2023. URL https://openreview.net/forum?id=qiKqsqwYXm

    work page 2023

  8. [8]

    Query of cc: Unearthing large scale domain-specific knowledge from public corpora, 2024

    Fei, Z., Shao, Y., Li, L., Zeng, Z., He, C., Yan, H., Lin, D., and Qiu, X. Query of cc: Unearthing large scale domain-specific knowledge from public corpora, 2024. URL https://arxiv.org/abs/2401.14624

    arXiv 2024

Show all 55 references
  1. [9]

    The pile: An 800gb dataset of diverse text for language modeling

    Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. The pile: An 800gb dataset of diverse text for language modeling. CoRR, abs/2101.00027, 2021. URL https://arxiv.org/abs/2101.00027

    2021 arXiv

  2. [10]

    A framework for few-shot language model evaluation, 07 2024

    Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac'h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., a...

    2024

  3. [11]

    Data mixing made efficient: A bivariate scaling law for language model pretraining, 2024

    Ge, C., Ma, Z., Chen, D., Li, Y., and Ding, B. Data mixing made efficient: A bivariate scaling law for language model pretraining, 2024. URL https://arxiv.org/abs/2405.14908

    2024 arXiv

  4. [12]

    S em E val-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning

    Gordon, A., Kozareva, Z., and Roemmele, M. S em E val-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In Agirre, E., Bos, J., Diab, M., Manandhar, S., Marton, Y., and Yuret, D. (eds.), * SEM 2012: The First Joint Conference on Lexi...

    2012

  5. [13]

    The llama 3 herd of models

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  6. [14]

    CMR scaling law: Predicting critical mixture ratios for continual pre-training of language models

    Gu, J., Yang, Z., Ding, C., Zhao, R., and Tan, F. CMR scaling law: Predicting critical mixture ratios for continual pre-training of language models. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Langua...

    2024 doi

  7. [15]

    Data selection via optimal control for language models

    Gu, Y., Dong, L., Wang, H., Hao, Y., Dong, Q., Wei, F., and Huang, M. Data selection via optimal control for language models. arXiv preprint arXiv: 2410.07064, 2024 b

    2024 arXiv

  8. [16]

    V., and Smith, K

    Guo, S., Ren, Y., Albrecht, S. V., and Smith, K. Sample relationships through the lens of learning dynamics with label information. In First Workshop on Interpolation Regularizers and Beyond at NeurIPS 2022, 2022. URL https://openreview.net/forum?id=mIl1mMA7Uz

    2022

  9. [17]

    H., and Friedman, J

    Hastie, T., Tibshirani, R., Friedman, J. H., and Friedman, J. H. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009

    2009

  10. [18]

    A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vin...

    2022 arXiv

  11. [19]

    A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Rae, J

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Ra...

    2022

  12. [20]

    Huber, P. J. Robust Estimation of a Location Parameter . The Annals of Mathematical Statistics, 35 0 (1): 0 73 -- 101, 1964. doi:10.1214/aoms/1177703732. URL https://doi.org/10.1214/aoms/1177703732

    1964

  13. [21]

    S., Schmidt-Thieme, L., and Grabocka, J

    Jomaa, H. S., Schmidt-Thieme, L., and Grabocka, J. Dataset2vec: Learning dataset meta-features. Data Mining and Knowledge Discovery, 35 0 (3): 0 964--985, 2021

    2021

  14. [22]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models, 2020. URL https://arxiv.org/abs/2001.08361

    2020 arXiv

  15. [23]

    Lightgbm: A highly efficient gradient boosting decision tree

    Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., and Liu, T.-Y. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30, 2017

    2017

  16. [24]

    Looking beyond the surface: A challenge set for reading comprehension over multiple sentences

    Khashabi, D., Chaturvedi, S., Roth, M., Upadhyay, S., and Roth, D. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Walker, M., Ji, H., and Stent, A. (eds.), Proceedings of the 2018 Conference of the North A merican Chapter of t...

    2018 doi

  17. [25]

    RACE : Large-scale R e A ding comprehension dataset from examinations

    Lai, G., Xie, Q., Liu, H., Yang, Y., and Hovy, E. RACE : Large-scale R e A ding comprehension dataset from examinations. In Palmer, M., Hwa, R., and Riedel, S. (eds.), Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp.\ 785--794, Copenh...

    2017 doi

  18. [26]

    Not all tokens are what you need for pretraining

    Lin, Z., Gou, Z., Gong, Y., Liu, X., Xu, R., Lin, C., Yang, Y., Jiao, J., Duan, N., Chen, W., et al. Not all tokens are what you need for pretraining. Advances in Neural Information Processing Systems, 37: 0 29029--29063, 2024

    2024

  19. [27]

    Logiqa: a challenge dataset for machine reading comprehension with logical reasoning

    Liu, J., Cui, L., Liu, H., Huang, D., Wang, Y., and Zhang, Y. Logiqa: a challenge dataset for machine reading comprehension with logical reasoning. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI'20, 2021. ISBN 9780999241165

    2021

  20. [28]

    Regmix: Data mixture as regression for language model pre-training, 2024

    Liu, Q., Zheng, X., Muennighoff, N., Zeng, G., Dou, L., Pang, T., Jiang, J., and Lin, M. Regmix: Data mixture as regression for language model pre-training, 2024. URL https://arxiv.org/abs/2407.01492

    2024 arXiv

  21. [29]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  22. [30]

    Some methods for classification and analysis of multivariate observations

    Macqueen, J. Some methods for classification and analysis of multivariate observations. In Proceedings of 5-th Berkeley Symposium on Mathematical Statistics and Probability/University of California Press, 1967

    1967

  23. [31]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Conference on Empirical Methods in Natural Language Processing, 2018. URL https://api.semanticscholar.org/CorpusID:52183757

    2018

  24. [32]

    Gpt-4 technical report, 2024

    OpenAI. Gpt-4 technical report, 2024. URL https://arxiv.org/abs/2303.08774

    2024 arXiv

  25. [33]

    Q., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fern \'a ndez, R

    Paperno, D., Kruszewski, G., Lazaridou, A., Pham, N. Q., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fern \'a ndez, R. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Erk, K. and Smith, N. A. (eds.), Proceedings of the 54th Annual Meeti...

    2016 doi

  26. [34]

    B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L

    Penedo, G., Kydl \' c ek, H., allal, L. B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L. V., and Wolf, T. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchma...

    2024

  27. [35]

    D-cpt law: Domain-specific continual pre-training scaling law for large language models

    Que, H., Liu, J., Zhang, G., Zhang, C., Qu, X., Ma, Y., Duan, F., Bai, Z., Wang, J., Zhang, Y., et al. D-cpt law: Domain-specific continual pre-training scaling law for large language models. Advances in Neural Information Processing Systems, 37: 0 90318--90354, 2024

    2024

  28. [36]

    Qwen2 technical report, 2024

    Qwen Team . Qwen2 technical report, 2024. URL https://arxiv.org/abs/2407.10671

    2024 arXiv

  29. [37]

    Improving language understanding by generative pre-training

    Radford, A. Improving language understanding by generative pre-training. 2018

    2018

  30. [38]

    Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., Rutherford, E., Hennigan, T., Menick, J., Cassirer, A., Powell, R., van den Driessche, G., Hendricks, L. A., Rauh, M., Huang, P.-S., Glaese, A., et al. S...

    2022 arXiv

  31. [39]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21 0 (1), jan 2020. ISSN 1532-4435

    2020

  32. [40]

    W., Hashimoto, T

    Sagawa*, S., Koh*, P. W., Hashimoto, T. B., and Liang, P. Distributionally robust neural networks. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=ryxGuJrFvS

    2020

  33. [41]

    L., Bhagavatula, C., and Choi, Y

    Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: an adversarial winograd schema challenge at scale. Commun. ACM, 64 0 (9): 0 99–106, August 2021. ISSN 0001-0782. doi:10.1145/3474381. URL https://doi.org/10.1145/3474381

    2021 doi

  34. [42]

    Social IQ a: Commonsense reasoning about social interactions

    Sap, M., Rashkin, H., Chen, D., Le Bras, R., and Choi, Y. Social IQ a: Commonsense reasoning about social interactions. In Inui, K., Jiang, J., Ng, V., and Wan, X. (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Intern...

    2019 doi

  35. [43]

    Self-influence guided data reweighting for language model pre-training

    Thakkar, M., Bolukbasi, T., Ganapathy, S., Vashishth, S., Chandar, S., and Talukdar, P. Self-influence guided data reweighting for language model pre-training. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https://openreview.net/forum?id...

    2023

  36. [44]

    C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., et al

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., et al. Llama 2: Open foundation...

    2023 arXiv

  37. [45]

    and Hinton, G

    Van der Maaten, L. and Hinton, G. Visualizing data using t-sne. Journal of machine learning research, 9 0 (11), 2008

    2008

  38. [46]

    Wang, X., Tissue, H., Wang, L., Li, L., and Zeng, D. D. Learning dynamics in continual pre-training for large language models. arXiv preprint arXiv: 2505.07796, 2025

    2025 arXiv

  39. [47]

    Redpajama: an open dataset for training large language models

    Weber, M., Fu, D., Anthony, Q., Oren, Y., Adams, S., Alexandrov, A., Lyu, X., Nguyen, H., Yao, X., Adams, V., Athiwaratkun, B., Chalamala, R., Chen, K., Ryabinin, M., Dao, T., Liang, P., Ré, C., Rish, I., and Zhang, C. Redpajama: an open dataset for training large language mod...

    2024 arXiv

  40. [48]

    F., and Gardner, M

    Welbl, J., Liu, N. F., and Gardner, M. Crowdsourcing multiple choice science questions. In Derczynski, L., Xu, W., Ritter, A., and Baldwin, T. (eds.), Proceedings of the 3rd Workshop on Noisy User-generated Text, pp.\ 94--106, Copenhagen, Denmark, September 2017. Association f...

    2017 doi

  41. [49]

    C-pack: Packaged resources to advance general chinese embedding, 2023

    Xiao, S., Liu, Z., Zhang, P., and Muennighoff, N. C-pack: Packaged resources to advance general chinese embedding, 2023

    2023

  42. [50]

    M., Pham, H., Dong, X., Du, N., Liu, H., Lu, Y., Liang, P., Le, Q

    Xie, S. M., Pham, H., Dong, X., Du, N., Liu, H., Lu, Y., Liang, P., Le, Q. V., Ma, T., and Yu, A. W. Doremi: Optimizing data mixtures speeds up language model pretraining. In Thirty-seventh Conference on Neural Information Processing Systems, 2023 a . URL https://openreview.ne...

    2023

  43. [51]

    M., Santurkar, S., Ma, T., and Liang, P

    Xie, S. M., Santurkar, S., Ma, T., and Liang, P. S. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36: 0 34201--34227, 2023 b

    2023

  44. [52]

    Data mixing laws: Optimizing data mixtures by predicting language modeling performance, 2024

    Ye, J., Liu, P., Sun, T., Zhou, Y., Zhan, J., and Qiu, X. Data mixing laws: Optimizing data mixtures by predicting language modeling performance, 2024. URL https://arxiv.org/abs/2403.16952

    2024 arXiv

  45. [53]

    Hellaswag: Can a machine really finish your sentence? In Annual Meeting of the Association for Computational Linguistics, 2019

    Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? In Annual Meeting of the Association for Computational Linguistics, 2019. URL https://api.semanticscholar.org/CorpusID:159041722

    2019

  46. [54]

    LIMA : Less is more for alignment

    Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., YU, L., Zhang, S., Ghosh, G., Lewis, M., Zettlemoyer, L., and Levy, O. LIMA : Less is more for alignment. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://...

    2023

  47. [55]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.