Pith. sign in

REVIEW 4 major objections 6 minor 63 references

Chameleon: A Flexible Data-mixing Framework for Language Model Pretraining and Finetuning

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that domain reweighting for language-model training reduces to computing kernel ridge leverage scores on averaged proxy-model embeddings, with inverse scores for pretraining and direct scores for finetuning.

desk verdict A cheap, practical data-mixing heuristic with consistent empirical gains, but the pretraining direction is under-ablated and temperatures are under-reported. read the letter →

arxiv 2505.24844 v1 pith:QT6I6LTY submitted 2025-05-30 cs.LG cs.CL

Wanyun Xie , Francesco Tonin , Volkan Cevher This is my paper
classification cs.LGcs.CL
keywords kernelridgeleveragescoresdomainreweightingdatamixinglargelanguagemodelpretrainingfinetuningtransferChristoffelfunctionSlimPajama
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Chameleon tries to establish that the optimal-looking data mixture for training a language model can be read off directly from the geometry of a small proxy model's embeddings, without running a per-domain optimization loop. It proposes computing kernel ridge leverage scores (KRLS) on the $k \times k$ matrix of pairwise domain similarities; domains whose averaged embeddings are easily reconstructed from the other domains (low KRLS, dense regions) are upweighted for pretraining, while domains that are hard to reconstruct (high KRLS, unique regions) are upweighted for finetuning. The experiments claim this single recipe matches or beats proxy-optimization baselines on pretraining perplexity and downstream reasoning with about a tenth to a fifth of their compute, that the weights transfer to a new 17-domain corpus without retraining the proxy, and that finetuning perplexity improves on all 14 domains tested. If true, domain reweighting becomes a cheap, stage-adaptive, data-centric step instead of a costly optimization procedure that must be rerun whenever the corpus changes.

What carries the argument

The carrying object is the kernel ridge leverage score computed on domain embeddings rather than on individual tokens. A proxy model (trained once, uniformly) supplies layer-$L$ hidden states; their within-domain averages form the rows of $X \in \mathbb{R}^{k \times p}$, and the linear kernel gives $\Omega_D = XX^\top$. The KRLS $S_\lambda(D_i)$, the $i$-th diagonal entry of the ridge hat matrix $\Omega_D(\Omega_D + k\lambda I)^{-1}$, is the influence measure: high values mark domains that are linearly independent of the others. Pretraining weights are a softmax of the inverse scores $S_\lambda^{-1}$, which the paper connects to Christoffel functions and hence to local data density; finetuning weights are a softmax of the raw scores. The whole computation is $O(k^3)$ plus one forward pass per sampled example, and it is the same score used in both directions that lets the method shift from general knowledge (dense regions) to specialization (unique regions).

What would settle it

Build a two-domain corpus where the domain with the lowest KRLS (highest inverse score, hence most upweighted by Chameleon) is deliberately degenerate text such as repeated boilerplate, while the high-KRLS domain is clean, informative text; if training with Chameleon's weights does not degrade downstream reasoning relative to uniform mixing, the claim that embedding density tracks pretraining value fails.

Watch

Extended reading notes

Core claim

The central claim is that a domain's importance for language-model training is encoded in the linear geometry of averaged hidden-layer embeddings. Concretely, after training a small proxy model on uniformly weighted data, each domain $D_i$ is represented by the mean embedding $x_i$ of its samples; the domain affinity matrix $\Omega_D = XX^\top$ collects pairwise similarities. The KRLS $S_\lambda(D_i) = [\Omega_D(\Omega_D + k\lambda I)^{-1}]_{ii}$ then measures how much of domain $i$'s representation is unique rather than explainable by the other domains. The paper asserts that for pretraining the useful signal is the opposite: domains with low KRLS, i.e., high Christoffel-function density or strong linear dependency, should be sampled more, while for finetuning the unique, high-KRLS domains should be sampled more. Across the SlimPajama, Pile, Wiki40b, and Stack experiments, the paper reports that this assignment improves average perplexity and reasoning accuracy over uniform mixing and reaches parity with or better than expensive baselines.

Load-bearing premise

The load-bearing premise is that a domain's usefulness for training is faithfully ranked by how easily its average embedding can be recreated as a linear combination of the other domains' average embeddings, with dense, shared regions best for pretraining and unique regions best for finetuning.

Editorial extensions

If this is right

  • A corpus can be extended with new domains at inference-only cost: forward passes through the existing proxy produce embeddings, from which new weights are computed without retraining.
  • The same KRLS score, inverted or not, serves both training stages, so a single data-mixing framework can replace separate pretraining and finetuning mixture sweeps.
  • Weight selection costs drop to $O(k^3)$ plus embedding extraction, which in the reported settings is roughly 10-20% of the FLOPs of prior proxy-optimization methods.
  • Because weights stabilize within about 1k-2k proxy steps and transfer from an 82M proxy to 684M and 1.2B base models, cheap small-scale proxy runs suffice for larger training pipelines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit is that the Christoffel/density justification does not uniquely single out KRLS; any density estimator on the same embeddings (e.g., local neighborhood counts) could be compared to see whether the ranking, not the specific score, drives the gains.
  • Because the scores depend only on pairwise inner products of averaged embeddings, they are invariant to orthogonal transformations of the embedding space; a testable consequence is that any model with similar relative domain geometry, even one not trained on the target corpus, would yield similar mixtures.
  • The finetuning results suggest a broader principle: data-mixing strategies should be phase-dependent, and the same uniqueness-vs-density axis could guide instruction-tuning or domain-adaptation data selection, not just the two stages tested here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript proposes Chameleon, a data-mixing framework that computes per-domain weights from kernel ridge leverage scores (KRLS) over averaged proxy-model embeddings. A domain affinity matrix is built from the linear kernel on domain-mean embeddings; KRLS scores S_λ are computed; pretraining weights are a softmax of inverse scores (α_PT), while finetuning weights are a softmax of the scores themselves (α_FT). The authors claim that α_PT matches DoReMi and DoGE at a fraction of the compute, that the method transfers to new data (the Pile) without retraining the proxy, and that α_FT improves finetuning perplexity on all tested domains. Experiments include 684M and 1.2B pretraining on SlimPajama, transfer to the Pile, and finetuning on Wiki40b and Stack.

Significance. If the results hold, Chameleon offers a simple, cheap, and flexible alternative to optimization-based mixture methods. Strengths of the paper include the public code release; the breadth of experiments (two base-model sizes, transfer to a 17-domain dataset, finetuning on 14 domains); the stability analysis across model size, λ, and embedding sample count (Table 16); and the finetuning direction ablation in Appendix B.9. However, the empirical evidence is weakened by single-run evaluations, hand-picked and unreported softmax temperatures, and the absence of any ablation of the inverse-score direction in pretraining. The theoretical motivation via Christoffel functions is not established at the operating point of k=7–17 domain-mean vectors.

major comments (4)
  1. [Section 3.2 / Algorithm 1] The central claim that inverse KRLS ordering improves pretraining is never ablated. The only direction ablation is for finetuning (Appendix B.9, Tables 22–23), where α_FT beats α_PT; there is no pretraining comparison of α_PT against α_FT, against a permuted or randomized score vector, or against a simple heuristic that upweights the visually 'central' domains CC and C4. Since the temperature τ_PT can drive the softmax toward uniformity, the reported Table 2 average perplexity (22.31 for Chameleon vs 24.20 for Uniform) could in principle be reproduced by a large-temperature softmax over almost any score vector. A pretraining direction ablation is required to establish that the KRLS geometry, rather than the softmax transformation or dataset-specific effects, is responsible for the gains.
  2. [Appendix B.2 / Table 16] The softmax temperatures τ_PT ∈ [5,10] and τ_FT ∈ [0.2,0.5] are hand-picked ranges, yet no per-experiment values are reported in Tables 2, 6, 7, 9, or 10, and the stability analysis in Table 16 varies model size, λ, and number of samples but not τ. With τ_PT = 10, exp(S^{-1}_λ/τ) is nearly uniform across domains, so the effective weight vector is much closer to uniform than the raw inverse-KRLS ordering would suggest. The paper's claim that the method is 'hyperparameter-robust' (Section B.5) therefore does not cover the parameter that most directly controls the final mixture. The authors should report τ for every experiment and include a sensitivity sweep over τ in the stability analysis.
  3. [Tables 2, 3, 6, 7, 9, 10] All main results are single runs with no error bars, multiple seeds, or significance tests. Several headline differences are extremely small: Table 2 average PPL is 22.31 for Chameleon versus 22.32 for DoGE (a difference of 0.01), and Table 3 average downstream accuracy is 39.6 versus 39.4 for DoGE. At the 1.2B scale (Table 5), Chameleon's average accuracy is 41.5 versus 41.1 for RegMix, a 0.4-point gap. Without variance estimates, the abstract's claim of matching or outperforming DoReMi/DoGE is not statistically distinguishable from noise. The finetuning results (Tables 9–10) show larger and uniform gains, but they too are single runs.
  4. [Section 4.2 / Table 7] The transfer-to-Pile comparison is not fully controlled. DoReMi and RegMix weights are taken from their respective papers (Appendix B.7, Table 19), whereas Chameleon and DoGE weights are 'from our own experiments'; the base-model training setup, compute budget, and hyperparameters may differ across these sources. The 'extra FLOPs' column in Table 7 mixes recomputed costs (Chameleon, DoGE) with published costs (DoReMi, RegMix), so the claimed factor-of-145-to-290 savings is not an apples-to-apples comparison. The authors should either recompute all baselines in the same pipeline or clearly state that the comparison is partly cross-paper and restrict the efficiency claim to the proxy-retraining cost.
minor comments (6)
  1. [Section 3.1 / Appendix A.1] The KRLS definition is inconsistent: Definition 3.1 uses (Ω_D + kλI)^{-1}, while equation (1) in Appendix A.1 uses (Ω_D + λI)^{-1}; please unify the notation.
  2. [Abstract / Sections 1, 4.3] The word 'pertaining' appears where 'pretraining' is intended (e.g., 'general knowledge for pertaining' and 'Besides pertaining'); please correct these typos.
  3. [Lemma A.1] The feature map's domain and codomain are swapped between the introductory paragraph (φ: R^p → R^d) and the lemma statement (φ: R^d → R^p); the matrix dimensions in the proof imply the former, so the lemma should be corrected.
  4. [Algorithm 1 / Appendix B.2] Algorithm 1 does not include the temperature parameter τ that is introduced in Appendix B.2; since τ is used in all reported experiments, the algorithm should either take τ as input or state the default values used.
  5. [Tables 2, 6, 7] The FLOPs values are typeset with broken superscripts (e.g., '1.36×10 17'), making them difficult to read; please use proper exponent formatting.
  6. [Figure 1] The labels 'KRLSS−1 λ' and 'KRLS Sλ' are garbled; they should be typeset with clear subscripts and superscripts, e.g., S^{-1}_λ and S_λ.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: KRLS weights are computed from unsupervised embeddings and evaluated on held-out perplexity and accuracy, with no target-derived fitting or load-bearing self-citations.

full rationale

The paper's derivation chain is not circular. Domain weights αPT and αFT are obtained by Algorithm 1 from proxy-model embeddings and the domain affinity matrix Ω_D = XX^T (Section 3.1, Definition 3.1), with no use of the held-out perplexities, downstream accuracies, or finetuning test losses that are later reported. The pretraining choice of inverse KRLS and the finetuning choice of direct KRLS are motivated by the Christoffel-function interpretation cited to Pauwels et al. (2018) and by the finetuning ablation in Tables 22–23 (Appendix B.9); the pretraining direction itself is not ablated, and the softmax temperature τ_PT ∈ [5,10] is not reported per run, but these are experimental and robustness gaps rather than reductions by construction. No load-bearing self-citations appear: the cited transferability results (Xie et al. 2023, Fan et al. 2024b, Liu et al. 2024) and Christoffel-function results are external to the present authors. Accordingly the reported improvements over Uniform, DoReMi, and DoGE, and the transfer to the Pile, are empirical claims whose outcome is not forced by the definition of the weights.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

Chameleon does not introduce new physical entities and its KRLS scores are computed from data rather than fitted to downstream metrics. The weights nevertheless depend on hand-chosen hyperparameters (lambda, two softmax temperatures, embedding layer) and on domain assumptions about embedding geometry and training-phase objectives. The least supported of these is the link between Christoffel-function density and pretraining usefulness.

free parameters (4)
  • Softmax temperature for pretraining weights (tau_PT) = Range [5,10]; exact value not reported
    Controls the sharpness of the inverse-KRLS weight distribution; the paper reports a typical range without a selection criterion, and the delivered mixtures depend on it.
  • Softmax temperature for finetuning weights (tau_FT) = Range [0.2,0.5]; exact value not reported
    Controls the sharpness of the KRLS weight distribution in finetuning; it is an order of magnitude smaller than tau_PT and no tuning protocol is given.
  • Regularization lambda in KRLS = Not stated for main runs; robustness shown for lambda in {1,10,100} (Table 16)
    Regularization changes the KRLS scores by soft-thresholding the spectrum of the affinity matrix, so the exact value matters for the reported weights.
  • Embedding layer index L = Not specified in main text
    The choice of hidden layer from which domain embeddings are extracted is not justified and can shift the affinity matrix and therefore the weights.
assumptions (5)
  • standard math The kernel ridge leverage score and its relation to the Christoffel function hold as established in the cited literature.
    The paper proves the RLS/KRLS equivalence in Section A.1 and relies on Pauwels et al. (2018) for the Christoffel connection in Section A.2, using it to justify the pretraining weight direction.
  • domain assumption Averaged mid-layer LM embeddings of a domain capture semantically and structurally meaningful domain characteristics, so inner products between these averages reflect inter-domain relationships.
    Section 3.1 defines the domain embedding as a mean of proxy embeddings and gives a UMAP visualization as evidence, not a proof.
  • domain assumption Upweighting high-density regions (low KRLS, high Christoffel function) during pretraining improves downstream generalization.
    Remark 3.2 states this as theoretical motivation; no theorem or controlled experiment isolates this mechanism from the choice of score direction.
  • domain assumption Upweighting high-KRLS regions (unique domains) during finetuning improves specialization.
    Section 3.2 asserts that pretraining and finetuning objectives differ fundamentally; the only direct evidence is the comparison in Tables 22-23, which tests alpha_FT against alpha_PT after both have been run.
  • domain assumption Domain weights computed from an 82M proxy transfer to 684M and 1.2B base models.
    Section 4.1 cites prior work and reports consistent transfer results, but the transferability is an empirical assumption rather than a guaranteed property.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Chameleon: A Flexible Data-mixing Framework for Language Model Pretraining and Finetuning." pith.science (2026). https://pith.science/paper/QT6I6LTY

@misc{pith2026250524844,
  author       = {Pith},
  title        = {Pith review of: Chameleon: A Flexible Data-mixing Framework for Language Model Pretraining and Finetuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QT6I6LTY}},
  note         = {Machine review of arXiv:2505.24844}
}
read the original abstract

Training data mixtures greatly impact the generalization performance of large language models. Existing domain reweighting methods often rely on costly weight computations and require retraining when new data is introduced. To this end, we introduce a flexible and efficient data mixing framework, Chameleon, that employs leverage scores to quantify domain importance within a learned embedding space. We first construct a domain affinity matrix over domain embeddings. The induced leverage scores determine a mixture that upweights domains sharing common representations in embedding space. This formulation allows direct transfer to new data by computing the new domain embeddings. In experiments, we demonstrate improvements over three key scenarios: (i) our computed weights improve performance on pretraining domains with a fraction of the compute of existing methods; (ii) Chameleon can adapt to data changes without proxy retraining, boosting few-shot reasoning accuracies when transferred to new data; (iii) our method enables efficient domain reweighting in finetuning, consistently improving test perplexity on all finetuning domains over uniform mixture. Our code is available at https://github.com/LIONS-EPFL/Chameleon.

Figures

Figures reproduced from arXiv: 2505.24844 by the authors.

Figure 1
Figure 1. Pipeline of domain reweighting via CHAMELEON. The given data is first embedded through the proxy model, previously trained on a corpus D with uniform weights. Domain embeddings are then determined by averaging the embeddings for each domain. The domain affinity matrix ⌦D is computed as the pairwise inner products between domain embeddings. Finally, (KRLS) is applied to ⌦D to obtain score S￾ indicating the degree of … view at source ↗
Figure 2
Figure 2. Domain weights on SlimPajama. We compare weights computed by data-mixing methods on SlimPajama [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. KRLS scores converge quickly. Domain weights across methods during proxy training. DoReMi and DoGE require more iterations to stabilize, while CHAMELEON con￾verges quickly after 1k iterations. The detailed discussion is in Section B.5. computation and for training 684M base model, showing that the proxy training cost is non-negligible. Compared to DoReMi and DoGE, we reduce computational overhead to less than 2% of … view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Domains in embedding space. 2D UMAP vi￾sualization of embeddings of SlimPajama learned by the proxy model. Semantically similar domains occupy sim￾ilar regions in embedding space, creating high-density clusters. Arxiv Book CC C4 Github Stackexchange Wikipedia Arxiv Boo…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

63 extracted references · 38 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Alaoui, A. E. and Mahoney, M. W. Fast Randomized Kernel Methods With Statistical Guarantees , 2015. arXiv:1411.0306

    arXiv 2015

  3. [3]

    Albalak, A., Pan, L., Raffel, C., and Wang, W. Y. Efficient online data mixing for language model pre-training. In R0-FoMo: Robustness of Few-shot and Zero-shot Learning in Large Foundation Models Workshop, 2023

    work page 2023

  4. [4]

    Sharp analysis of low-rank kernel matrix approximations

    Bach, F. Sharp analysis of low-rank kernel matrix approximations. In Conference on Learning Theory, pp.\ 185--209. PMLR, 2013

    work page 2013

  5. [5]

    B., and Stylianopoulos, N

    Beckermann, B., Putinar, M., Saff, E. B., and Stylianopoulos, N. Perturbations of christoffel–darboux kernels: Detection of outliers. Foundations of Computational Mathematics, 21 0 (1): 0 71--124, 2021

    work page 2021

  6. [6]

    Piqa: Reasoning about physical commonsense in natural language

    Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense in natural language. In AAAI Conference on Artificial Intelligence, 2020

    work page 2020

  7. [7]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp.\ 1877--1901, 2020

    1901

  8. [8]

    Analysis of nystr \"o m method with sequential ridge leverage score sampling

    Calandriello, D., Lazaric, A., and Valko, M. Analysis of nystr \"o m method with sequential ridge leverage score sampling. In Uncertainty in Artificial Intelligence Conference, 2016

    work page 2016

Show all 63 references
  1. [9]

    and Yang, Y

    Chen, Y. and Yang, Y. Fast statistical leverage score approximation in kernel ridge regression. In International Conference on Artificial Intelligence and Statistics (AISTATS), pp.\ 2935--2943. PMLR, 2021

    2021

  2. [10]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  3. [11]

    B., Musco, C., and Musco, C

    Cohen, M. B., Musco, C., and Musco, C. Ridge leverage scores for low-rank approximation. arXiv preprint arXiv:1511.07263, 6, 2015

    2015 arXiv

  4. [12]

    B., Musco, C., and Musco, C

    Cohen, M. B., Musco, C., and Musco, C. Input sparsity time low-rank approximation via ridge leverage score sampling. In Proceedings of the Twenty-Eighth Annual ACM-SIAM Symposium on Discrete Algorithms, pp.\ 1758--1777. SIAM, 2017

    2017

  5. [13]

    Multivariate christoffel functions and hyperinterpolation

    De Marchi, S., Sommariva, A., and Vianello, M. Multivariate christoffel functions and hyperinterpolation. Dolomites Research Notes on Approximation, 7 0 (Special Issue), 2014

    2014

  6. [14]

    M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A

    Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Firat, O., et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning (ICML), pp.\ 5547--5569. PMLR, 2022

    2022

  7. [15]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  8. [16]

    Leveraging the christoffel function for outlier detection in data streams

    Ducharlet, K., Travé-Massuyès, L., Lasserre, J.-B., Le Lann, M.-V., and Miloudi, Y. Leveraging the christoffel function for outlier detection in data streams. International Journal of Data Science and Analytics, 2024

    2024

  9. [17]

    Dunkl, C. F. and Xu, Y. Orthogonal polynomials of several variables, volume 155. Cambridge University Press, 2014

    2014

  10. [18]

    Dynamic gradient alignment for online data mixing

    Fan, S., Grangier, D., and Ablin, P. Dynamic gradient alignment for online data mixing. arXiv preprint arXiv:2410.02498, 2024 a

    2024 arXiv

  11. [19]

    DOGE : Domain reweighting with generalization estimation

    Fan, S., Pagliardini, M., and Jaggi, M. DOGE : Domain reweighting with generalization estimation. In International Conference on Machine Learning (ICML), 2024 b

    2024

  12. [20]

    Fanuel, M., Schreurs, J., and Suykens, J. A. Nystr \"o m landmark sampling and regularized christoffel functions. Machine Learning, 111 0 (6): 0 2213--2254, 2022

    2022

  13. [21]

    Maximize your data's potential: Enhancing llm accuracy with two-phase pretraining

    Feng, S., Prabhumoye, S., Kong, K., Su, D., Patwary, M., Shoeybi, M., and Catanzaro, B. Maximize your data's potential: Enhancing llm accuracy with two-phase pretraining. arXiv preprint arXiv:2412.15285, 2024

    2024 arXiv

  14. [22]

    The pile: An 800gb dataset of diverse text for language modeling

    Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020

    2020 arXiv

  15. [23]

    A framework for few-shot language model evaluation, 2024

    Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac'h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., a...

    2024

  16. [24]

    An introduction to statistical learning: with applications in R

    Gareth, J., Daniela, W., Trevor, H., and Robert, T. An introduction to statistical learning: with applications in R. Spinger, 2013

    2013

  17. [25]

    Wiki-40b: Multilingual language model dataset

    Guo, M., Dai, Z., Vrande c i \'c , D., and Al-Rfou, R. Wiki-40b: Multilingual language model dataset. In Proceedings of the Twelfth Language Resources and Evaluation Conference, 2020

    2020

  18. [26]

    The elements of statistical learning: data mining, inference and prediction

    Hastie, T., Tibshirani, R., Friedman, J., and Franklin, J. The elements of statistical learning: data mining, inference and prediction. The Mathematical Intelligencer, 27 0 (2): 0 83--85, 2005

    2005

  19. [27]

    Jiang, Y., Zhou, A., Feng, Z., Malladi, S., and Kolter, J. Z. Adaptive data optimization: Dynamic sample selection with scaling laws. arXiv preprint arXiv:2410.11820, 2024

    2024 arXiv

  20. [28]

    Autoscale: Automatic prediction of compute-optimal data composition for training llms

    Kang, F., Sun, Y., Wen, B., Chen, S., Song, D., Mahmood, R., and Jia, R. Autoscale: Automatic prediction of compute-optimal data composition for training llms. arXiv preprint arXiv:2407.20177, 2024

    2024

  21. [29]

    Looking beyond the surface: A challenge set for reading comprehension over multiple sentences

    Khashabi, D., Chaturvedi, S., Roth, M., Upadhyay, S., and Roth, D. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistic...

    2018

  22. [30]

    B., Li, J., Mou, C., Ferrandis, C

    Kocetkov, D., Li, R., Allal, L. B., Li, J., Mou, C., Ferrandis, C. M., Jernite, Y., Mitchell, M., Hughes, S., Wolf, T., et al. The stack: 3 tb of permissively licensed source code. arXiv preprint arXiv:2211.15533, 2022

    2022 arXiv

  23. [31]

    Race: Large-scale reading comprehension dataset from examinations

    Lai, G., Xie, Q., Liu, H., Yang, Y., and Hovy, E. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683, 2017

    2017 arXiv

  24. [32]

    Lasserre, J. B. and Pauwels, E. The empirical christoffel function with applications in data analysis. Advances in Computational Mathematics, 45 0 (3): 0 1439--1468, 2019

    2019

  25. [33]

    L., and Peng, R

    Li, M., Miller, G. L., and Peng, R. Iterative row sampling. In 2013 IEEE 54th Annual Symposium on Foundations of Computer Science, pp.\ 127--136, 2013

    2013

  26. [34]

    Logiqa: A challenge dataset for machine reading comprehension with logical reasoning

    Liu, J., Cui, L., Liu, H., Huang, D., Wang, Y., and Zhang, Y. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. arXiv preprint arXiv:2007.08124, 2020

    2007 arXiv

  27. [35]

    Regmix: Data mixture as regression for language model pre-training

    Liu, Q., Zheng, X., Muennighoff, N., Zeng, G., Dou, L., Pang, T., Jiang, J., and Lin, M. Regmix: Data mixture as regression for language model pre-training. arXiv preprint arXiv:2407.01492, 2024

    2024 arXiv

  28. [36]

    A pretrainer`s guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity

    Longpre, S., Yauney, G., Reif, E., Lee, K., Roberts, A., Zoph, B., Zhou, D., Wei, J., Robinson, K., Mimno, D., and Ippolito, D. A pretrainer`s guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. In Proceedings of the 2024 Conference...

    2024

  29. [37]

    At which training stage does code data help llms reasoning? arXiv preprint arXiv:2309.16298, 2023

    Ma, Y., Liu, Y., Yu, Y., Zhang, Y., Jiang, Y., Wang, C., and Li, S. At which training stage does code data help llms reasoning? arXiv preprint arXiv:2309.16298, 2023

    2023 arXiv

  30. [38]

    Mahoney, M. W. and Drineas, P. CUR matrix decompositions for improved data analysis. Proceedings of the National Academy of Sciences, 106 0 (3): 0 697--702, 2009

    2009

  31. [39]

    When not to trust language models: Investigating effectiveness of parametric and non-parametric memories

    Mallen, A., Asai, A., Zhong, V., Das, R., Khashabi, D., and Hajishirzi, H. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Annual Meeting of the Association for Computational Linguistics (ACL), Toronto, Canada, July ...

    2023

  32. [40]

    Umap: Uniform manifold approximation and projection for dimension reduction

    McInnes, L., Healy, J., and Melville, J. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426, 2018

    2018 arXiv

  33. [41]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789, 2018

    2018 arXiv

  34. [42]

    and Musco, C

    Musco, C. and Musco, C. Recursive sampling for the nystrom method. In Advances in Neural Information Processing Systems (NeurIPS), volume 30. Curran Associates, Inc., 2017

    2017

  35. [43]

    N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fern \'a ndez, R

    Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q. N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fern \'a ndez, R. The LAMBADA dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.06031, 2016

    2016 arXiv

  36. [44]

    Data, data everywhere: A guide for pretraining dataset construction

    Parmar, J., Prabhumoye, S., Jennings, J., Liu, B., Jhunjhunwala, A., Wang, Z., Patwary, M., Shoeybi, M., and Catanzaro, B. Data, data everywhere: A guide for pretraining dataset construction. arXiv preprint arXiv:2407.06380, 2024

    2024 arXiv

  37. [45]

    B., Zafar, A., Khan, A., and Shahid, A

    Parthasarathy, V. B., Zafar, A., Khan, A., and Shahid, A. The ultimate guide to fine-tuning llms from basics to breakthroughs: An exhaustive review of technologies, research, best practices, applied research challenges and opportunities. arXiv preprint arXiv:2408.13296, 2024

    2024 arXiv

  38. [46]

    Relating leverage scores and density using regularized christoffel functions

    Pauwels, E., Bach, F., and Vert, J.-P. Relating leverage scores and density using regularized christoffel functions. In Advances in Neural Information Processing Systems (NeurIPS), volume 31, 2018

    2018

  39. [47]

    Falkon: An optimal large scale kernel method

    Rudi, A., Carratino, L., and Rosasco, L. Falkon: An optimal large scale kernel method. In Advances in Neural Information Processing Systems (NeurIPS), volume 30. Curran Associates, Inc., 2017

    2017

  40. [48]

    On fast leverage score sampling and optimal learning

    Rudi, A., Calandriello, D., Carratino, L., and Rosasco, L. On fast leverage score sampling and optimal learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 31. Curran Associates, Inc., 2018

    2018

  41. [49]

    W., Hashimoto, T

    Sagawa, S., Koh, P. W., Hashimoto, T. B., and Liang, P. Distributionally robust neural networks. In International Conference on Learning Representations (ICLR), 2020

    2020

  42. [50]

    L., Bhagavatula, C., and Choi, Y

    Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64, 2021

    2021

  43. [51]

    Socialiqa: Commonsense reasoning about social interactions

    Sap, M., Rashkin, H., Chen, D., LeBras, R., and Choi, Y. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728, 2019

    1904 arXiv

  44. [52]

    Superglue: Learning feature matching with graph neural networks

    Sarlin, P.-E., DeTone, D., Malisiewicz, T., and Rabinovich, A. Superglue: Learning feature matching with graph neural networks. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020

    2020

  45. [53]

    Sch \"o lkopf, B., Herbrich, R., and Smola, A. J. A generalized representer theorem. In Conference on Learning Theory, pp.\ 416--426. Springer, 2001

    2001

  46. [54]

    Slimpajama-dc: Understanding data combinations for llm training

    Shen, Z., Tao, T., Ma, L., Neiswanger, W., Liu, Z., Wang, H., Tan, B., Hestness, J., Vassilieva, N., Soboleva, D., et al. Slimpajama-dc: Understanding data combinations for llm training. arXiv preprint arXiv:2309.10818, 2023

    2023 arXiv

  47. [55]

    R., Hestness, J., and Dey, N

    Soboleva, D., Al-Khateeb, F., Myers, R., Steeves, J. R., Hestness, J., and Dey, N. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama , 2023. URL https://huggingface.co/datasets/cerebras/SlimPajama-627B

    2023

  48. [56]

    Vapnik, V. N. An overview of statistical learning theory. IEEE Transactions on Neural Networks, 10 0 (5): 0 988--999, 1999

    1999

  49. [57]

    N., Kaiser, L

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017

    2017

  50. [58]

    Glue: A multi-task benchmark and analysis platform for natural language understanding

    Wang, A. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018

    2018 arXiv

  51. [59]

    F., and Gardner, M

    Welbl, J., Liu, N. F., and Gardner, M. Crowdsourcing multiple choice science questions. arXiv preprint arXiv:1707.06209, 2017

    2017 arXiv

  52. [60]

    M., Pham, H., Dong, X., Du, N., Liu, H., Lu, Y., Liang, P., Le, Q

    Xie, S. M., Pham, H., Dong, X., Du, N., Liu, H., Lu, Y., Liang, P., Le, Q. V., Ma, T., and Yu, A. W. DoReMi : Optimizing data mixtures speeds up language model pretraining. In Advances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  53. [61]

    N., and Mirzasoleiman, B

    Yang, Y., Mishra, S., Chiang, J. N., and Mirzasoleiman, B. SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Models . In Advances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  54. [62]

    Data mixing laws: Optimizing data mixtures by predicting language modeling performance

    Ye, J., Liu, P., Sun, T., Zhou, Y., Zhan, J., and Qiu, X. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. arXiv preprint arXiv:2403.16952, 2024

    2024 arXiv

  55. [63]

    H ella S wag: Can a machine really finish your sentence? In Annual Meeting of the Association for Computational Linguistics (ACL), pp.\ 4791--4800, 2019

    Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. H ella S wag: Can a machine really finish your sentence? In Annual Meeting of the Association for Computational Linguistics (ACL), pp.\ 4791--4800, 2019

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.