Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Combatting Dimensional Collapse in LLM Pre-Training Data via Diversified File Selection

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Selecting pre-training files to maximize embedding decorrelation, not domain similarity, lets 1.5% of SlimPajama (about 9B tokens) beat full-data pre-training within a 50B-token budget on nine tasks.

desk verdict Solid idea, honestly reported; the headline 5x data-efficiency number is not yet established because the 1.5% budget was chosen post hoc and the main runs have no seed variance. read the letter →

arxiv 2504.20644 v1 pith:6JV3NVMQ submitted 2025-04-29 cs.LG

Ziqing Fan , Siyuan Du , Shengchao Hu , Pingjie Wang , Li Shen , Ya Zhang , Dacheng Tao , Yanfeng Wang This is my paper · ORCID
classification cs.LG
keywords dataselectionpre-trainingcurationdimensionalcollapsefeaturediversityFrobeniusnormdecorrelationweaklysubmodularoptimizationefficiencylargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the right criterion for choosing LLM pre-training data is not similarity to a 'high-quality' domain but diversity: pick text files whose embeddings are decorrelated from one another. Domain-targeted selectors such as DSIR and QuRating, the paper argues, suffer a diversity dilemma — dimensional collapse in feature space that helps domain-like tasks while hurting general performance. DiSF selects files by minimizing the Frobenius norm of the feature covariance matrix, which by the paper's Lemma 1 is the same as flattening the eigenvalue spectrum toward a uniform distribution. Trained on just 1.5% of SlimPajama's roughly 590M files (about 9B tokens), TinyLlama models from 120M to 1.1B parameters outperform full-data pre-training within a 50B-token budget across nine evaluation tasks, gaining about 1.5x training efficiency and 5x data efficiency. A sympathetic reader should care because this suggests redundancy, not source quality, is the dominant correctable cost in pre-training data.

What carries the argument

The carrying mechanism is the covariance-Frobenius proxy $F^{\text{DiSF}}_M(U) = e^{-\|C(U,M)\|_F}$, computed from Contriever embeddings that are standardized per dimension. Lemma 1 is the hinge: because standardized features have trace $d$, the eigenvalue spread equals $\sum_i(\lambda_i - \bar{\lambda})^2 = \|C(U,M)\|_F^2 - d$, so minimizing the Frobenius norm is exactly flattening the eigenvalues — decorrelation and dimensional diversity become one objective. A classical greedy algorithm maximizes the proxy at batch scale (the paper uses a selection scale of $b = 1024$), and the analysis bounds the weak-submodularity ratio below by $e^{-2\mu}\frac{e^{2\mu-\epsilon}-1}{e^{2\mu}-1}$, which yields the greedy approximation guarantee. The same machinery diagnoses the baselines: the dominance score $\sum_{i=1}^k \lambda_i / \sum_j \lambda_j$ quantifies how collapsed each method's selected feature distribution is, and DiSF attains the most uniform spectrum of all methods compared.

What would settle it

Run DiSF's selection pipeline after destroying the geometry the criterion relies on: apply a random fixed rotation or projection to the Contriever embeddings before computing the covariance, so that decorrelation in the scrambled space no longer corresponds to any real semantic diversity. If the scrambled-feature selection still matches or beats full-data pre-training on the nine tasks, the decorrelation criterion itself is not carrying the gains; if performance collapses toward random-selection levels, the frozen feature space is confirmed as load-bearing. A complementary check is to compare DiSF with a deliberate non-uniform-spectrum diversity objective (for example, greedy log-determinant selection, the variant the paper calls DiSF-LD) at identical budgets: if that variant matches DiSF, generic diversity rather than eigenvalue uniformity is the active mechanism.

Watch

Extended reading notes

Core claim

The paper's central discovery is a diagnosis paired with a cure: file selection driven by similarity to a target domain (heuristic classification, DSIR, QuRating) concentrates the embeddings of chosen files in a thin, lower-dimensional slice of feature space, and this dimensional collapse is what inflates domain-like task scores while dragging down general performance. The proposed remedy, DiSF, treats a set of files as good when its feature covariance matrix $C(U,M)$ has minimal Frobenius norm $\|C(U,M)\|_F$, because for standardized features $\sum_i (\lambda_i - 1)^2 = \|C(U,M)\|_F^2 - d$: uniform eigenvalues and a small Frobenius norm are the same target. Maximizing the non-negative proxy $F^{\text{DiSF}}_M(U) = e^{-\|C(U,M)\|_F}$ with a batch greedy algorithm yields, the paper proves via $\gamma$-weak submodularity, a $(1-e^{-\gamma})$-approximation to the optimal selection; empirically, the chosen 1.5% slice outperforms full-data pre-training at every model size tested. Stated as the authors would state it: decorrelating selected files in a frozen, off-the-shelf feature space is a sufficient proxy for the diversity that improves general LLM performance under a fixed token budget.

Load-bearing premise

The entire method rests on assuming that 'diverse' text, as measured by how evenly its Contriever embeddings spread across feature dimensions, is the same thing as text that improves LLM pre-training; the paper replaces the true objective of final task performance with this proxy without proving they coincide, and its formal guarantee for greedy selection depends on constants $\epsilon$ and $\mu$ (Assumptions 1 and 2 in Appendix A.5.3) that are left uncomputed.

Editorial extensions

If this is right

  • DiSF reaches or exceeds full-data pre-training performance using 1.5% of SlimPajama's files (about 9B tokens) inside a 50B-token budget, yielding about 5x data efficiency and about 1.5x training efficiency; the appendix quantifies the token share needed as 0.15 to 0.20 of the full-data corpus across model sizes.
  • The diversity dilemma is demonstrably avoidable: unlike DSIR, QuRating-W, and heuristic filtering, DiSF does not trade reading-comprehension gains for losses on physical-world tasks, and its average advantage over DSIR grows with training budget (from 2.1% to 3.4% on the 1.1B model) and with model scale (from 2.8% to 3.4% from 120M to 1.1B parameters).
  • The selection transfers across architectures (TinyLlama, Pythia, OPT), to a larger 3B model, and to a code-augmented corpus (SlimPajama plus StarcoderData), where domain-targeted selection visibly degrades code ability.
  • Selection ratio is a tunable dial with a peak near 3% of files (about 20B tokens) for TinyLlama 120M at a 50B budget; beyond that, performance declines toward full-data levels, so more selected data is not monotonically better.
  • No trained proxy or target domain is needed: a frozen off-the-shelf encoder (Contriever) suffices, and the selection cost is $O(|S|^2 b d^2)$, independent of total corpus size.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the criterion is defined on any embedding space, a natural untested extension is an adaptive feature extractor — for instance, re-selecting files from the model's own hidden states during training — which would tighten the loop between the diversity proxy and the representation actually being learned.
  • If spectrum-flatness is the right lens for diversity, scaling-law practice should treat redundancy as a first-class variable: the marginal value of an extra pre-training token should shrink as the selected corpus's covariance spectrum approaches uniformity, making the optimal corpus size a function of the compute budget rather than only of the data source.
  • The paper's own peak at a 3% selection ratio hints at a coupling the authors leave implicit: the optimal selection ratio and the token budget are linked, since a larger budget can absorb more decorrelated files before marginal returns vanish; mapping DiSF's optimum at other budgets (for instance 100B tokens) would chart this trade-off.
  • A direct extension to domain-mixture weighting suggests itself: apply the same covariance-Frobenius objective at the domain level to set mixture proportions, which would connect DiSF to mixture-optimization methods while keeping its freedom from trained proxies.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper proposes DiSF, a file-selection method for LLM pretraining that greedily chooses text files to minimize the Frobenius norm of the feature covariance matrix in a frozen Contriever embedding space, framed as γ-weakly submodular maximization. Experiments on TinyLlama (120M/560M/1.1B) with 10B and 50B token budgets on SlimPajama claim consistent average improvements over baseline selectors on nine Harness tasks, and that selecting 1.5% of files (about 9B tokens) outperforms full-data pretraining under a 50B budget, yielding approximately 1.5x training efficiency and 5x data efficiency.

Significance. If established, the efficiency gains would make DiSF a practical and inexpensive addition to LLM data curation: it uses an off-the-shelf feature extractor, its selection cost is moderate, and the code is provided. The paper also offers a benchmark across model sizes and ablations over architectures and feature extractors. However, the central efficiency claim is not yet secured, because the budget was chosen post hoc on the evaluation tasks and the main comparisons are single runs; the theoretical analysis in the appendix is also incomplete. These issues are fixable and do not invalidate the core idea, but they must be addressed before the paper's headline claims can be accepted.

major comments (4)
  1. [Section 4.1, Section 4.3, Figure 8] Section 4.1 states: "we choose to report performance with a 1.5% selection budget, since it achieves comparable performance compared to Full Data pre-training under 50B pre-training budget on TinyLlama 1.1B." This openly acknowledges that the 1.5% budget was selected after inspecting the evaluation results. Because Figure 8 shows that the budget-performance curve is non-monotonic (the 120M model peaks at 3% and declines), the "5x data efficiency" and "1.5x training efficiency" claims in Section 4.3 and the Abstract are not properties of a fixed selection protocol; they are the result of an oracle choice on the evaluation tasks. Please report results for a pre-registered budget or for a range of budgets with error bars, and derive the efficiency claims from that protocol.
  2. [Table 2, Figures 5-6] All main results in Table 2 and Figures 5-6 are single training runs without seed variance. The headline comparison to Full Data relies on differences of roughly 0.5-1 point on the 7-task average for TinyLlama 1.1B with 50B tokens (e.g., DiSF 45.2 vs. QuRating-A 44.8 and Doremi 44.5 in Table 2). These margins are comparable to typical seed-to-seed variation in small-model pretraining, and the confidence intervals in Table 3 are evaluation-level only. Please provide seed-averaged results for the main table and the efficiency curves, or otherwise justify why single runs are sufficient for the claimed superiority.
  3. [Section 3.2, Appendix A.5.3] The derivations leading to the claimed weak-submodularity lower bound appear to have a direction error. Assumption 2 gives e^{-Δ(e|B)} ≤ e^{2μ}; since the function (e^{-Δ-ε}-1)/(e^{-Δ}-1) is increasing in e^{-Δ} for e^{-Δ}<1, the substitution yields an upper bound rather than the claimed lower bound e^{-2μ}(e^{2μ-ε}-1)/(e^{2μ}-1). Moreover, the constants ε and μ are never estimated, and the result is conditioned on "positive gain" without a precise characterization. Please correct the derivation or explicitly state that the theoretical guarantee rests on unvalidated assumptions; if the bound cannot be repaired, consider removing the approximation guarantee from the main text.
  4. [Section 3.1, Eq. (5)] The selection objective replaces the true objective in Eq. (1) with the Frobenius norm of the covariance matrix in a fixed Contriever feature space, but the paper does not establish that this proxy is aligned with downstream performance beyond the qualitative correlation in Figure 4. This is a load-bearing assumption for the method's validity. Please add a limitation paragraph discussing this gap, or provide an analysis (e.g., rank correlation between proxy values and held-out task performance across several feature extractors) to justify the proxy.
minor comments (6)
  1. [Sections 2.2, 3.1, 4.1, and References] The citation "Renduchintala et al., 1991" is incorrect; INGENIOUS was published at EMNLP 2023, and the reference entry should be updated accordingly.
  2. [Section 3.2] The phrase "a element x" should read "an element x", and the domain of the set function should be written consistently as 2^Ω rather than 2^N when the ground set is denoted Ω.
  3. [Algorithm 1] In the pseudocode, the removal step "bi←bi\{x*}" is placed after the while-condition check, so the initial random sample is removed only in the first loop iteration; please reorder the lines to make the loop invariant explicit.
  4. [Appendix A.5.3] The proof uses the letter e both for the added sample and for Euler's number, which makes the formulas difficult to follow; please use a different symbol (e.g., s) for the added sample.
  5. [Table 2 caption] The caption promises color highlighting ("bold blue, black, and red") that is not visible in the monochrome text; please use textual markers or an accessible color scheme instead.
  6. [Section 4.1] The sentence explaining the choice of 1.5% selection budget should be accompanied by a discussion of the risk of post-hoc selection, or moved to a limitations paragraph, since the same evaluation tasks are used for the headline result.

Circularity Check

1 steps flagged · score 1.0 of 10

No material circularity: DiSF's objective is defined externally and evaluated on independent benchmarks; the only by-construction element is the eigenvalue-uniformity metric, which is the optimized objective itself.

  1. self definitional [Section 3.1 (Eqs. 4-6, Lemma 1) and Figure 3]
    "our goal is to prevent dimensional collapse by ensuring more uniform eigenvalues in the covariance matrix of the selected samples. Directly calculating and selecting based on eigenvalues are costly, but it is feasible to optimize the Frobenius norm of the covariance matrix ∥C∥F ... From Lemma 1, it is evident that ensuring the uniformity of the eigenvalues of the covariance matrix can be translated into minimizing the Frobenius norm of the covariance matrix."

    The selection objective in Eq. 5 is defined as arg max -||C(U,M)||_F, and Lemma 1 is the identity that the eigenvalue variance equals ||C||_F^2 - d. Consequently the reported eigenvalue uniformity / reduced dominance score of DiSF (Figure 3) is not an empirical discovery; it is the objective function itself. The paper optimizes exactly the quantity it then presents as evidence of success, so this part of the validation is true by construction. This does not affect the downstream Harness-task comparisons, which are external to the selection objective.

full rationale

The central derivation chain is self-contained against external evidence. DiSF's proxy space comes from a fixed, off-the-shelf Contriever encoder, and the headline improvements are measured on independent Harness tasks; no downstream benchmark value appears in Eq. 5, so the selection is not fitted to the evaluation it is validated on. The theoretical claims (gamma-weak submodularity, Lemma 1) are mathematical statements, not circular reductions, although the submodularity guarantee depends on uncomputed constants epsilon and mu. The only by-construction element is the eigenvalue-uniformity claim, which is the optimization objective restated; the paper is transparent that this is the goal. Separately, the 1.5% budget and derived 5x data-efficiency claim are selected after observing comparable performance on the same benchmark (Section 4.1), which is a post-hoc reporting concern rather than a circularity in the algorithm; it weakens the headline efficiency number but does not make the method's derivation equivalent to its inputs. Self-citations (e.g., Fan et al. 2024 for dimensional collapse) are not load-bearing because the concept is independently established and cited. Overall score 1: no material circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on an empirical proxy (feature-space covariance) and on two unestimated theoretical constants; no new physical entities are introduced.

free parameters (4)
  • selection budget ratio = 1.5%
    Chosen in Section 4.1 because it achieves comparable performance to full-data pretraining at 50B tokens, so it is selected after inspecting the outcome.
  • selection scale b = 1024
    Chosen via ablation in Figure 7 as the ideal point balancing compute and performance; a tuning decision.
  • epsilon (Assumption 1) = unknown
    Upper bound on gain difference in Assumption 1; never estimated, so the weak-submodularity guarantee is qualitative.
  • mu (Assumption 2) = unknown
    Upper bound on average utility in Assumption 2; never estimated, making the gamma-weak submodularity bound non-quantitative.
assumptions (5)
  • domain assumption Contriever embeddings capture the semantic diversity relevant for LLM pretraining
    Eq. 3 and Eq. 5 replace the training objective with a covariance penalty on these features; no justification that this feature space is aligned with text diversity that improves pretraining.
  • domain assumption Uniform eigenvalues of the feature covariance imply better pretraining data
    Section 2.3 posits that lower dominance score (more uniform eigenvalues) indicates richer information; this is borrowed from self-supervised learning and not proven for LLM pretraining.
  • ad hoc to paper The proxy function FDiSF is monotone and weakly submodular with positive gain
    Section 3.2 and Figure 4 provide empirical monotonicity, and the submodularity ratio lower bound uses Assumptions 1 and 2 which are introduced ad hoc for this proof.
  • domain assumption The Harness tasks are a valid measure of generic performance
    Eq. 1 assumes A can be inferred from nine tasks; the paper uses these as the sole metric.
  • standard math Standard normalization z=(f-mu)/sigma makes trace(C)=d
    Used in Lemma 1 to relate eigenvalue uniformity to Frobenius norm; the identity is algebraically correct.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Combatting Dimensional Collapse in LLM Pre-Training Data via Diversified File Selection." pith.science (2026). https://pith.science/paper/6JV3NVMQ

@misc{pith2026250420644,
  author       = {Pith},
  title        = {Pith review of: Combatting Dimensional Collapse in LLM Pre-Training Data via Diversified File Selection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6JV3NVMQ}},
  note         = {Machine review of arXiv:2504.20644}
}
abstract

Selecting high-quality pre-training data for large language models (LLMs) is crucial for enhancing their overall performance under limited computation budget, improving both training and sample efficiency. Recent advancements in file selection primarily rely on using an existing or trained proxy model to assess the similarity of samples to a target domain, such as high quality sources BookCorpus and Wikipedia. However, upon revisiting these methods, the domain-similarity selection criteria demonstrates a diversity dilemma, i.e.dimensional collapse in the feature space, improving performance on the domain-related tasks but causing severe degradation on generic performance. To prevent collapse and enhance diversity, we propose a DiverSified File selection algorithm (DiSF), which selects the most decorrelated text files in the feature space. We approach this with a classical greedy algorithm to achieve more uniform eigenvalues in the feature covariance matrix of the selected texts, analyzing its approximation to the optimal solution under a formulation of $\gamma$-weakly submodular optimization problem. Empirically, we establish a benchmark and conduct extensive experiments on the TinyLlama architecture with models from 120M to 1.1B parameters. Evaluating across nine tasks from the Harness framework, DiSF demonstrates a significant improvement on overall performance. Specifically, DiSF saves 98.5% of 590M training files in SlimPajama, outperforming the full-data pre-training within a 50B training budget, and achieving about 1.5x training efficiency and 5x data efficiency.

Figures

Figures reproduced from arXiv: 2504.20644 by the authors.

Figure 1
Figure 1. The t-SNE (Van der Maaten & Hinton, 2008) visualization of text features (normalized to the unit sphere) selected by different methods on SlimPajama. We use Contriever (Izacard et al., 2021) to extract features. (a) and (b) show Heuristic classification and DSIR based on the Wikipedia and Book domains, while (c) depicts QuRating based on writing judgments. We visualize top 500 text features selected by their criteri… view at source ↗
Figure 2
Figure 2. Commonsense reasoning abil￾ity of pre-trained TinyLlama 1B us￾ing various selection methods evalu￾ated on seven tasks of Harness. DSIR uses Wikipedia and BookCorpus as high quality source and QuRating-W selects samples with writing style score. All methods select 1.5% of training files in SlimPajama and pre-train 50B tokens. In our paper, we revisit these algorithms by visualizing the feature representations of thei… view at source ↗
Figure 3
Figure 3. The dominance score for recent methods calculated as Pk Pi=1 λi d j=1 λj , where λi represents the i-th largest eigen￾value of the feature covariance matrix, and d is the dimension of the feature space. We use Contriever model to ex￾tract features. We select the top 500 text samples based on their respective selec￾tion criteria. For D4, we select 500 ran￾dom samples after reducing redundancy. As shown in [PITH_FULL… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Proxy value, as defined in equation 6, calculated on different meth￾ods. For each method, we randomly choose samples from their selected text files with 1.5% selection budget. All cases generally demonstrate the property of monotonicity. Moreover, our selec￾tion method…
Figure 5
Figure 5. Figure 5: Average performance of TinyLlama pre-trained after 50B tokens on files selected by our [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Performance of TinyLlama pre-trained under our DiSF with 1.5% selection budget compared [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Performance of pre-trained TinyL￾lama under different selection scales (log2 b). With 1.5% selection budget and 50B training budget, we identify an ideal point with ac￾ceptable computational cost and near-optimal performance, marked by a red star. In this section, we t…
Figure 8
Figure 8. Figure 8: Average commonsense reasoning performance of TinyLlama 120M pre-trained under DiSF with different selection budget and 50B training budget. Blue and grey lines respectively denote Full Data pre-training and random selection with 1.5% selection budget. Selection scale. …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. IDEAL: Data Equilibrium Adaptation for Multi-Capability Language Model Alignment

    cs.AI 2025-05 reject novelty 5.0 of 10

    IDEAL tunes SFT data mixture proportions per domain with influence-function gradients, claiming about 7% average benchmark improvement over uniform mixing.

Reference graph

Works this paper leans on

77 extracted references · 30 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Semdedup: Data-efficient learning at web-scale through semantic deduplication

    Amro Kamal Mohamed Abbas, Kushal Tirumala, Daniel Simig, Surya Ganguli, and Ari S Morcos. Semdedup: Data-efficient learning at web-scale through semantic deduplication. In ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024

    work page 2023

  3. [3]

    Diverse client selection for federated learning via submodular maximization

    Ravikumar Balakrishnan, Tian Li, Tianyi Zhou, Nageen Himayat, Virginia Smith, and Jeff Bilmes. Diverse client selection for federated learning via submodular maximization. In International Conference on Learning Representations, 2022

    work page 2022

  4. [4]

    Vicreg: Variance-invariance-covariance regularization for self-supervised learning

    Adrien Bardes, Jean Ponce, and Yann Lecun. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. In ICLR 2022-International Conference on Learning Representations, 2022

    work page 2022

  5. [5]

    Pythia: A suite for analyzing large language models across training and scaling

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp.\ 2397--2430. PMLR, 2023

    2023

  6. [6]

    Piqa: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp.\ 7432--7439, 2020

    2020

  7. [8]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020 b

    arXiv 2005

  8. [9]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp.\ 1597--1607. PMLR, 2020

    2020

Show all 77 references
  1. [10]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24 0 (240): 0 1--113, 2023 a

    2023

  2. [11]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24 0 (240): 0 1--113, 2023 b

    2023

  3. [12]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019

    1905 arXiv

  4. [13]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1, 2018

    2018 arXiv

  5. [14]

    Redpajama: an open dataset for training large language models, 2023

    Together Computer. Redpajama: an open dataset for training large language models, 2023. URL https://github.com/togethercomputer/RedPajama-Data

    2023

  6. [15]

    Flashattention: Fast and memory-efficient exact attention with io-awareness

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher R \'e . Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35: 0 16344--16359, 2022

    2022

  7. [16]

    Submodular meets spectral: Greedy algorithms for subset selection, sparse approximation and dictionary selection

    A DAS. Submodular meets spectral: Greedy algorithms for subset selection, sparse approximation and dictionary selection. In Proc. 28th Int. Conf. on Machine Learning (ICML'11), pp.\ 1057--1064, 2011

    2011

  8. [17]

    Federated learning under partially class-disjoint data via manifold reshaping

    Ziqing Fan, Jiangchao Yao, Ruipeng Zhang, Lingjuan Lyu, Ya Zhang, and Yanfeng Wang. Federated learning under partially class-disjoint data via manifold reshaping. arXiv preprint arXiv:2405.18983, 2024

    2024 arXiv

  9. [18]

    Submodular functions and optimization

    Satoru Fujishige. Submodular functions and optimization. Elsevier, 2005

    2005

  10. [19]

    A framework for few-shot language model evaluation, 07 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024

  11. [20]

    Openllama: An open reproduction of llama, May 2023

    Xinyang Geng and Hao Liu. Openllama: An open reproduction of llama, May 2023. URL https://github.com/openlm-research/open_llama

    2023

  12. [21]

    Online submodular set cover, ranking, and repeated active learning

    Andrew Guillory and Jeff A Bilmes. Online submodular set cover, ranking, and repeated active learning. Advances in neural information processing systems, 24, 2011

    2011

  13. [22]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021

    2021

  14. [23]

    Training compute-optimal large language models

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022

    2022 arXiv

  15. [24]

    Diversified batch selection for training acceleration

    Feng Hong, Yueming Lyu, Jiangchao Yao, Ya Zhang, Ivor Tsang, and Yanfeng Wang. Diversified batch selection for training acceleration. In Forty-first International Conference on Machine Learning, 2024

    2024

  16. [25]

    Unsupervised dense information retrieval with contrastive learning, 2021

    Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning, 2021. URL https://arxiv.org/abs/2112.09118

    2021 arXiv

  17. [26]

    Efficient data subset selection to generalize training across models: transductive and inductive networks

    Eeshaan Jain, Tushar Nandy, Gaurav Aggarwal, Ashish Tendulkar, Rishabh Iyer, and Abir De. Efficient data subset selection to generalize training across models: transductive and inductive networks. Advances in Neural Information Processing Systems, 36, 2024

    2024

  18. [27]

    Fine-tuning with reserved majority for noise reduction

    Shuyang Jiang, Yusheng Liao, Yanfeng Wang, Ya Zhang, and Yu Wang. Fine-tuning with reserved majority for noise reduction. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=ZV7CLf0RHK

    2025

  19. [28]

    Understanding dimensional collapse in contrastive self-supervised learning

    Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In International Conference on Learning Representations, 2022

    2022

  20. [29]

    Orient: Submodular mutual information measures for data subset selection under distribution shift

    Athresh Karanam, Krishnateja Killamsetty, Harsha Kokel, and Rishabh Iyer. Orient: Submodular mutual information measures for data subset selection under distribution shift. Advances in neural information processing systems, 35: 0 31796--31808, 2022

    2022

  21. [30]

    Learning from less data: A unified data subset selection and active learning framework for computer vision

    Vishal Kaushal, Rishabh Iyer, Suraj Kothawade, Rohan Mahadev, Khoshrav Doctor, and Ganesh Ramakrishnan. Learning from less data: A unified data subset selection and active learning framework for computer vision. In 2019 IEEE Winter Conference on Applications of Computer Vision...

    2019

  22. [31]

    Prism: A rich class of parameterized submodular information measures for guided data subset selection

    Suraj Kothawade, Vishal Kaushal, Ganesh Ramakrishnan, Jeff Bilmes, and Rishabh Iyer. Prism: A rich class of parameterized submodular information measures for guided data subset selection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.\ 10238--...

    2022

  23. [32]

    Submodular function maximization

    Andreas Krause and Daniel Golovin. Submodular function maximization. Tractability, 3 0 (71-104): 0 3, 2014

    2014

  24. [33]

    An end-to-end submodular framework for data-efficient in-context learning

    Lilly Kumari, Shengjie Wang, Arnav Das, Tianyi Zhou, and Jeff Bilmes. An end-to-end submodular framework for data-efficient in-context learning. In Findings of the Association for Computational Linguistics: NAACL 2024, pp.\ 3293--3308, 2024

    2024

  25. [34]

    Disentangling hate in online memes

    Roy Ka-Wei Lee, Rui Cao, Ziqing Fan, Jing Jiang, and Wen-Haw Chong. Disentangling hate in online memes. In Proceedings of the 29th ACM international conference on multimedia, pp.\ 5138--5147, 2021

    2021

  26. [35]

    Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161, 2023

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161, 2023

    2023 arXiv

  27. [36]

    LightningAI. Litgpt. https://github.com/Lightning-AI/litgpt, 2023

    2023

  28. [37]

    Optimal selection of limited vocabulary speech corpora

    Hui Lin and Jeff A Bilmes. Optimal selection of limited vocabulary speech corpora. In INTERSPEECH, pp.\ 1489--1492, 2011

    2011

  29. [38]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019

    2019

  30. [39]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, 2018

    2018

  31. [40]

    An analysis of approximations for maximizing submodular set functions—i

    George L Nemhauser, Laurence A Wolsey, and Marshall L Fisher. An analysis of approximations for maximizing submodular set functions—i. Mathematical programming, 14: 0 265--294, 1978

    1978

  32. [41]

    Self-alignment of large language models via monopolylogue-based social scene simulation

    Xianghe Pang, Shuo Tang, Rui Ye, Yuxin Xiong, Bolun Zhang, Yanfeng Wang, and Siheng Chen. Self-alignment of large language models via monopolylogue-based social scene simulation. arXiv preprint arXiv:2402.05699, 2024

    2024 arXiv

  33. [42]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019

    2019

  34. [43]

    Carbon emissions and large neural network training

    David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint arXiv:2104.10350, 2021

    2021 arXiv

  35. [44]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  36. [45]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp...

    2021

  37. [46]

    Scaling language models: Methods, analysis & insights from training gopher

    Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021

    2021 arXiv

  38. [47]

    Ingenious: Using informative data subsets for efficient pre-training of language models

    HSVNS Kowndinya Renduchintala, Krishnateja Killamsetty, Sumit Bhatia, Milan Aggarwal, Ganesh Ramakrishnan, Rishabh K Iyer, and Balaji Krishnamurthy. Ingenious: Using informative data subsets for efficient pre-training of language models. In The 2023 Conference on Empirical Met...

    2023

  39. [48]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64 0 (9): 0 99--106, 2021

    2021

  40. [49]

    Discrete location theory

    Said Salhi. Discrete location theory. Journal of the Operational Research Society, 42 0 (12): 0 1124--1125, 1991 a

    1991

  41. [50]

    Discrete location theory

    Said Salhi. Discrete location theory. Journal of the Operational Research Society, 42 0 (12): 0 1124--1125, 1991 b

    1991

  42. [51]

    Weakly submodular function maximization using local submodularity ratio

    Richard Santiago and Yuichi Yoshida. Weakly submodular function maximization using local submodularity ratio. arXiv preprint arXiv:2004.14650, 2020

    2004 arXiv

  43. [52]

    Mimicking the oracle: An initial phase decorrelation approach for class incremental learning

    Yujun Shi, Kuangqi Zhou, Jian Liang, Zihang Jiang, Jiashi Feng, Philip HS Torr, Song Bai, and Vincent YF Tan. Mimicking the oracle: An initial phase decorrelation approach for class incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern ...

    2022

  44. [53]

    Beyond neural scaling laws: beating power law scaling via data pruning

    Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35: 0 19523--19536, 2022

    2022

  45. [54]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri \`a Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprin...

    2022 arXiv

  46. [55]

    Challenging big-bench tasks and whether chain-of-thought can solve them

    Mirac Suzgun, Nathan Scales, Nathanael Sch \"a rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022

    arXiv

  47. [56]

    Synthesizing post-training data for llms through multi-agent simulation

    Shuo Tang, Xianghe Pang, Zexi Liu, Bohan Tang, Rui Ye, Xiaowen Dong, Yanfeng Wang, and Siheng Chen. Synthesizing post-training data for llms through multi-agent simulation. arXiv preprint arXiv:2410.14251, 2024

    2024 arXiv

  48. [57]

    Lamda: Language models for dialog applications

    Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239, 2022

    2022 arXiv

  49. [58]

    D4: Improving llm pretraining via document de-duplication and diversification

    Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari Morcos. D4: Improving llm pretraining via document de-duplication and diversification. Advances in Neural Information Processing Systems, 36, 2023

    2023

  50. [60]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023 b

    2023 arXiv

  51. [61]

    Visualizing data using t-sne

    Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9 0 (11), 2008

    2008

  52. [62]

    Reconstruct the pruned model without any retraining

    Pingjie Wang, Ziqing Fan, Shengchao Hu, Zhe Chen, Yanfeng Wang, and Yu Wang. Reconstruct the pruned model without any retraining. arXiv preprint arXiv:2407.13331, 2024

    2024 arXiv

  53. [63]

    Submodularity in data subset selection and active learning

    Kai Wei, Rishabh Iyer, and Jeff Bilmes. Submodularity in data subset selection and active learning. In International conference on machine learning, pp.\ 1954--1963. PMLR, 2015

    1954

  54. [64]

    Qurating: Selecting high-quality data for training language models

    Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. Qurating: Selecting high-quality data for training language models. In Forty-first International Conference on Machine Learning, 2024

    2024

  55. [65]

    Doremi: Optimizing data mixtures speeds up language model pretraining

    Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36, 2023 a

    2023

  56. [66]

    Data selection for language models via importance resampling

    Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36: 0 34201--34227, 2023 b

    2023

  57. [67]

    Are we there yet? revealing the risks of utilizing large language models in scholarly peer review

    Rui Ye, Xianghe Pang, Jingyi Chai, Jiaao Chen, Zhenfei Yin, Zhen Xiang, Xiaowen Dong, Jing Shao, and Siheng Chen. Are we there yet? revealing the risks of utilizing large language models in scholarly peer review. arXiv preprint arXiv:2412.01708, 2024

    2024 arXiv

  58. [68]

    On the vulnerability of safety alignment in open-access llms

    Jingwei Yi, Rui Ye, Qisi Chen, Bin Zhu, Siheng Chen, Defu Lian, Guangzhong Sun, Xing Xie, and Fangzhao Wu. On the vulnerability of safety alignment in open-access llms. In Findings of the Association for Computational Linguistics ACL 2024, pp.\ 9236--9260, 2024

    2024

  59. [69]

    Barlow twins: Self-supervised learning via redundancy reduction

    Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and St \'e phane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International conference on machine learning, pp.\ 12310--12320. PMLR, 2021

    2021

  60. [70]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  61. [71]

    Tinyllama: An open-source small language model

    Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385, 2024 a

    2024 arXiv

  62. [72]

    Communication-efficient decentralized online continuous dr-submodular maximization

    Qixin Zhang, Zengde Deng, Xiangru Jian, Zaiyi Chen, Haoyuan Hu, and Yu Yang. Communication-efficient decentralized online continuous dr-submodular maximization. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pp.\ 3330--3339, 2023

    2023

  63. [73]

    Boosting gradient ascent for continuous dr-submodular maximization

    Qixin Zhang, Zongqi Wan, Zengde Deng, Zaiyi Chen, Xiaoming Sun, Jialin Zhang, and Yu Yang. Boosting gradient ascent for continuous dr-submodular maximization. arXiv preprint arXiv:2401.08330, 2024 b

    2024 arXiv

  64. [74]

    Near-optimal online learning for multi-agent submodular coordination: Tight approximation and communication efficiency

    Qixin Zhang, Zongqi Wan, Yu Yang, Li Shen, and Dacheng Tao. Near-optimal online learning for multi-agent submodular coordination: Tight approximation and communication efficiency. arXiv preprint arXiv:2502.05028, 2025

    2025 arXiv

  65. [75]

    Opt: Open pre-trained transformer language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022

    2022 arXiv

  66. [76]

    Minimax curriculum learning: Machine teaching with desirable difficulties and scheduled diversity

    Tianyi Zhou and Jeff Bilmes. Minimax curriculum learning: Machine teaching with desirable difficulties and scheduled diversity. In International conference on learning representations, 2018

    2018

  67. [77]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  68. [78]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  69. [79]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.