Pith. sign in

REVIEW 3 major objections 5 minor 61 references

ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Selectively training only the least predictable tokens can cut language-model pretraining compute by about six percent, this paper argues.

desk verdict A clean token-selection scheme with an honest wall-clock appendix, but the efficiency claim rests on FLOPs accounting and a missing random-drop control. read the letter →

arxiv 2505.19893 v1 pith:S2HT6JMP submitted 2025-05-26 cs.LG cs.CL

Melis Ilayda Bal , Volkan Cevher , Michael Muehlebach This is my paper
classification cs.LGcs.CL
keywords selectivelanguagemodelingtoken-levelselectionvalue-at-riskconditionalpretrainingefficiencydistributionallyrobustoptimizationknowledgedistillationGPT-2
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ESLM claims that most language-model pretraining compute is wasted on tokens the model already handles well, and that a training loop can retain quality while spending less compute by backpropagating only through the highest-risk tokens in each batch. The paper introduces a self-supervised, online rule: per token, use predictive entropy or loss as a risk score, then keep only tokens above a value-at-risk threshold (the top $1-\alpha$ fraction). On GPT-2 models from 124M to 774M parameters, it reports reaching a target validation perplexity with on average $5.85\%$ fewer training FLOPs than standard causal language modeling, with downstream accuracy at least as good or better under fixed compute budgets. If the claim holds, token-level risk filtering is a cheap, model-agnostic lever for making pretraining more compute-efficient without touching the architecture or the data pipeline.

What carries the argument

The load-bearing object is the per-token risk score $S_\theta(x_j) \in \{\ell_\theta(x_j), H_\theta(x_j)\}$, combined with value-at-risk thresholding: at each step the algorithm computes the empirical quantile $\operatorname{VaR}_\alpha$ over the batch, keeps tokens in the upper tail $\tilde B = \{x_j : S_\theta(x_j) \ge \operatorname{VaR}_\alpha\}$, and minimizes the expected loss restricted to $\tilde B$. When the score is the loss, the restricted expectation is the conditional value-at-risk $\operatorname{CVaR}_\alpha$, which connects the selection rule to distributionally robust optimization and to a two-player game in which a masked adversary chooses worst-case subsets under a threshold constraint. The same machinery powers the adaptive variant, which adjusts $\alpha$ from changes in CVaR, and the distillation variant, which uses the student's own risk scores to decide where teacher supervision is applied.

What would settle it

Run the same GPT-2 pretraining setup with a matched control that randomly drops $1-\alpha$ of the tokens from each batch at both equal steps and equal FLOPs; if the random-drop control matches ESLM's validation-perplexity and downstream curves, then the risk-aware selection rule is not what drives the reported gains.

Watch

Extended reading notes

Core claim

The paper's central claim is that selective token-level training, driven by per-token risk scores, improves both the compute-efficiency and the distributional robustness of LLM pretraining. Concretely, each training step computes a risk score per token—predictive entropy (VaR-entropy variant) or per-token loss (CVaR-loss variant)—and a value-at-risk threshold then discards the low-risk tail of the batch, so gradient computation is performed only over the retained high-risk subset. In the loss-based variant this shaped objective is exactly conditional value-at-risk minimization of the token loss, and the paper develops a bilevel-game and distributionally robust optimization reading of the procedure: the model improves against a masking adversary that forces training onto the hardest available tokens. Empirically, ESLM reports average FLOPs savings of about $5.85\%$ to reach target validation perplexity, improved average downstream accuracy under equal compute, and consistent gains across model sizes, corpora, and an adaptive confidence variant.

Load-bearing premise

The whole method rests on the premise that a token's current loss or entropy tells you how much it is worth training on, and the paper does not include a random token-dropping control at the same retention rate, so the benefits could in principle come from training on fewer tokens rather than from selecting the risky ones.

Editorial extensions

If this is right

  • With $\alpha = 0.1$ (keeping the top 90 percent of tokens per batch), ESLM needs on average 5.85 percent fewer training FLOPs than CLM to reach the same target validation perplexity across 124M, 350M, and 774M GPT-2 models.
  • Under a fixed compute budget of roughly $3 \times 10^{17}$ FLOPs, ESLM-CVaR-loss and ESLM-VaR-entropy achieve higher average downstream accuracy than CLM, Rho-1, and GREATS on the 124M SlimPajama-6B-Unif setting.
  • Because low-risk tokens are skipped during backpropagation, ESLM can be run with larger batches under the same FLOPs budget, and batch-scaled ESLM also improves average downstream performance.
  • ADA-ESLM, which adapts the confidence $\alpha$ through a normalized CVaR-based controller, reaches the target perplexity with fewer FLOPs than fixed-$\alpha$ and baseline methods while acting as a token-level curriculum.
  • ESLM-KD applies the same VaR selection to knowledge distillation, so a 774M student trained with a 124M teacher converges to target perplexity with substantially fewer FLOPs than dense distillation or SALT.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested control in the paper is whether the same FLOPs savings and perplexity are obtained by randomly dropping an equal fraction of tokens per batch; without that control, part of the reported gain could be attributable to training on fewer tokens rather than to risk-aware selection.
  • The paper's reported wall-clock overhead (roughly 13.5 hours versus 9.3 hours for CLM at 124M) comes from sparse backpropagation on dense hardware, so on sparsity-aware accelerators or with specialized kernels the FLOPs savings could plausibly convert into genuine wall-clock speedups.
  • Since entropy-based selection tends to retain syntactic and punctuation tokens in ambiguous contexts while loss-based selection favors rare semantic tokens, the two variants may produce measurably different robustness and calibration profiles on out-of-distribution or multilingual data, a distinction the paper only describes qualitatively.
  • The CVaR and distributionally robust optimization connection suggests a direct testable extension: measuring ESLM-pretrained models on distribution-shifted benchmarks could determine whether the claimed distributional robustness materializes beyond the nine reported tasks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ESLM, a token-level selection method for autoregressive pretraining that computes per-token entropy or loss during the forward pass and retains only the top-(1-alpha) fraction of tokens by a value-at-risk threshold, with an adaptive variant (ADA-ESLM) and a distillation variant (ESLM-KD). The central empirical claim, stated in the abstract and Section 5.1, is that ESLM reduces training FLOPs by about 5.85% on average to reach target validation perplexity while maintaining or improving validation loss and downstream accuracy across GPT-2 sizes (124M-774M) and several corpora. The paper also gives a risk-theoretic framing via CVaR and a distributionally robust optimization interpretation.

Significance. If the FLOPs-reduction claim holds, ESLM is a useful, simple, self-supervised token-selection method: it requires no external reference model, no offline scoring, and no gradient-based influence estimates, and it integrates naturally with knowledge distillation. The experimental coverage is a strength: three model sizes, three data mixtures, an adaptive variant, and a distillation study are reported, with hyperparameters and implementation details in the appendix. The paper is also honest about practical limitations, explicitly acknowledging in Appendix D.3 that ESLM is slower in wall-clock time than standard CLM on current hardware. However, the currently missing random-token-dropping control means the paper has not yet established that the risk-aware selection rule, rather than mere token subsampling, is what drives the reported gains; this is the main load-bearing gap. The downstream improvements are small and often within standard error, so the strength of the generalization claims needs to be tempered or statistically supported.

major comments (3)
  1. [Section 5, Algorithm 1] The central efficiency claim is not yet isolated from a trivial subsampling effect. With the default alpha = 0.1, ESLM backpropagates through only 90% of the tokens per batch, so any method that drops 10% of tokens would incur a similar per-step theoretical FLOPs reduction. The paper compares against CLM, Rho-1, and GREATS, but never against a random-token-drop baseline at the same retention rate. Without such a control, the reported ~5.85% average FLOPs reduction and the validation-loss improvements could in principle be explained by training on 90% of the tokens rather than by the VaR/CVaR selection rule. I ask the authors to add a random-drop baseline with the same 10% token mask, matching all other settings, and report validation loss, downstream accuracy, and FLOPs-to-target for it.
  2. [Appendix D.3, Table 7] The paper's efficiency claim is stated in terms of theoretical FLOPs, but the reported wall-clock results contradict the abstract's broader phrasing. For 124M models under the same ~3E17 FLOPs budget, ESLM takes 13.50-13.53 hours versus 9.32 hours for CLM, i.e., roughly 45% slower. The manuscript explains this as a mismatch between sparse backpropagation and dense hardware, which is a reasonable and honest caveat, but the abstract and Section 1 should not claim improved 'training efficiency' without making the FLOPs-versus-wall-clock distinction explicit. I recommend either reporting throughput results on the target hardware or restricting the headline efficiency claim to theoretical FLOPs.
  3. [Section 5.1, Table 1; Appendix C, Table 3] The downstream generalization claims are stronger than the evidence. In Table 1, the average accuracy difference between ESLM-CVaR-loss (0.39115) and CLM (0.38434) is about 0.007, while per-task standard errors are 0.005-0.05 and at least one task (LAMBADA) favors CLM; individual differences are therefore not statistically resolvable. The distillation results in Table 3 are even smaller: ESLM-KD averages 0.4301/0.4307 versus 0.4299 for Dense-KD and 0.4304 for SALT. The text in Section 5.1 and Appendix C says ESLM 'significantly outperforms' baselines; this should be replaced by a more measured statement, or supported with multiple seeds, paired tests, or confidence intervals on the averages.
minor comments (5)
  1. [Throughout] There are several formatting and typographical issues, including 'Corrrespondance' in the footnote, inconsistent spacing in 'ADA-E SLM' vs 'ADA-ESLM', and artifacts such as 'E SLM' and 'Crim e' in appendix figures; these should be cleaned up.
  2. [Section 3] The statement that ESLM 'recovers conditional value-at-risk loss minimization' is a direct consequence of thresholding the empirical loss distribution, not a derived equivalence; consider labeling it as an observation rather than a theoretical contribution.
  3. [Section 3.1, Algorithm 2] The adaptive alpha update rule uses a multiplicative exponential update but does not specify a projection or clipping to keep alpha in (0,1); the paper should state the bounds and how they are enforced, especially since Figure 6 shows alpha converging to a narrow range.
  4. [Section 5.1, Figure 3] The paper reports an 'average FLOPs reduction of 5.85%' but does not specify how the average is computed across model sizes and whether it refers only to Figure 3 (SlimPajama-6B-Unif) or to all datasets; please clarify the aggregation procedure.
  5. [Appendix F] The token-selection qualitative examples are illustrative, but it would be helpful to quantify the claim that the selection rule targets rare or informative tokens, for example by reporting selection rates as a function of token frequency or part-of-speech.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical claims are tested against external baselines, and the CVaR connection is a mathematical equivalence rather than a fitted prediction.

full rationale

The paper's central empirical claims are evaluated against external baselines (CLM, Rho-1, GREATS) on held-out validation perplexity and downstream benchmarks, with no constants fitted to the target FLOPs-reduction result. The identification that loss-based VaR thresholding recovers the CVaR objective is a mathematical equivalence stated via Eqs. (2)-(3); it re-frames the selection rule without importing the experimental conclusion. ESLM's confidence level, adaptation rate, distillation weight, and temperature are manually specified hyperparameters (Appendix D.1 and Appendix C), not fitted to the reported target perplexities. The lack of a random-token-dropping control and the wall-clock overhead noted in Appendix D.3 are experimental limitations and correctness risks, not circular steps. No load-bearing self-citation or imported uniqueness theorem is present. Accordingly, no circular step can be exhibited with quoted text.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends primarily on the proxy assumption that high-loss and high-entropy tokens carry more learning signal, together with the FLOPs-as-efficiency accounting. The risk-measure theory is standard CVaR vocabulary, and the method's hyperparameters are tuned rather than derived. No new physical or model entities are postulated; the masking adversary is a conceptual game framing, not a mechanism.

free parameters (4)
  • confidence level alpha = 0.1 default; 0.2 for batch scaling; ablation over [0.05, 0.5]
    Controls the fraction of tokens retained per batch and directly determines the compute-quality trade-off; chosen by hand after the ablation in Figure 5d.
  • adaptation rate gamma = 0.5
    Used in the ADA-ESLM multiplicative update for alpha; set without a sensitivity analysis.
  • distillation weight lambda = 0.5
    Weight balancing KL divergence to the teacher and standard cross-entropy in ESLM-KD; selected by hyperparameter tuning (Appendix C).
  • teacher temperature rho = 1.0
    Temperature for teacher logits in knowledge distillation; selected by hyperparameter tuning (Appendix C).
assumptions (4)
  • domain assumption Per-token loss or entropy is a valid proxy for the learning value of a token.
    Section 2 states both measures 'serve as proxies for token difficulty and informativeness'; Section 3 builds the selection rule on this. The paper gives qualitative examples but no controlled test such as random token dropping to prove that risk-aware selection, rather than mere subsampling, drives the gains.
  • domain assumption The empirical VaR threshold computed on a single batch approximates the population VaR/CVaR of the training distribution.
    Algorithm 1 computes the threshold from the batch's empirical score distribution (Eq. 3) and treats it as exact; finite-batch approximation error is not analyzed.
  • domain assumption Theoretical FLOPs accounting reflects training efficiency.
    FLOPs estimates follow Chowdhery et al. (2023) and count skipped tokens as saved, but Appendix D.3 Table 7 shows ESLM is slower in wall-clock time than CLM (13.50-13.53h vs 9.32h on 124M).
  • domain assumption GPT-2 scale results transfer to larger models and other corpora.
    Experiments stop at 774M parameters and 6B-9B token corpora; the Limitations section concedes scaling to frontier models is future work.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining." pith.science (2026). https://pith.science/paper/S2HT6JMP

@misc{pith2026250519893,
  author       = {Pith},
  title        = {Pith review of: ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S2HT6JMP}},
  note         = {Machine review of arXiv:2505.19893}
}
read the original abstract

Large language model pretraining is compute-intensive, yet many tokens contribute marginally to learning, resulting in inefficiency. We introduce Efficient Selective Language Modeling (ESLM), a risk-aware algorithm that improves training efficiency and distributional robustness by performing online token-level batch selection. ESLM leverages per-token statistics (e.g., entropy or loss) and applies value-at-risk thresholding to retain only the most informative tokens per batch. This data-centric mechanism reshapes the training loss, prioritizing high-risk tokens and eliminating redundant gradient computation. We frame ESLM as a bilevel game: the model competes with a masking adversary that selects worst-case token subsets under a constrained thresholding rule. In the loss-based setting, ESLM recovers conditional value-at-risk loss minimization, providing a principled connection to distributionally robust optimization. We extend our approach to Ada-ESLM, which adaptively tunes the selection confidence during training. Experiments on GPT-2 pretraining show that ESLM significantly reduces training FLOPs while maintaining or improving both perplexity and downstream performance compared to baselines. Our approach also scales across model sizes, pretraining corpora, and integrates naturally with knowledge distillation.

Figures

Figures reproduced from arXiv: 2505.19893 by the authors.

Figure 1
Figure 1. The illustration of ESLM approach. ESLM computes token-level risk scores and retains only high-risk tokens via a value-at-risk threshold. This reshapes the effective training distribution and loss by focusing computational resources on tokens with higher learning value. entropy (Shannon, 1948) or loss, and retains only the highest-risk tokens using value-at-risk (VaR) thresholding. This dynamic filtering shapes the … view at source ↗
Figure 2
Figure 2. Training FLOPs (↓) required to reach target validation (log) perplexity. We report the training FLOPs required by the methods with model sizes {124M, 350M, 774M} to achieve a target validation loss threshold across datasets. ESLM reduces training cost by focusing optimization on the high-risk tokens, eliminating redundant gradient computation. This efficiency gain holds consistently across model scales. See Appendix… view at source ↗
Figure 3
Figure 3. Validation loss vs training FLOPs. We report convergence of validation loss vs training FLOPs (axes are in log scale for better visibility) of models trained on SlimPajama-6B-Unif mixture. ESLM variants with α = 0.1 consistently reach lower loss with fewer FLOPs, with increased efficiency gains as the model scales. See Appendix E.1 for results on other pretraining corpora. memory requirements, even with ghost inner … view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: 5-shot accuracy (norm) (↑) performance on HellaSwag throughout training. ESLM variants discover higher accuracy levels than baselines, with par￾ticular gains in the later training stages. Downstream performance [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Extended analyses demonstrating use cases of ESLM. (a): ESLM enables batch scaling, improving generalization accuracy (↑) over baselines under the same compute budget. (b): ADA￾ESLM reduces training FLOPs required to reach the target validation (log) perplexity (↓) by …
Figure 6
Figure 6. Figure 6: ADA-ESLM confidence level (α) during training. ADA-ESLM adjusts α dynamically using a CVaR-based controller to stabilize training. The learned α values converge to the [0.1, 0.2] range—previously shown in Section 5.2 (Figure 5d) to balance training efficiency and data …
Figure 7
Figure 7. Figure 7: ADA-ESLM efficiency and generalization performance. (a): ADA-ESLM adaptively tunes the α level based on training dynamics, achieving the target validation (log) perplexity with fewer training FLOPs compared to baselines. (b): ADA-ESLM further improves generalization on…
Figure 8
Figure 8. Figure 8: Validation loss vs training FLOPs on OpenWebText. We report convergence of validation loss vs training FLOPs (axes are in log scale for better visibility) of models trained on OpenWebText. ESLM variants with α = 0.1 reach lower loss with fewer FLOPs, consistently provi…
Figure 9
Figure 9. Figure 9: Validation loss vs training FLOPs on SlimPajama-6B-DoReMi mixture. We report convergence of validation loss vs training FLOPs (axes are in log scale for better visibility) of models trained on SlimPajama-6B-DoReMi mixture. ESLM variants with α = 0.1 reach lower loss wi…
Figure 10
Figure 10. Figure 10: Perplexity and generalization performance of batch-scaled ESLM (124M models). (a): ESLM under larger batch converges faster in compute space to lower validation loss than standard CLM training on the SlimPajama-6B-Unif dataset. (b): Batch-scaled ESLM further achieves …
Figure 11
Figure 11. Figure 11: further illustrates the frequency of top-20 selected tokens by 774M ESLM models, from the validation examples given in Figures 14-15. The results reveal that as we allow for more tokens to be selected (α decreases), ESLM-VaR-entropy selects syntactically ambiguous tok…
Figure 12
Figure 12. Figure 12: Example inputs from SlimPajama-6B-Unif mixture showing the [PITH_FULL_IMAGE:figures/full_fig_p026_12.png]
Figure 13
Figure 13. Figure 13: Example inputs from SlimPajama-6B-Unif mixture showing the [PITH_FULL_IMAGE:figures/full_fig_p027_13.png]
Figure 14
Figure 14. Figure 14: Example inputs from SlimPajama-6B-Unif mixture showing the [PITH_FULL_IMAGE:figures/full_fig_p028_14.png]
Figure 15
Figure 15. Figure 15: Example inputs from SlimPajama-6B-Unif mixture showing the [PITH_FULL_IMAGE:figures/full_fig_p029_15.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 37 canonical work pages

  1. [1]

    M., Longpre, S., Lambert, N., Wang, X., Muennighoff, N., Hou, B., Pan, L., Jeong, H., Raffel, C., Chang, S., Hashimoto, T., and Wang, W

    Albalak, A., Elazar, Y., Xie, S. M., Longpre, S., Lambert, N., Wang, X., Muennighoff, N., Hou, B., Pan, L., Jeong, H., Raffel, C., Chang, S., Hashimoto, T., and Wang, W. Y. (2024). A survey on data selection for language models. ArXiv , 2402.16827

    arXiv 2024

  2. [2]

    Artzner, P., Delbaen, F., Eber, J.-M., and Heath, D. (1999). Coherent measures of risk. Mathematical Finance , 9(3):203--228

    work page 1999

  3. [3]

    Ben-Tal, A., Ghaoui, L., and Nemirovski, A. (2009). Robust Optimization . Princeton Series in Applied Mathematics. Princeton University Press

    work page 2009

  4. [4]

    Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. (2020). PIQA : Reasoning about physical commonsense in natural language. AAAI Conference on Artificial Intelligence , pages 7432--7439

    work page 2020

  5. [5]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems , 33:1877--1901

    work page 2020

  6. [6]

    Buciluǎ, C., Caruana, R., and Niculescu-Mizil, A. (2006). Model compression. International Conference on Knowledge Discovery and Data Mining , pages 535--541

    work page 2006

  7. [7]

    Chaudhary, S., Dinesha, U., Kalathil, D., and Shakkottai, S. (2024). Risk-averse fine-tuning of large language models. Advances in Neural Information Processing Systems , 37:107003--107038

    work page 2024

  8. [8]

    W., Sutton, C., Gehrmann, S., et al

    Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2023). PaLM : Scaling language modeling with pathways. Journal of Machine Learning Research , 24(240):1--113

    work page 2023

Show all 61 references
  1. [9]

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. (2018). Think you have S olved Q uestion A nswering? T ry ARC , the AI2 R easoning C hallenge. ArXiv , 1803.05457

    2018 arXiv

  2. [10]

    Coleman, C., Yeh, C., Mussmann, S., Mirzasoleiman, B., Bailis, P., Liang, P., Leskovec, J., and Zaharia, M. (2020). Selection via proxy: Efficient data selection for deep learning. International Conference on Learning Representations

    2020

  3. [11]

    Y., Jegelka, S., and Krause, A

    Curi, S., Levy, K. Y., Jegelka, S., and Krause, A. (2020). Adaptive sampling for stochastic risk-averse learning. Advances in Neural Information Processing Systems , 33:1036--1047

    2020

  4. [12]

    Dao, T. (2023). Flash A ttention-2: Faster attention with better parallelism and work partitioning. ArXiv , 2307.08691

    2023 arXiv

  5. [13]

    and Namkoong, H

    Duchi, J. and Namkoong, H. (2021). Learning models with uniform performance via distributionally robust optimization. The Annals of Statistics , 49(3):1378--1406

    2021

  6. [14]

    and Jaggi, M

    Fan, S. and Jaggi, M. (2023). Irreducible curriculum for language model pretraining. ArXiv , 2310.15389

    2023 arXiv

  7. [15]

    Fan, S., Pagliardini, M., and Jaggi, M. (2023). DoGE : Domain reweighting with generalization estimation. ArXiv , 2310.15393

    2023 arXiv

  8. [16]

    and Dayan, P

    Gagne, C. and Dayan, P. (2021). Two steps to risk sensitivity. Advances in Neural Information Processing Systems , 34:22209--22220

    2021

  9. [17]

    Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac'h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., a...

    2024

  10. [18]

    and Cohen, V

    Gokaslan, A. and Cohen, V. (2019). Open W eb T ext C orpus. http://Skylion007.github.io/OpenWebTextCorpus

    2019

  11. [19]

    Hinton, G. (2015). Distilling the knowledge in a neural network. ArXiv , 1503.02531

    2015 arXiv

  12. [20]

    Y., Zhou, T., Wu, Y., Song, X., Song, X., and Zhou, D

    Hou, L., Pang, R. Y., Zhou, T., Wu, Y., Song, X., Song, X., and Zhou, D. (2022). Token dropping for efficient BERT pretraining. Annual Meeting of the Association for Computational Linguistics , pages 3774--3784

    2022

  13. [21]

    and Waegeman, W

    H \"u llermeier, E. and Waegeman, W. (2021). Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine learning , 110(3):457--506

    2021

  14. [22]

    Isik, B., Ponomareva, N., Hazimeh, H., Paparas, D., Vassilvitskii, S., and Koyejo, S. (2025). Scaling laws for downstream task performance in machine translation. International Conference on Learning Representations

    2025

  15. [23]

    H., Wong, D

    Jiang, A. H., Wong, D. L.-K., Zhou, G., Andersen, D. G., Dean, J., Ganger, G. R., Joshi, G., Kaminksy, M., Kozuch, M., Lipton, Z. C., et al. (2019). Accelerating deep learning by focusing on the biggest losers. ArXiv , 1910.00762

    2019 arXiv

  16. [24]

    Joshi , M., Choi , E., Weld , D., and Zettlemoyer , L. (2017). TriviaQA : A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension . ArXiv , 1705.03551

    2017 arXiv

  17. [25]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020). Scaling laws for neural language models. ArXiv , 2001.08361

    2020 arXiv

  18. [26]

    Karpathy, A. (2022). NanoGPT [ GitHub repository]. https://github.com/karpathy/nanoGPT

    2022

  19. [27]

    and Fleuret, F

    Katharopoulos, A. and Fleuret, F. (2018). Not all samples are created equal: Deep learning with importance sampling. International Conference on Machine Learning , pages 2525--2534

    2018

  20. [28]

    Killamsetty, K., Durga, S., Ramakrishnan, G., De, A., and Iyer, R. (2021). Grad-match: Gradient matching based data subset selection for efficient deep model training. International Conference on Machine Learning , pages 5464--5474

    2021

  21. [29]

    and Rush, A

    Kim, Y. and Rush, A. M. (2016). Sequence-level knowledge distillation. Conference on Empirical Methods in Natural Language Processing , pages 1317--1327

    2016

  22. [30]

    Kuhn, D., Shafiee, S., and Wiesemann, W. (2025). Distributionally robust optimization. ArXiv , 2411.02549

    2025 arXiv

  23. [31]

    Lin, Z., Gou, Z., Gong, Y., Liu, X., Shen, Y., Xu, R., Lin, C., Yang, Y., Jiao, J., Duan, N., et al. (2024). Rho-1: Not all tokens are what you need. ArXiv , 2404.07965

    2024 arXiv

  24. [32]

    and Hutter, F

    Loshchilov, I. and Hutter, F. (2015). Online batch selection for faster training of neural networks. ArXiv , 1511.06343

    2015 arXiv

  25. [33]

    Maehara, T. (2015). Risk averse submodular utility maximization. Operations Research Letters , 43(5):526--529

    2015

  26. [34]

    Marion, M., Üstün, A., Pozzobon, L., Wang, A., Fadaee, M., and Hooker, S. (2023). When less is more: Investigating data pruning for pretraining LLM s at scale. ArXiv , 2309.04564

    2023 arXiv

  27. [35]

    Mayilvahanan, P., Wiedemer, T., Mallick, S., Bethge, M., and Brendel, W. (2025). LLM s on the line: Data determines loss-to-loss scaling laws. ArXiv , 2502.12120

    2025 arXiv

  28. [36]

    Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. (2018). Can a suit of armor conduct electricity? a new dataset for open book question answering. ArXiv , 1809.02789

    2018 arXiv

  29. [37]

    M., Razzak, M

    Mindermann, S., Brauner, J. M., Razzak, M. T., Sharma, M., Kirsch, A., Xu, W., H \"o ltgen, B., Gomez, A. N., Morisot, A., Farquhar, S., et al. (2022). Prioritized training on points that are learnable, worth learning, and not yet learnt. International Conference on Machine Le...

    2022

  30. [38]

    B., and Liang, P

    Oren, Y., Sagawa, S., Hashimoto, T. B., and Liang, P. (2019). Distributionally robust language modeling. ArXiv , 1909.02060

    2019 arXiv

  31. [39]

    Q., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernandez, R

    Paperno, D., Kruszewski, G., Lazaridou, A., Pham, N. Q., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernandez, R. (2016). The LAMBADA dataset: Word prediction requiring a broad discourse context. Annual Meeting of the Association for Computational Linguistics , pa...

    2016

  32. [40]

    Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. (2019). PyTorch: A...

    2019

  33. [41]

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI blog , 1(8):9

    2019

  34. [42]

    Ramanujan, V., Nguyen, T., Oh, S., Farhadi, A., and Schmidt, L. (2023). On the connection between pre-training data diversity and fine-tuning robustness. Advances in Neural Information Processing Systems , 36:66426--66437

    2023

  35. [43]

    S., Sadhanala, V., Rostamizadeh, A., Chakrabarti, A., Jitkrittum, W., Feinberg, V., Kim, S., Harutyunyan, H., Saunshi, N., Nado, Z., et al

    Rawat, A. S., Sadhanala, V., Rostamizadeh, A., Chakrabarti, A., Jitkrittum, W., Feinberg, V., Kim, S., Harutyunyan, H., Saunshi, N., Nado, Z., et al. (2024). A little help goes a long way: Efficient LLM training by leveraging small LM s. ArXiv , 2410.18779

    2024 arXiv

  36. [44]

    Rockafellar, R. T. and Uryasev, S. (2002). Conditional value-at-risk for general loss distributions. Journal of Banking & Finance , 26(7):1443--1471

    2002

  37. [45]

    T., Uryasev, S., et al

    Rockafellar, R. T., Uryasev, S., et al. (2000). Optimization of conditional value-at-risk. Journal of Risk , 2:21--42

    2000

  38. [46]

    H., Caverlee, J., McAuley, J., and Cheng, D

    Sachdeva, N., Coleman, B., Kang, W.-C., Ni, J., Hong, L., Chi, E. H., Caverlee, J., McAuley, J., and Cheng, D. Z. (2024). How to train data-efficient LLM s. ArXiv , 2402.09668

    2024 arXiv

  39. [47]

    Sennrich, R., Haddow, B., and Birch, A. (2016). Neural machine translation of rare words with subword units. Annual Meeting of the Association for Computational Linguistics , pages 1715--1725

    2016

  40. [48]

    Shannon, C. E. (1948). A mathematical theory of communication. The Bell system technical journal , 27(3):379--423

    1948

  41. [49]

    R., Hestness, J., and Dey, N

    Soboleva, D., Al-Khateeb, F., Myers, R., Steeves, J. R., Hestness, J., and Dey, N. (2023). SlimPajama: A 627B token cleaned and deduplicated version of RedPajama . https://cerebras.ai/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama

    2023

  42. [50]

    Sow, D., Woisetschl \"a ger, H., Bulusu, S., Wang, S., Jacobsen, H.-A., and Liang, Y. (2025). Dynamic loss-based sample reweighting for improved large language model pretraining. ArXiv , 2502.06733

    2025 arXiv

  43. [51]

    Tirumala, K., Simig, D., Aghajanyan, A., and Morcos, A. (2023). D4: Improving LLM pretraining via document de-duplication and diversification. Advances in Neural Information Processing Systems , 36:53983--53995

    2023

  44. [52]

    Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. (2019). Super GLUE : A stickier benchmark for general-purpose language understanding systems. Advances in Neural Information Processing Systems , 32

    2019

  45. [53]

    T., Wu, T., Song, D., Mittal, P., and Jia, R

    Wang, J. T., Wu, T., Song, D., Mittal, P., and Jia, R. (2024). GREATS : Online selection of high-quality data for LLM training in every iteration. Advances in Neural Information Processing Systems , 37:131197--131223

    2024

  46. [54]

    F., and Gardner, M

    Welbl, J., Liu, N. F., and Gardner, M. (2017). Crowdsourcing multiple choice science questions. ArXiv , 1707.06209

    2017 arXiv

  47. [55]

    Wettig, A., Gupta, A., Malik, S., and Chen, D. (2024). QuRating : Selecting high-quality data for training language models. International Conference on Machine Learning , pages 52915--52971

    2024

  48. [56]

    and Menon, A

    Williamson, R. and Menon, A. (2019). Fairness risk measures. International Conference on Machine Learning , pages 6786--6797

    2019

  49. [57]

    Xia, M., Malladi, S., Gururangan, S., Arora, S., and Chen, D. (2024). LESS : Selecting influential data for targeted instruction tuning. International Conference on Machine Learning , pages 54104--54132

    2024

  50. [58]

    M., Pham, H., Dong, X., Du, N., Liu, H., Lu, Y., Liang, P

    Xie, S. M., Pham, H., Dong, X., Du, N., Liu, H., Lu, Y., Liang, P. S., Le, Q. V., Ma, T., and Yu, A. W. (2023a). Do R e M i: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems , 36:69798--69818

    2023

  51. [59]

    M., Santurkar, S., Ma, T., and Liang, P

    Xie, S. M., Santurkar, S., Ma, T., and Liang, P. S. (2023b). Data selection for language models via importance resampling. Advances in Neural Information Processing Systems , 36:34201--34227

    2023

  52. [60]

    Yu, Z., Das, S., and Xiong, C. (2024). MATES : Model-aware data selection for efficient pretraining with data influence models. Advances in Neural Information Processing Systems , 37:108735--108759

    2024

  53. [61]

    Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. (2019). HellaSwag : Can a machine really finish your sentence? ArXiv , 1905.07830

    2019 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.