Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

BLISS claims pretraining data can be scored and selected from scratch, without any external model, by training a small proxy to convergence and using its validation loss to learn influence scores.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 11:12 UTC pith:A6KF5IVY

load-bearing objection BLISS is a fresh bilevel data-selection idea with a serious theory–practice gap and contradictory FLOPs numbers that undermine the headline speedup. the 3 major comments →

arxiv 2510.06048 v5 pith:A6KF5IVY submitted 2025-10-07 cs.LG

BLISS: A Lightweight Bilevel Influence Scoring Method for Data Selection in Language Model Pretraining

Jie Hao , Rui Yu , Wei Zhang , Huixia Wang , Jie Xu , Mingrui Liu This is my paper
classification cs.LG
keywords data selectionLLM pretrainingbilevel optimizationinfluence scoringproxy modelknowledge distillationdata curationlong-term influence
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that the right pretraining data for a large language model can be identified without any outside pretrained oracle: train a small 'proxy' model on a weighted training loss until it converges, and simultaneously train a 'score' model so that the weights it assigns make the converged proxy perform best on validation data. The trained score model then assigns an influence score to every sample, and keeping the top 20% of samples produces a dataset that trains a much larger model faster and to higher downstream accuracy than selecting by a single-step influence estimate. The paper argues this accounts for the long-term effect of data across full training, not just one gradient step. A reader should care because data curation is a bottleneck in LLM pretraining; if this works, high-value data can be found cheaply and without depending on or paying for third-party models.

Core claim

On the paper's own terms, the central discovery is that solving the bilevel problem — choose per-sample weights so that a small proxy model trained to convergence on the weighted loss achieves the lowest validation loss — produces influence scores whose top-20% selection outperforms the state-of-the-art single-step selection method. In the 1B-scale experiment the selected data reaches the same downstream accuracy 1.7 times faster; scaled to 2.8B parameters with data selected by the 1B run, it gains 1.4% average accuracy over the baseline; and the same recipe transfers to a different model architecture. The long-term component is the load-bearing difference: the proxy is trained to convergenc

What carries the argument

The central object is a bilevel optimization loop. The lower level trains a lightweight proxy model on a loss in which each sample is weighted by a softmax-normalized score from the score model, plus a KL term that keeps the proxy's output logits close to the target LLM's logits (so the proxy's preferences mirror the LLM's) and a weight-decay term. The upper level minimizes the converged proxy's loss on validation data; its gradient with respect to the score model is a hypergradient computed with Hessian-vector products, run for three inner steps. The optimized score model is a small regression head (pooling + linear + sigmoid) built from the proxy's transformer, and it emits the per-sample

Load-bearing premise

The load-bearing premise is that the ranking of data learned by a 31M-160M proxy transfers to the much larger target model; the paper supports this with a small number of runs, and if the ranking does not transfer, the whole bilevel machinery is optimizing a surrogate that does not matter.

What would settle it

Train a 2.8B model for the same token budget on three datasets: the top 20% by BLISS scores, the bottom 20%, and a random 20%; if the top-20% does not beat random (or if the bottom-20% beats top-20%) on held-out downstream tasks, the score model is not measuring value for that target. A cheaper check: measure the correlation between proxy-model score rankings and per-sample influence computed directly on the target model over a few hundred samples; near-zero or negative correlation falsifies the transfer premise.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Data selection can be done entirely from scratch, removing reliance on proprietary or external pretrained models and the biases and legal constraints they introduce.
  • Because the score model is trained on validation loss after convergence, the selected data reflects long-term training value rather than one-step loss change.
  • At the 1B scale, the method reaches a fixed downstream accuracy 1.7x faster in FLOPs than the single-step baseline; at 2.8B scale it improves average accuracy by 1.4% with the same 1B-selected data.
  • The recipe transfers across model families (the paper tests it on a LLaMA-style 0.5B model with a 134M proxy) and across proxy sizes (31M proxy suffices for a 410M target).
  • Multi-round selection works with score-model reuse: the score model continues learning across rounds, while the proxy is reset each round to avoid overfitting the previous shard.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If ranking transfer holds generally, this turns data selection into a cheap filtering pass: a single small score model could be released and reused to score any new corpus for a family of target models, amortizing the bilevel cost.
  • The KL-alignment term means the selected data is judged relative to what the target LLM currently knows; extending to multimodal pretraining would require a comparable alignment signal between the small proxy and the target vision-language model, which the paper does not demonstrate.
  • A testable extension: use the score model to score held-out data at later stages of LLM training and compare with round-specific scores; if scores drift, the method's multi-round reuse depends on the score model tracking the LLM's evolving preferences.
  • The evidence for scale transfer rests on a limited number of runs; a stronger claim would need repeated seeds and a direct measurement of how proxy rank correlates with target-model influence.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes BLISS, a data-selection method for LLM pretraining that operates without external pretrained oracles. It introduces a score model that weights training samples and a small proxy model, coupled through a bilevel optimization problem; the lower level trains the proxy on weighted samples with KL alignment to the target LLM, and the upper level updates the score model via a hypergradient to minimize a validation loss. After score-model training, the top-20% highest-scoring samples are selected for each pretraining round. Experiments on Pythia-410M/1B/2.8B and LLaMA-0.5B with C4 data report improvements over MATES, including a claimed 1.7× speedup at 1B scale and 1.4% improvement at 2.8B scale. The paper also provides ablations on the bilevel formulation, KL divergence, softmax weighting, proxy size, initialization, and validation data.

Significance. If the central claims hold, BLISS is a useful contribution: it provides a principled, model-agnostic method for data selection that avoids external pretrained models and explicitly targets the long-term effect of data under continued training. The bilevel formulation is clean, the experiments span multiple model sizes and architectures, and the ablations give insight into the components. However, the paper's strongest claims currently rest on an implementation that does not match the formal convergence assumption, and the headline FLOPs comparison contains a numerical inconsistency. The lack of code and the single-run transfer experiments further limit confidence. These issues are fixable, but they are load-bearing for the central message.

major comments (3)
  1. [§4.2, Eq. (3), Algorithm 1, Table 9] The hypergradient in Eq. (3) is derived under the assumption that θ_p is the exact argmin of the lower-level objective in Eq. (1), and the text repeatedly says the proxy is 'trained to convergence.' In practice, Algorithm 1 performs one proxy update per score-model step (Pythia-410M) or five inner steps (Pythia-1B), and Appendix E fixes the proxy at 3,000 total steps (1,000 for LLaMA) with no convergence check. Figures 6-7 show the lower-level loss declining and then rising, so θ_p is not near a stationary point. The computed update is therefore not the hypergradient of the stated objective, and the claimed distinction from MATES—explicitly accounting for convergence—is not established. The authors should either implement a lower-level solver that reaches convergence, measure the truncation error of the inner loop, or reframe the algorithm as a finite-horizon heuristic while adjusting th
  2. [Table 1 vs. Table 4; §5.5; Figure 2] There is a direct numerical contradiction in the reported FLOPs. Table 1 lists BLISS at 1B scale as 8.08×10^19, while Table 4, for the same setting, gives a total of 19.53×10^19. The 8.08 value equals the 410M BLISS total, suggesting a copy-paste error. This matters because the abstract, Section 5.5, and Figure 2 use FLOPs to claim that BLISS is much more efficient than MATES, with a 1.7× speedup. If the true total is 19.53×10^19, the speedup is not 1.7× and the advantage over MATES is only about 2%. Please correct the discrepancy and recompute the headline comparison with the actual total.
  3. [Tables 2 and 3; §5.4] The central practical assumption is that rankings learned by the small proxy/score model (31M/160M/134M) transfer to the much larger target model (410M/1B/2.8B). The only transfer evidence is Tables 2 and 3, which report single runs without standard errors, and no code is provided. In Table 2, random selection is actually better at Round 1 (45.2 vs. 45.9), and the claimed 1.4% advantage appears only at Round 3. In Table 3, the largest difference is 0.64%. Given the cost of the experiments, it is acceptable to show one run, but the paper should at least report multiple seeds or error bars, and releasing code/checkpoints is necessary for the reader to verify that the transfer is not an artifact of a particular run. Without this, the generality claim is under-supported.
minor comments (5)
  1. [§5.3] Typo: 'visulize' should be 'visualize'.
  2. [Table 5 caption] Typos: 'multiuple' should be 'multiple'; 'updata' should be 'update'.
  3. [Figure 1 caption] The caption says 'There are four main steps', but the list contains five items (warm-up, bilevel optimization, prediction/selection, retraining, evaluation). Please renumber or reword.
  4. [Appendix B] The LLaMA variant changes several algorithmic components at once—periodic resets every 50 steps, removal of the KL regularization from the lower-level loss, 4 inner steps, and a different number of total steps. This makes it difficult to attribute the improvement to the core bilevel mechanism. Please consider presenting it as a separate extension rather than direct evidence for the same method.
  5. [General] The phrase 'from scratch' is used while the target model is warmed up on random data for 10k steps before selection. Clarify that 'from scratch' means 'without external pretrained models', not 'without any warm-up training'.

Circularity Check

0 steps flagged

No significant circularity: the score model is fit to a validation proxy (LAMBADA) and the reported results are held-out downstream benchmarks, so the empirical claims do not reduce to the fitting objective.

full rationale

The central derivation is a standard bilevel surrogate: Eq. (1) defines the score model as the minimizer of the proxy's validation loss with respect to the converged weighted proxy, and the hypergradient in Eqs. (3)-(6) is the corresponding update. There is no step in which a quantity is fitted to the evaluation target and then reported as a prediction of that same target. The score model is trained using LAMBADA validation loss (Section 5.1), while the reported results are average zero-shot accuracies on SciQ, ARC-E/C, LogiQA, OBQA, BoolQ, HellaSwag, PIQA, and WinoGrande (Table 1); the evaluation set is therefore held out from the fitting objective. The 'long-term influence' language is a modeling assumption that equips the lower-level problem with a convergence argmin, not a fitted restatement of the final numbers. The self-citations to Hao et al. (2023, 2024) and Gong et al. (2024) concern bilevel optimization methodology and are not used to justify the empirical claim. The reviewer's concern that the lower level is not actually run to convergence (Algorithm 1 lines 9-11; Table 9: 3k proxy steps, K=3 GDLS steps; LLaMA uses periodic resets) is a substantive approximation gap between Eq. (1) and the implementation, but it is not circularity: a truncated optimization of a surrogate is still an independent empirical test against held-out benchmarks.

Axiom & Free-Parameter Ledger

8 free parameters · 4 axioms · 0 invented entities

No physical entities are introduced; the score and proxy models are algorithmic components, not independently falsifiable entities. The core load-bearing assumptions are the transferability of small-model rankings to large models and the adequacy of a finite-step proxy as a stand-in for convergence.

free parameters (8)
  • Selection ratio (top 20%) = 20%
    Chosen by hand; not swept in paper. Directly controls the size of the selected subset.
  • KL regularization coefficient gamma = 1e-2
    Hand-chosen; ablation only tests with/without KL, not a sweep.
  • Weight decay lambda = 1e-6
    Standard regularizer value, not tuned in the paper.
  • GDLS steps K = 3
    Approximates inverse-Hessian-vector product; no convergence check for the linear system solve.
  • Proxy/score training steps per round = 3k (Pythia), 1k (LLaMA)
    Not enough to guarantee lower-level convergence despite the paper's 'trained to convergence' language.
  • Validation set D_val = LAMBADA
    Choice of validation distribution; ablation in Appendix C.4 shows different validation sets shift downstream results.
  • Warm-up length = unspecified (10k steps example)
    The warm-up budget on random data is not precisely specified, but it is part of the pipeline.
  • Lower-level steps per iteration = 1 (410M), 5 (1B), 4 (LLaMA)
    Heuristic double-loop schedule to approximate the lower-level solution; varies by model setting.
axioms (4)
  • domain assumption The proxy model's learned influence ranking transfers to the larger target LLM.
    Core transfer assumption; supported only by the 2.8B transfer and LLaMA experiments, with single runs and no code.
  • domain assumption The lower-level problem has a well-defined minimizer and the Hessian inverse in Eq. (3) exists.
    The hypergradient formula assumes smoothness and invertibility that are not guaranteed for transformer loss landscapes.
  • domain assumption KL distillation aligns the proxy model's data preferences with the LLM's preferences.
    Used to justify the proxy as a surrogate; no theoretical guarantee, and the LLM is imperfect during training.
  • domain assumption The validation set (LAMBADA) is representative of downstream task performance.
    The upper-level objective is optimized against LAMBADA; ablations show different validation sets change results, so representativeness is not automatic.

pith-pipeline@v1.3.0-alltime-deepseek · 20058 in / 11819 out tokens · 75612 ms · 2026-08-04T11:12:18.153583+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of BLISS: A Lightweight Bilevel Influence Scoring Method for Data Selection in Language Model Pretraining." pith.science (2026). https://pith.science/paper/A6KF5IVY

@misc{pith2026251006048,
  author       = {Pith},
  title        = {Pith review of: BLISS: A Lightweight Bilevel Influence Scoring Method for Data Selection in Language Model Pretraining},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A6KF5IVY}},
  note         = {Machine review of arXiv:2510.06048}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Effective data selection is essential for pretraining large language models (LLMs), enhancing efficiency and improving generalization to downstream tasks. However, existing approaches often require leveraging external pretrained models, making it difficult to disentangle the effects of data selection from those of the external pretrained models. In addition, they often overlook the long-term impact of selected data if the model is trained to convergence, primarily due to the prohibitive cost of full-scale LLM pretraining. In this paper, we introduce BLISS (\textbf{B}ileve\textbf{L} \textbf{I}nfluence \textbf{S}coring method for data \textbf{S}election): a lightweight data selection method that operates entirely \emph{from scratch}, without relying on any external pretrained oracle models, while explicitly accounting for the long-term impact of selected data. BLISS leverages a small proxy model as a surrogate for the LLM and employs a score model to estimate the long-term influence of training samples if the proxy model is trained to convergence. We formulate data selection as a bilevel optimization problem, where the upper-level objective optimizes the score model to assign importance weights to training samples, ensuring that minimizing the lower-level objective (i.e., training the proxy model over the weighted training loss until convergence) leads to best validation performance. Once optimized, the trained score model predicts influence scores for the dataset, enabling efficient selection of high-quality samples for LLM pretraining. We validate BLISS by pretraining 410M/1B/2.8B Pythia and LLaMA-0.5B models on selected subsets of the C4 dataset. Notably, under the 1B model setting, BLISS achieves $1.7\times$ speedup in reaching the same performance as the state-of-the-art method, demonstrating superior performance across multiple downstream tasks.

Figures

Figures reproduced from arXiv: 2510.06048 by Huixia Wang, Jie Hao, Jie Xu, Mingrui Liu, Rui Yu, Wei Zhang.

Figure 1
Figure 1. Figure 1: The pipeline of data selection and pretraining procedure. There are four main steps in one [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The downstream performance of Pythia-410M/1B model w.r.t. pretraining FLOPs and [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The performance comparison of bilevel optimization with/without KL divergence. The [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The evolution of the lower-level training loss and KL divergence for different proxy model [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Comparison of BLISS trained with different validation datasets (410M model, 10B tokens). [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The evolution of the lower-level train￾ing loss and upper-level validation loss in round 2. (a) Training loss vs. steps (b) Training loss vs. steps [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: The visualization of lower-level training loss and the upper-level validation loss in round 2 [PITH_FULL_IMAGE:figures/full_fig_p019_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. BLADE: Scalable Bi-level Adaptive Data Selection for LLM Training

    cs.LG 2026-06 unverdicted novelty 6.0

    BLADE converts influence-based bi-level data selection into a Hessian-free penalized objective with a dynamic reference model, proves first-order convergence, and reports better performance than prior methods on LLM training.

  2. Let the Target Select for Itself: Data Selection via Target-Aligned Paths

    cs.LG 2026-05 unverdicted novelty 6.0

    Target-aligned data selection via normalized endpoint loss drop on a validation-induced reference path achieves competitive performance with reduced computational overhead.

Reference graph

Works this paper leans on

72 extracted references · 33 linked inside Pith · cited by 2 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Semdedup: Data-efficient learning at web-scale through semantic deduplication

    Amro Abbas, Kushal Tirumala, D \'a niel Simig, Surya Ganguli, and Ari S Morcos. Semdedup: Data-efficient learning at web-scale through semantic deduplication. arXiv preprint arXiv:2303.09540, 2023

    Pith/arXiv arXiv 2023

  3. [3]

    Efficient online data mixing for language model pre-training

    Alon Albalak, Liangming Pan, Colin Raffel, and William Yang Wang. Efficient online data mixing for language model pre-training. In R0-FoMo: Robustness of Few-shot and Zero-shot Learning in Large Foundation Models, 2023

    2023

  4. [4]

    A survey on data selection for language models

    Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. A survey on data selection for language models. arXiv preprint arXiv:2402.16827, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    Piqa: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp.\ 7432--7439, 2020

    2020

  6. [6]

    Coresets via bilevel optimization for continual learning and streaming

    Zal \'a n Borsos, Mojmir Mutny, and Andreas Krause. Coresets via bilevel optimization for continual learning and streaming. Advances in Neural Information Processing Systems, 33: 0 14879--14890, 2020

    2020

  7. [7]

    Mathematical programs with optimization problems in the constraints

    Jerome Bracken and James T McGill. Mathematical programs with optimization problems in the constraints. Operations Research, 21 0 (1): 0 37--44, 1973

    1973

  8. [8]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin,...

    1901

  9. [9]

    On bilevel optimization without lower-level strong convexity

    Lesi Chen, Jing Xu, and Jingzhao Zhang. On bilevel optimization without lower-level strong convexity. arXiv preprint arXiv:2301.00712, 2023

    Pith/arXiv arXiv 2023

  10. [10]

    Skill-it! a data-driven skills framework for understanding and training language models

    Mayee Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher R \'e . Skill-it! a data-driven skills framework for understanding and training language models. Advances in Neural Information Processing Systems, 36, 2024

    2024

  11. [11]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24 0 (240): 0 1--113, 2023

    2023

  12. [12]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019

    Pith/arXiv arXiv 1905

  13. [13]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    Pith/arXiv arXiv 2018

  14. [14]

    Cross-lingual language model pretraining

    Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining. Advances in neural information processing systems, 32, 2019

    2019

  15. [15]

    Detection of influential observation in linear regression

    R Dennis Cook. Detection of influential observation in linear regression. Technometrics, 19 0 (1): 0 15--18, 1977

    1977

  16. [16]

    A framework for bilevel optimization that enables stochastic and global variance reduction algorithms

    Mathieu Dagr \'e ou, Pierre Ablin, Samuel Vaiter, and Thomas Moreau. A framework for bilevel optimization that enables stochastic and global variance reduction algorithms. arXiv preprint arXiv:2201.13409, 2022

    Pith/arXiv arXiv 2022

  17. [17]

    Foundations of bilevel programming

    Stephan Dempe. Foundations of bilevel programming. Springer Science & Business Media, 2002

    2002

  18. [18]

    Glam: Efficient scaling of language models with mixture-of-experts

    Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, pp.\ 5547--5569. PMLR, 2022

    2022

  19. [19]

    What's in my big data? arXiv preprint arXiv:2310.20707, 2023

    Yanai Elazar, Akshita Bhagia, Ian Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, et al. What's in my big data? arXiv preprint arXiv:2310.20707, 2023

    Pith/arXiv arXiv 2023

  20. [20]

    Dsdm: Model-aware dataset selection with datamodels

    Logan Engstrom, Axel Feldmann, and Aleksander Madry. Dsdm: Model-aware dataset selection with datamodels. arXiv preprint arXiv:2401.12926, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    Doge: Domain reweighting with generalization estimation

    Simin Fan, Matteo Pagliardini, and Martin Jaggi. Doge: Domain reweighting with generalization estimation. arXiv preprint arXiv:2310.15393, 2023

    Pith/arXiv arXiv 2023

  22. [22]

    Model-agnostic meta-learning for fast adaptation of deep networks

    Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp.\ 1126--1135. PMLR, 2017

    2017

  23. [23]

    Bilevel programming for hyperparameter optimization and meta-learning

    Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In International Conference on Machine Learning, pp.\ 1568--1577. PMLR, 2018

    2018

  24. [24]

    The pile: An 800gb dataset of diverse text for language modeling

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020

    Pith/arXiv arXiv 2020

  25. [25]

    A framework for few-shot language model evaluation

    Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. A framework for few-shot language model evaluation. Version v0. 0.1. Sept, 10: 0 8--9, 2021

    2021

  26. [26]

    Approximation methods for bilevel programming

    Saeed Ghadimi and Mengdi Wang. Approximation methods for bilevel programming. arXiv preprint arXiv:1802.02246, 2018

    Pith/arXiv arXiv 2018

  27. [27]

    A nearly optimal single loop algorithm for stochastic bilevel optimization under unbounded smoothness

    Xiaochuan Gong, Jie Hao, and Mingrui Liu. A nearly optimal single loop algorithm for stochastic bilevel optimization under unbounded smoothness. In Forty-first International Conference on Machine Learning, 2024

    2024

  28. [28]

    Gemini API Additional Terms of Service , 2024

    Google . Gemini API Additional Terms of Service , 2024. URL https://ai.google.dev/gemini-api/terms. Accessed: January 30, 2025

    2024

  29. [29]

    Bilevel optimization to learn training distributions for language modeling under domain shift

    David Grangier, Pierre Ablin, and Awni Hannun. Bilevel optimization to learn training distributions for language modeling under domain shift. In NeurIPS 2023 Workshop on Distribution Shifts: New Frontiers with Foundation Models, 2023

    2023

  30. [30]

    Bilevel optimization with a lower-level contraction: Optimal sample complexity without warm-start

    Riccardo Grazzi, Massimiliano Pontil, and Saverio Salzo. Bilevel optimization with a lower-level contraction: Optimal sample complexity without warm-start. arXiv preprint arXiv:2202.03397, 2022

    Pith/arXiv arXiv 2022

  31. [31]

    The influence curve and its role in robust estimation

    Frank R Hampel. The influence curve and its role in robust estimation. Journal of the american statistical association, 69 0 (346): 0 383--393, 1974

    1974

  32. [32]

    Bilevel coreset selection in continual learning: A new formulation and algorithm

    Jie Hao, Kaiyi Ji, and Mingrui Liu. Bilevel coreset selection in continual learning: A new formulation and algorithm. Advances in Neural Information Processing Systems, 36, 2023

    2023

  33. [33]

    Bilevel optimization under unbounded smoothness: A new algorithm and convergence analysis

    Jie Hao, Xiaochuan Gong, and Mingrui Liu. Bilevel optimization under unbounded smoothness: A new algorithm and convergence analysis. In The Twelfth International Conference on Learning Representations, 2024

    2024

  34. [34]

    A two-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actor-critic

    Mingyi Hong, Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang. A two-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actor-critic. SIAM Journal on Optimization, 33 0 (1): 0 147--180, 2023

    2023

  35. [35]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    Pith/arXiv arXiv 2021

  36. [36]

    Bilevel optimization: Convergence analysis and enhanced design

    Kaiyi Ji, Junjie Yang, and Yingbin Liang. Bilevel optimization: Convergence analysis and enhanced design. In International conference on machine learning, pp.\ 4882--4892. PMLR, 2021

    2021

  37. [37]

    Understanding black-box predictions via influence functions

    Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pp.\ 1885--1894. PMLR, 2017

    2017

  38. [38]

    A fully first-order method for stochastic bilevel optimization

    Jeongyeol Kwon, Dohyun Kwon, Stephen Wright, and Robert D Nowak. A fully first-order method for stochastic bilevel optimization. In International Conference on Machine Learning, pp.\ 18083--18113. PMLR, 2023

    2023

  39. [39]

    The bigscience roots corpus: A 1.6 tb composite multilingual dataset

    Hugo Lauren c on, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo Gonz \'a lez Ponferrada, Huu Nguyen, et al. The bigscience roots corpus: A 1.6 tb composite multilingual dataset. Advances in Neural Information Processing Systems, 35: 0 31809--31826, 2022

    2022

  40. [40]

    Deduplicating training data makes language models better

    Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021

    Pith/arXiv arXiv 2021

  41. [41]

    Datacomp-lm: In search of the next generation of training sets for language models

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, et al. Datacomp-lm: In search of the next generation of training sets for language models. arXiv preprint arXiv:2406.11794, 2024

    Pith/arXiv arXiv 2024

  42. [42]

    Residuals and influence in regression, 1984

    Robert F Ling. Residuals and influence in regression, 1984

    1984

  43. [43]

    Logiqa: A challenge dataset for machine reading comprehension with logical reasoning

    Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. arXiv preprint arXiv:2007.08124, 2020

    Pith/arXiv arXiv 2007

  44. [44]

    Rephrasing the web: A recipe for compute and data-efficient language modeling

    Pratyush Maini, Skyler Seto, He Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly. Rephrasing the web: A recipe for compute and data-efficient language modeling. arXiv preprint arXiv:2401.16380, 2024

    Pith/arXiv arXiv 2024

  45. [45]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789, 2018

    Pith/arXiv arXiv 2018

  46. [46]

    OpenAI Terms of Service , 2024

    OpenAI . OpenAI Terms of Service , 2024. URL https://openai.com/terms. Accessed: Jan 30, 2025

    2024

  47. [47]

    Distributionally robust language modeling

    Yonatan Oren, Shiori Sagawa, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust language modeling. arXiv preprint arXiv:1909.02060, 2019

    Pith/arXiv arXiv 1909

  48. [48]

    Scalebio: Scalable bilevel optimization for llm data reweighting

    Rui Pan, Jipeng Zhang, Xingyuan Pan, Renjie Pi, Xiaoyu Wang, and Tong Zhang. Scalebio: Scalable bilevel optimization for llm data reweighting. arXiv preprint arXiv:2406.19976, 2024

    Pith/arXiv arXiv 2024

  49. [50]

    Trak: Attributing model behavior at scale

    Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. Trak: Attributing model behavior at scale. arXiv preprint arXiv:2303.14186, 2023

    Pith/arXiv arXiv 2023

  50. [51]

    The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only

    Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116, 2023

    Pith/arXiv arXiv 2023

  51. [52]

    Scaling language models: Methods, analysis & insights from training gopher

    Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021

    Pith/arXiv arXiv 2021

  52. [53]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

    2020

  53. [54]

    Squad: 100,000+ questions for machine comprehension of text

    P Rajpurkar. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016

    Pith/arXiv arXiv 2016

  54. [55]

    Distributionally robust neural networks

    Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations (ICLR), 2019

    2019

  55. [56]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64 0 (9): 0 99--106, 2021

    2021

  56. [57]

    Seal: Safety-enhanced aligned llm fine-tuning via bilevel data selection

    Han Shen, Pin-Yu Chen, Payel Das, and Tianyi Chen. Seal: Safety-enhanced aligned llm fine-tuning via bilevel data selection. arXiv preprint arXiv:2410.07471, 2024

    Pith/arXiv arXiv 2024

  57. [58]

    Bi-level finetuning with task-dependent similarity structure for low-resource training

    Sai Ashish Somayajula, Lifeng Jin, Linfeng Song, Haitao Mi, and Dong Yu. Bi-level finetuning with task-dependent similarity structure for low-resource training. In Findings of the Association for Computational Linguistics: ACL 2023, pp.\ 8569--8588, 2023

    2023

  58. [59]

    Beyond neural scaling laws: beating power law scaling via data pruning

    Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35: 0 19523--19536, 2022

    2022

  59. [60]

    D4: Improving llm pretraining via document de-duplication and diversification

    Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari Morcos. D4: Improving llm pretraining via document de-duplication and diversification. Advances in Neural Information Processing Systems, 36: 0 53983--53995, 2023

    2023

  60. [61]

    Crowdsourcing multiple choice science questions

    Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. arXiv preprint arXiv:1707.06209, 2017

    Pith/arXiv arXiv 2017

  61. [62]

    Ccnet: Extracting high quality monolingual datasets from web crawl data

    Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzm \'a n, Armand Joulin, and Edouard Grave. Ccnet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.00359, 2019

    Pith/arXiv arXiv 1911

  62. [63]

    Qurating: Selecting high-quality data for training language models

    Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. Qurating: Selecting high-quality data for training language models. arXiv preprint arXiv:2402.09739, 2024

    Pith/arXiv arXiv 2024

  63. [64]

    Sheared llama: Accelerating language model pre-training via structured pruning

    Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared llama: Accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.06694, 2023

    Pith/arXiv arXiv 2023

  64. [65]

    Less: Selecting influential data for targeted instruction tuning

    Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. arXiv preprint arXiv:2402.04333, 2024

    Pith/arXiv arXiv 2024

  65. [66]

    Doremi: Optimizing data mixtures speeds up language model pretraining

    Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 2023 a

    2023

  66. [67]

    Data selection for language models via importance resampling

    Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36: 0 34201--34227, 2023 b

    2023

  67. [68]

    Mates: Model-aware data selection for efficient pretraining with data influence models

    Zichun Yu, Spandan Das, and Chenyan Xiong. Mates: Model-aware data selection for efficient pretraining with data influence models. arXiv preprint arXiv:2406.06046, 2024

    Pith/arXiv arXiv 2024

  68. [69]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    Pith/arXiv arXiv 1905

  69. [70]

    Probabilistic bilevel coreset selection

    Xiao Zhou, Renjie Pi, Weizhong Zhang, Yong Lin, Zonghao Chen, and Tong Zhang. Probabilistic bilevel coreset selection. In International Conference on Machine Learning, pp.\ 27287--27302. PMLR, 2022

    2022

  70. [71]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  71. [72]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  72. [73]

    Natural language processing of symptoms documented in free-text narratives of electronic health records: a systematic review

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    Pith/arXiv arXiv 2016