Pith. sign in

REVIEW 4 major objections 7 minor 1 cited by

LLM Data Selection and Utilization via Dynamic Bi-level Optimization

T0 review · 4 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A per-batch data weighting model, learned end-to-end, improves LLM pretraining beyond what random selection alone achieves.

desk verdict Overstated headline claim obscures a plausible but thinly supported result; DWM reliably helps random/DSIR data at 370M, but QuRating still beats it and the meta-learning core is uncited. read the letter →

arxiv 2507.16178 v1 pith:75QJ73FW submitted 2025-07-22 cs.LG cs.AI

Yang Yu , Kai Han , Hang Zhou , Yehui Tang , Kaiqi Huang , Yunhe Wang , Dacheng Tao This is my paper
classification cs.LGcs.AI
keywords dataselectionweightingbi-leveloptimizationlargelanguagemodelspretrainingefficiencydynamicpreferenceper-batchreweightingLAMBADAvalidation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that how training data is used matters as much as which data is selected: a small per-batch weighting model can reweight the samples in every batch of an LLM pretraining run and improve downstream accuracy without changing the data budget. The authors train this Data Weighting Model (DWM) through a bi-level loop, where the inner loop trains the LLM on weighted batches and the outer loop adjusts the weights to maximize the model's validation accuracy. The headline result is that a 370M model pretrained on randomly selected SlimPajama data with DWM averages 46.4 two-shot accuracy across nine tasks, versus 45.1 for the same random data without DWM and 45.3 for DSIR-selected data. The same learned weighting model transfers to a 1.3B model and to data selected by other methods, and its preferences shift over training from well-written general text toward expertise-heavy reasoning content.

What carries the argument

The working object is the Data Weighting Model (DWM), a small Transformer that reads all samples in a micro-batch and outputs a weight per sample; these weights rescale the per-sample autoregressive losses before backpropagation. The carrying identity is the chained bi-level update in Eq. (6), obtained by unrolling one step of LLM training and differentiating the validation reward through the unrolled parameters. What makes it work, per the paper, is the stage-wise alternation that re-trains the weighting model on the current LLM state, so the weights reflect the model's present preferences rather than a fixed quality score.

What would settle it

Run the same 370M random-data setup but replace the LAMBADA reward with an i.i.d. held-out split of SlimPajama; if the two-shot average gain over RANDOM (1.3 points in Table 2) disappears or reverses, the reported benefit comes from the proxy task rather than from dynamic weighting per se.

Watch

Extended reading notes

Core claim

The central claim is that per-batch data weighting can be learned by treating model training as the lower level of a bi-level optimization: $\theta^*(\theta_w) = \arg\min_\theta \sum_i \omega_i \mathcal{L}_{\mathrm{train},i}(\theta)$ with weights $\omega = \theta_w(X_1,\dots,X_{bs})$ produced by the weighting model, while the upper level maximizes a validation reward $R_{\mathrm{val}}(\theta^*(\theta_w))$, implemented as accuracy on LAMBADA. Replacing the inner argmin by a single gradient step allows the weighting model to be updated by the chain rule, and repeating this alternation across five stages lets the weights track the model's changing data preferences. The paper reports that this DWM improves average two-shot accuracy from 45.1 to 46.4 on nine tasks for a 370M model trained on randomly selected data, that it also improves DSIR- and QuRating-selected data on average, and that a weighting model learned at 370M transfers to a 1.3B model with roughly 9% additional training FLOPs.

Load-bearing premise

Everything hinges on the assumption that one gradient step of LLM training, scored by LAMBADA accuracy, is a faithful guide to how batch weights will affect the final model on nine other downstream tasks.

Editorial extensions

If this is right

  • Models trained on randomly selected data can match or exceed the accuracy of some curated selections when every batch is reweighted by DWM.
  • The same weighting model trained on a 370M model transfers to a 1.3B model without retraining, adding roughly 9% to training FLOPs.
  • Applying DWM to DSIR-selected and QuRating-selected data yields further average gains, and the gains are complementary to the selection method itself.
  • Dynamic, stage-wise reweighting outperforms using either a fixed early-stage or a fixed late-stage weighting model.
  • The preferred data properties shift as training progresses: later stages favor expertise- and reasoning-heavy content over polished general-purpose writing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If DWM works as claimed, the notion of a fixed 'high-quality' dataset is incomplete: the usefulness of a token appears to depend on training stage and batch context, so future selection systems could output stage-dependent weights rather than static rankings.
  • The paper's reward is a single task (LAMBADA); a natural extension the authors do not pursue is to make the reward a combination of downstream tasks, which would allow DWM to steer pretraining toward specific capabilities but would require more validation compute and a multi-objective outer loop.
  • Because Sec. 5.3 reports that the stage-2 weighting model can temporarily lower performance relative to uniform weighting, a production deployment would need a checkpoint-selection or early-stopping rule for the weighting model itself, something the paper leaves as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes a Data Weighting Model (DWM) that assigns per-sample weights within each batch during LLM pre-training, learned through a dynamic bi-level optimization in which the outer objective is validation accuracy on LAMBADA after a one-step inner update of the LLM. The method is evaluated by pre-training 370M and 1.3B Llama-2-style models on 30B tokens selected from SlimPajama, comparing random, DSIR, and QuRating baselines, and by transferring the learned weighting model across selection methods and model sizes. The paper also reports ablations on the number of stages and on the validation task, and analyzes how learned weights correlate with QuRating-style data-quality dimensions over training stages.

Significance. The central idea—optimizing per-batch data weights against a validation signal during pre-training rather than relying on static, training-agnostic selection—is timely and potentially valuable. The paper's ablations (stage count, validation task, static vs. dynamic weighting) and the analysis of evolving data preferences are thoughtful and go beyond a single accuracy table. The transfer experiments to a 1.3B model and to other selection methods are a good-faith test of generality. However, the current evidence is thin: all results are single runs, average gains are small with several task-level regressions, and the headline claim in Section 1 that RANDOM+DWM outperforms carefully selected data is directly contradicted by Tables 3 and 8. The bi-level derivation also lacks the differentiable surrogate needed to backpropagate through an accuracy reward. These issues are fixable, but the manuscript as written overclaims and would need substantial revision before the contribution is convincingly supported.

major comments (4)
  1. [Section 1 / Tables 3 and 8] The claim in Section 1 that 'the model trained with DWM and randomly selected data outperforms both models trained with randomly-selected data and those with carefully selected data' is contradicted by the paper's own numbers. In the two-shot 370M comparison (Table 3), RANDOM+DWM averages 46.4 while QURATING averages 48.5, with QuRating winning 6 of 9 tasks (ARC-C 27.9 vs 24.7, ARC-E 56.6 vs 46.8, BoolQ 58.6 vs 56.6, HellaSwag 38.1 vs 36.5, OBQA 32.0 vs 28.2, SciQ 82.3 vs 80.5). The zero-shot results (Table 8) show the same pattern (45.0 vs 46.6). The supported claim is the more modest one in the abstract: DWM improves over randomly selected data, and it can be transferred to other selection methods with mixed task-wise outcomes. The introduction and Section 5.2 should be reworded to match the evidence, and the conclusion's 'competitive results' wording is appropriate and should be used consistently.
  2. [Section 3.4, Eqs. (4)-(6)] The gradient update in Eq. (6) differentiates the validation reward Rval, which is described as accuracy (Section 3.2). Accuracy is piecewise-constant, so the gradient is zero almost everywhere and undefined at jump points; no surrogate (e.g., soft accuracy, temperature-scaled logits, or a loss-based reward) is specified. This makes the core weighting-model update ill-posed as written. Moreover, Eq. (4) replaces the inner argmin in Eq. (1) with a single gradient step, an approximation that is not justified; the instability acknowledged in Section 5.3 (stage-2 weighting model causing a performance drop, visible in Table 1 at Stage 3 where RANDOM+DWM is 43.2 vs RANDOM 43.6) may stem from this approximation. Please specify the surrogate, provide the exact form of Rval used in the experiments, and report sensitivity to the unrolling step and the weighting-model learning rate η.
  3. [Section 4 / Tables 1-4] All reported results appear to come from single training runs with no variance estimates. The average improvements are small (e.g., +1.3 two-shot and +1.0 zero-shot at 370M for RANDOM vs RANDOM+DWM) and many individual tasks regress (Table 4 BoolQ 56.0→51.1 for RANDOM+DWM at 1.3B; Table 2 ARC-C 25.5→24.7; Table 1 Stage 3 zero-shot 43.6→43.2). Without multiple seeds or error bars, it is impossible to tell whether these differences are robust or within run-to-run noise, which is especially important for the transfer claims. Please provide at least three seeds for the key comparisons (RANDOM vs RANDOM+DWM at both scales, and QURATING vs QURATING+DWM) and report mean and standard deviation or confidence intervals.
  4. [Section 5.2 / Tables 4 and 9] The text claims 'consistent performance improvements' when DWM is transferred to a 1.3B model. However, Table 4 (two-shot) shows DSIR 48.7 vs DSIR+DWM 48.7, and Table 9 (zero-shot) shows DSIR 46.8 vs DSIR+DWM 46.8, i.e., no improvement; the gain over QuRating is 0.7 points (two-shot) and 1.4 points (zero-shot), within the range that could be noise given single runs. The interpretation in Section 5.2 about 'training saturation' and model capacity is speculative and not supported by additional experiments. Please either temper the transferability claim to match the data or provide additional evidence (e.g., more seeds, more downstream tasks, or a direct test of the saturation hypothesis).
minor comments (7)
  1. [Section 3.3] The notation for the weighting model is inconsistent (θw in Eq. (2) is used as a function producing a vector of weights, while elsewhere θw is a parameter vector; Eq. (3) and Section 3.4 sometimes use θω). Please standardize the notation and explicitly define how ω is obtained from θw and the batch (e.g., softmax over scores, with or without normalization).
  2. [Section 4] The description of the DWM architecture ('one attention block and two linear layers') is insufficient for reproducibility; please provide the full architecture, the input representation of a batch, and the weight normalization/constraint used in the experiments.
  3. [Section 5.2] 'Tale 3' should be 'Table 3'; throughout the paper 'Specially' should be 'Specifically' and 'preferred'/'unpreferred' should be used consistently.
  4. [Section 5.3] 'pro-collected' should be 'pre-collected', and 'prosperity' should be 'properties' in the analysis of Figure 3. The phrase 'as shown in the bottom-left corner of the Figure 3' is confusing because Figure 3 is a bar chart, not a performance curve; it likely refers to the stage-wise accuracy plots in Figure 2.
  5. [Tables 1-4] The table headers contain formatting artifacts (e.g., 'B OOL Q', 'L OQI QA') that should be cleaned up for readability.
  6. [Equation (6)] The displayed equation has a spacing issue with '∂θ ∗' and the chain rule would be clearer if the intermediate variables were named explicitly; please also check the consistent use of θw vs θω in this equation.
  7. [References] Reference Langley (2000) appears in the reference list but is never cited in the text; either cite it or remove it.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DWM's LAMBADA validation objective and the reported downstream benchmarks are disjoint, and the bi-level update is derived from first principles in Eqs. (4)-(6).

full rationale

The derivation chain is self-contained. The weighting model is trained in Eqs. (4)-(6) by one-step unrolling of the training loss (Eq. 3) and optimizing a reward Rval on LAMBADA, while the trained model is evaluated on nine downstream tasks (ARC, SciQ, LogiQA, BoolQ, OBQA, HellaSwag, PIQA, WinoGrande) that do not include LAMBADA. Thus the reported improvements are not forced by the objective by construction. No load-bearing self-citation is present: the only same-author citation in the background (Zhou et al., 2024) supports a general statement about data diversity, not the method's core claim, and no uniqueness theorem or ansatz is imported from prior work by these authors. The paper's own Table 5 ablation shows that the choice of validation task changes results, and Sec. 5.3 candidly admits that the stage-2 weighting model can cause a performance drop and leaves better weighting models to future work; these are empirical limitations rather than circular reductions. The Sec. 1 claim that RANDOM+DWM outperforms 'carefully selected data' is contradicted by Table 3 (QuRating 48.5 vs. RANDOM+DWM 46.4 at 370M two-shot), but that is an overstatement/correctness issue, not circularity, because the comparison is measured independently rather than derived from the method's definitions. Therefore the circularity score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The paper's central machinery, the DWM, is a new network component but relies on standard bi-level optimization and a hand-picked validation objective. The free parameters and assumptions listed above are the main degrees of freedom that are not derived from first principles and are not independently verified.

free parameters (5)
  • Number of training stages T = 5
    Chosen as a balance; ablation (Table 6/14) shows 8 stages slightly better and 2 stages worse, so 5 is a hand-picked hyperparameter rather than derived.
  • Micro-batch size bs = 8
    Sets how many samples the weighting model considers jointly; affects memory and the interaction modeling; chosen by hand.
  • Validation task for Rval = LAMBADA
    The validation set is a hand-picked choice; Table 5 shows it outperforms an i.i.d. held-out set, indicating the choice is consequential.
  • Weighting model learning rate eta = not reported
    The update in Eq. (7) requires a learning rate eta for the weighting model; the paper never states its value, so it is an unspecified tuned parameter.
  • Weighting model initialization schedule = partial init from trained model at stage 2
    The partial initialization is a design choice that affects transfer and stability, but is not specified precisely.
assumptions (4)
  • domain assumption LAMBADA accuracy is a representative proxy for the desired downstream generalization.
    The weighting model is trained to maximize Rval on LAMBADA (Sec 3.4, Eq. 5), and the paper relies on this to improve the nine evaluation tasks.
  • domain assumption One-step inner-loop unrolling approximates the true bi-level argmin.
    Eq. (4) replaces the exact theta_star(theta_w) in Eq. (1) with a single gradient step; the paper does not assess how many inner steps are needed.
  • domain assumption The learned weighting policy transfers across model sizes and data selection methods.
    Sec. 5.2 transfers the DWM from 370M to 1.3B and to DSIR/QuRating data without retraining, assuming the weighting function is portable.
  • domain assumption Weights assigned to one micro-batch generalize to other batches.
    The weighting model is trained on batches from the pre-training stream and then applied to new batches, so generalization across data distribution is assumed.
invented entities (1)
  • Data Weighting Model (DWM)
    purpose: A 370M-parameter network that maps a micro-batch of training samples to per-sample weights in Eq. (2).
    No external prediction or falsifiable handle is provided; its utility is only demonstrated in this paper's experiments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLM Data Selection and Utilization via Dynamic Bi-level Optimization." pith.science (2026). https://pith.science/paper/75QJ73FW

@misc{pith2026250716178,
  author       = {Pith},
  title        = {Pith review of: LLM Data Selection and Utilization via Dynamic Bi-level Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/75QJ73FW}},
  note         = {Machine review of arXiv:2507.16178}
}
read the original abstract

While large-scale training data is fundamental for developing capable large language models (LLMs), strategically selecting high-quality data has emerged as a critical approach to enhance training efficiency and reduce computational costs. Current data selection methodologies predominantly rely on static, training-agnostic criteria, failing to account for the dynamic model training and data interactions. In this paper, we propose a new Data Weighting Model (DWM) to adjust the weight of selected data within each batch to achieve a dynamic data utilization during LLM training. Specially, to better capture the dynamic data preference of the trained model, a bi-level optimization framework is implemented to update the weighting model. Our experiments demonstrate that DWM enhances the performance of models trained with randomly-selected data, and the learned weighting model can be transferred to enhance other data selection methods and models of different sizes. Moreover, we further analyze how a model's data preferences evolve throughout training, providing new insights into the data preference of the model during training.

Figures

Figures reproduced from arXiv: 2507.16178 by the authors.

Figure 1
Figure 1. The framework of the proposed bi-level optimization process with DWM, where the LLM model and the weighting model are trained alternated. During model training, the weighting model is frozen and the pre-training loss is the weighted sum of the loss of each data sample in one batch. Besides, to capture the data preference of the trained model, the weighting model is updated to minimize the validation loss of the mode… view at source ↗
Figure 2
Figure 2. Multi-stage performance of the 370M model using randomly-selected data with and without DWM. 5. Evaluation Results In this section, we want to evaluate the effect of the proposed DWM. We conduct experiments to answer the following questions: 1) Is it necessary to consider the data utilization in model pre-training with data selection? 2) What is the transferring performance of the DWM to larger models or other selec… view at source ↗
Figure 3
Figure 3. Preferred (red) and unpreferred (blue) data of the weight￾ing model in different training stages, considering properties of writing, expertise, facts and educational values. generalization, reducing bias, or enhancing specific capabil￾ities. On the contrary, DWM aims to capture the dynamic data preference during model training, which focuses on the interaction of data as well as the shifty data preference of model d… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparison of static and dynamic weighting model. DWM w1 means only using the weighting model of the first stage. DWM w4 means using the weighting model in the last stage. 5.4. Ablation Study Note that DWM emphasizes to learn the data preference dynamically, and uses a…
Figure 5
Figure 5. Figure 5: Preferred (red) and unpreferred (blue) data of the weighting model in different training stages, considering properties of different data domains [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Bilevel Data Curation for LLM Fine-tuning: Offline Selection and Online Self-Refining Generation

    cs.LG 2025-11 reject novelty 6.0 of 10

    A bilevel data-curation method for LLM fine-tuning that selects validation-aligned offline data and reweights online self-refined responses via importance ratios.

Reference graph

Works this paper leans on

19 extracted references · 3 canonical work pages · cited by 1 Pith paper

  1. [1]

    M., Longpre, S., Lambert, N., Wang, X., Muennighoff, N., Hou, B., Pan, L., Jeong, H., et al

    Albalak, A., Elazar, Y ., Xie, S. M., Longpre, S., Lambert, N., Wang, X., Muennighoff, N., Hou, B., Pan, L., Jeong, H., et al. A survey on data selection for language models. arXiv preprint arXiv:2402.16827,

    arXiv

  2. [5]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044,

    arXiv 1905

  3. [7]

    org/abs/2401.12926,

    URL https://arxiv. org/abs/2401.12926,

    arXiv

  4. [8]

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556,

    arXiv

  5. [10]

    Crafting papers on machine learning

    Langley, P. Crafting papers on machine learning. In Langley, P. (ed.),Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp. 1207–1216, Stan- ford, CA,

    2000

  6. [11]

    Logiqa: A challenge dataset for machine reading comprehension with logical reasoning

    Liu, J., Cui, L., Liu, H., Huang, D., Wang, Y ., and Zhang, Y . Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. arXiv preprint arXiv:2007.08124,

    arXiv 2007

  7. [12]

    Regmix: Data mixture as regression for language model pre-training

    Liu, Q., Zheng, X., Muennighoff, N., Zeng, G., Dou, L., Pang, T., Jiang, J., and Lin, M. Regmix: Data mixture as regression for language model pre-training. arXiv preprint arXiv:2407.01492,

    arXiv

  8. [13]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789,

    arXiv

Show all 19 references
  1. [14]

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi`ere, B., Goyal, N., Hambro, E., Azhar, F., et al

    URL https://huggingface.co/ datasets/cerebras/SlimPajama-627B. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi`ere, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2...

    arXiv

  2. [15]

    Qurating: Selecting high-quality data for training language models

    Wettig, A., Gupta, A., Malik, S., and Chen, D. Qurating: Selecting high-quality data for training language models. arXiv preprint arXiv:2402.09739,

    arXiv

  3. [16]

    Mates: Model-aware data selection for efficient pretraining with data influence mod- els

    Yu, Z., Das, S., and Xiong, C. Mates: Model-aware data selection for efficient pretraining with data influence mod- els. arXiv preprint arXiv:2406.06046,

    arXiv

  4. [17]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830,

    Zellers, R., Holtzman, A., Bisk, Y ., Farhadi, A., and Choi, Y . Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830,

    1905 arXiv

  5. [18]

    X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y ., Min, Y ., Zhang, B., Zhang, J., Dong, Z., et al

    10 LLM Data Selection and Utilization via Dynamic Bi-level Optimization Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y ., Min, Y ., Zhang, B., Zhang, J., Dong, Z., et al. A survey of large language models. arXiv preprint arXiv:2303.18223,

    arXiv

  6. [19]

    Architecture of pre-training models with 370M and 1.3B parameters. Hyperparameter 1.3B Model Value 370M Model Value V ocabulary Size 32,000 32,000 Hidden Size 2048 1024 FFN Hidden Size 5504 2812 Number of Layers 24 24 Number of Attention Heads 16 8 Number of KV Attention Heads...

  7. [2019]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457,

    arXiv

  8. [2020]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. 9 LLM Data Selection and Utilization via Dynamic Bi-level Optimization Advances in neural information pro...

    1901

  9. [2022]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361,

    2001 arXiv

  10. [2023]

    H., Peng, J., Zhuang, X., Zhang, C., Wu, L., Qiu, J., Zhang, W., Yuan, B., et al

    Bai, T., Yang, L., Wong, Z. H., Peng, J., Zhuang, X., Zhang, C., Wu, L., Qiu, J., Zhang, W., Yuan, B., et al. Multi-agent collaborative data selection for efficient llm pretraining. arXiv preprint arXiv:2410.08102,

    arXiv

  11. [2024]

    L., and Paul, M

    Ankner, Z., Blakeney, C., Sreenivasan, K., Marion, M., Leavitt, M. L., and Paul, M. Perplexed by perplexity: Perplexity-based pruning with small reference models. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models,

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.