Pith. sign in

REVIEW 4 major objections 3 minor 52 references

Training language models on examples sorted by their training data influence outperforms random-order training by more than 10 percentage points on benchmarks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Sorting pre-training examples by training-data influence instead of human-judged difficulty reportedly gives over 10 percentage point benchmark gains over random order in limited-data language model pre-training.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Influence-based curriculum ordering for LM pretraining is a fresh idea, but the headline gain is unverified without variance or reference-set details, and the corrupted full text blocks any check. the 4 major comments →

arxiv 2508.15475 v2 pith:KKUPXHBT submitted 2025-08-21 cs.CL cs.LG

Influence-driven Curriculum Learning for Pre-training on Limited Data

Loris Schoenegger , Lukas Thoma , Terra Blevins , Benjamin Roth This is my paper
classification cs.CL cs.LG
keywords curriculum learninglanguage model pre-trainingtraining data influencedata orderinglimited datamodel-centric difficultybenchmark evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that curriculum learning can help language-model pre-training if difficulty is defined from the model's own behavior rather than from human intuition about text complexity. The authors sort training examples by their training data influence—an estimate of how much each example changes the model's output—and train on that order. In a limited-data setting, models trained on these influence-sorted curricula outperform models trained on the same data in random order by over 10 percentage points on benchmarks. If correct, this means the earlier limited success of curriculum learning for pre-training came from using the wrong notion of difficulty, not from curriculum learning itself.

Core claim

The paper's central claim is that curriculum learning becomes competitive for language-model pre-training when difficulty is measured by training data influence rather than by human-centered metrics. Training data influence is a score estimating how much an individual training example affects the model's output, and the paper uses it as a model-centric difficulty signal to order the training corpus. Models trained on curricula sorted by this influence score outperform models trained on the same data in random order by more than 10 percentage points on benchmarks. The intended conclusion is that curriculum learning is beneficial for language-model pre-training as long as the difficulty metric

What carries the argument

The central object is the training data influence score, defined as an estimate of the effect of an individual training example on the model's output. It carries the argument by replacing human-centered difficulty metrics with a model-centric ordering: examples are ranked by how strongly the model's behavior would change if that example were included in training. The paper treats this ordering as a curriculum and shows that training on it beats random order in a limited-data pre-training setting.

Load-bearing premise

Influence scores, estimated from a model checkpoint, measure per-example difficulty as the model actually experiences it during training, so that sorting by them and retraining improves learning.

What would settle it

Take the same corpus and compute influence scores from two very different sources—an untrained randomly initialized model and a fully trained model. If influence-sorted curricula from both sources produce the same >10-point gain over random order, difficulty is robust; if only one does, the effect depends on the checkpoint, not on intrinsic difficulty. Similarly, evaluate the trained models on benchmarks outside the reference set used for influence; if the gain disappears there, the curriculum is tuning toward the evaluation set rather than teaching harder material.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Curriculum learning can help language-model pre-training when difficulty is defined from the model's own influence estimates, not from human judgments.
  • Reordering an existing fixed corpus can produce more than 10 percentage points of benchmark improvement over random order, with no extra data or compute.
  • The result provides a practical, automatic difficulty signal for pre-training data curation.
  • Human-centered difficulty metrics are likely not the fundamental obstacle; alignment with observed training dynamics is what matters.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If influence scores mostly track similarity to the benchmark or reference set used to compute them, part of the gain may be evaluation-set alignment rather than intrinsic difficulty; a test on out-of-distribution benchmarks would separate these.
  • Influence computed once from an early checkpoint may be stale; periodically recomputing scores could either increase the gain or show it saturates.
  • The benefit may be specific to limited-data regimes, where order strongly shapes which patterns are learned; at web-scale data, random order may already supply enough curriculum effect.
  • The influence score could also be capturing redundancy or noise rather than difficulty; ablating duplicated examples or controlling for example length would clarify the mechanism.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper proposes replacing human-crafted difficulty metrics in curriculum learning with a 'training data influence' score, defined as the effect of individual training examples on the model's output, and claims that pre-training language models on data sorted by this score yields benchmark improvements of more than 10 percentage points over random-order training in a limited-data setting. The abstract frames this as evidence that curriculum learning is beneficial when difficulty is measured in a model-centric way. However, the submitted full text is corrupted and largely unreadable (mojibake), so the method, experiments, and tables cannot be inspected. From the readable portions, the paper does not define the influence estimator, the reference set against which influence is computed, the checkpoint used, the benchmark list, the number of seeds, or the random-order baseline protocol. The central quantitative claim is therefore unverifiable from the submitted manuscript.

Significance. The research question is timely: if ordering examples by influence genuinely improves limited-data LM pre-training, it would be a useful and practically relevant result for curriculum learning. The proposed direction—model-centric rather than human-centric difficulty—is plausible and worth investigating. However, the manuscript as submitted provides no verifiable evidence. The abstract's >10 percentage point claim is not backed by statistical detail, and the body's unreadable state prevents any assessment of the method. There are no machine-checked proofs, reproducible code, or parameter-free derivations visible in the readable portion. The significance of the result cannot be evaluated without a complete, readable manuscript.

major comments (4)
  1. [Abstract] The headline claim—'by over 10 percentage points in benchmarks'—is reported without any experimental protocol: no number of random seeds, no standard deviation or error bars, no benchmark list, and no description of the random-order baseline or its tuning budget. In limited-data LM pre-training, double-digit differences can be within seed variance. Without this information the claim is quantitatively unsupported.
  2. [Abstract] The 'training data influence' score is defined only as 'the effect of individual training examples on the model's output.' Such an effect is relative to a reference set, but neither the reference set nor the estimator nor the model checkpoint from which it is computed is specified. If the reference set overlaps the evaluation benchmarks, sorting by influence may be equivalent to aligning training data with the evaluation distribution; the observed gain would then be benchmark-fitting, not evidence about curriculum difficulty. This distinction is essential and must be addressed.
  3. [Full text (all sections)] The body of the manuscript is corrupted and unreadable (mojibake). Equations, method descriptions, experimental details, and result tables cannot be interpreted. I cannot verify the central derivation, the computation of influence, the training setup, or the benchmark results. This is a load-bearing deficiency: the paper's central claim is entirely unverifiable from the submitted file.
  4. [Full text / Method] The paper asserts, rather than demonstrates, that influence scores measure 'example difficulty as observed during model training.' The comparison to random order cannot distinguish a genuine curriculum-ordering benefit from an effect of influence-based data selection or reweighting, especially if influence is computed against a reference set that is not held out. A controlled experiment varying only the order, with a pre-specified influence estimator and a reference set disjoint from the evaluation benchmarks, is required to support the causal interpretation.
minor comments (3)
  1. [Abstract] The phrase 'in benchmarks' is vague; please name the benchmarks or refer to a specific table.
  2. [Full text] The submission's encoding must be fixed; the current file is not reviewable. Please ensure equations and tables render correctly.
  3. [Abstract] 'Confirming that curriculum learning is beneficial' is too strong for a single study; consider 'is consistent with' or 'provides evidence that.'

Circularity Check

0 steps flagged

No circularity identifiable: the readable abstract contains no derivation chain, and the full text is corrupted, so no specific reduction can be exhibited.

full rationale

The only readable portion of the manuscript is the abstract. It claims that sorting training examples by 'training data influence'—defined as a score estimating the effect of individual training examples on the model's output—produces curricula that outperform random-order training by over 10 percentage points. To establish circularity we would need to see the definition of the influence estimator, the checkpoint used to compute it, and the reference set against which the effect on the model's output is measured. The abstract provides none of these details, and the supplied full text is mojibake, so the paper's own equations and derivations cannot be inspected. No passage can be quoted in which a prediction reduces by construction to an input, no fitted parameter is renamed as a prediction, and no load-bearing self-citation or imported uniqueness theorem appears in the accessible text. The abstract's claim is an empirical outcome, not a derivation that is definitionally equivalent to its inputs. The possibility that the influence scores were computed against the evaluation benchmarks would be a contamination or selection concern, not a demonstrated circularity, and there is no textual evidence for it. Consistent with the rule that circularity must be exhibited by quoting the paper and showing a specific reduction, the honest finding is no significant circularity. Score 0.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

The abstract provides no quantitative details, so no free parameters can be audited; the influence estimator, curriculum schedule, and any thresholds are undefined in the abstract. No new entities are introduced; the influence score is an existing construct applied to a new purpose. The axioms listed are the domain assumptions any empirical validation of the central claim would require.

axioms (3)
  • domain assumption The training data influence score, as implemented, is a valid proxy for per-example difficulty during pre-training for the model being trained.
    Central to the method; the abstract asserts the correspondence ('difficulty as observed during model training') but gives no estimator definition or validation that influence tracks difficulty.
  • domain assumption The evaluation benchmarks and the random-order baseline provide a fair comparison, and the reported >10 percentage point gain is stable across seeds.
    The headline comparison is meaningless if the baseline is undertuned or the gain is within seed variance; neither variance nor baseline details appear in the abstract.
  • domain assumption The limited-data setting studied generalizes to LM pre-training more broadly.
    The abstract concludes curriculum learning 'is beneficial for language model pre-training' from a limited-data configuration; scaling behavior is not reported in the abstract.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Influence-driven Curriculum Learning for Pre-training on Limited Data." pith.science (2026). https://pith.science/paper/KKUPXHBT

@misc{pith2026250815475,
  author       = {Pith},
  title        = {Pith review of: Influence-driven Curriculum Learning for Pre-training on Limited Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KKUPXHBT}},
  note         = {Machine review of arXiv:2508.15475}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Curriculum learning, a training technique where data is presented to the model in order of example difficulty (e.g., from simpler to more complex documents), has shown limited success for pre-training language models. In this work, we investigate whether curriculum learning becomes competitive if we replace conventional human-centered difficulty metrics with one that more closely corresponds to example difficulty as observed during model training. Specifically, we experiment with sorting training examples by their \textit{training data influence}, a score which estimates the effect of individual training examples on the model's output. Models trained on our curricula are able to outperform ones trained in random order by over 10 percentage points in benchmarks, confirming that curriculum learning is beneficial for language model pre-training, as long as a more model-centric notion of difficulty is adopted.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

52 extracted references · 24 canonical work pages · 3 internal anchors

  1. [1]

    Ahmed Abdelali, Francisco Guzman, Hassan Sajjad, and Stephan Vogel. 2014. https://aclanthology.org/L14-1675/ The AMARA Corpus : Building Parallel Language Resources for the Educational Domain . In Proceedings of the Ninth International Conference on Language Resources and Evaluation ( LREC `14) , pages 1856--1862, Reykjavik, Iceland. European Language Res...

    work page 2014

  2. [2]

    Robert Baldock, Hartmut Maennel, and Behnam Neyshabur. 2021. https://proceedings.neurips.cc/paper/2021/hash/5a4b25aaed25c2ee1b74de72dc03c14e-Abstract.html Deep Learning Through the Lens of Example Difficulty . In Advances in Neural Information Processing Systems , volume 34, pages 10876--10889. Curran Associates, Inc

    work page 2021

  3. [3]

    Irina Bejan, Artem Sokolov, and Katja Filippova. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.625 Make Every Example Count : On the Stability and Utility of Self - Influence for Learning from Noisy NLP Datasets . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 10107--10121, Singapore. Association for ...

    doi:10.18653/v1/2023.emnlp-main.625 2023

  4. [4]

    Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. https://doi.org/10.1145/1553374.1553380 Curriculum learning . In Proceedings of the 26th Annual International Conference on Machine Learning , ICML '09, pages 41--48, New York, NY, USA. Association for Computing Machinery

    arXiv 2009

  5. [5]

    Eden Bensaid, Mauro Martino, Benjamin Hoover, and Hendrik Strobelt. 2021. https://doi.org/10.48550/arXiv.2108.04324 FairyTailor : A Multimodal Generative Framework for Storytelling . arXiv preprint. ArXiv:2108.04324 [cs]

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2108.04324 2021

  6. [6]

    Nasim Borazjanizadeh. 2023. https://doi.org/10.18653/v1/2023.conll-babylm.32 Optimizing GPT -2 Pretraining on BabyLM Corpus with Difficulty -based Sentence Reordering . In Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning , pages 356--365, Singapore. Association for Computational Linguistics

    work page doi:10.18653/v1/2023.conll-babylm.32 2023

  7. [7]

    Lucas Charpentier, Leshem Choshen, Ryan Cotterell, Mustafa Omer Gul, Michael Hu, Jaap Jumelet, Tal Linzen, Jing Liu, Aaron Mueller, Candace Ross, Raj Sanjay Shah, Alex Warstadt, Ethan Wilcox, and Adina Williams. 2025. https://doi.org/10.48550/arXiv.2502.10645 BabyLM Turns 3: Call for papers for the 2025 BabyLM workshop . arXiv preprint. ArXiv:2502.10645 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.10645 2025

  8. [8]

    Hu, Tal Linzen, Aaron Mueller, Candace Ross, Alex Warstadt, Ethan Wilcox, Adina Williams, and Chengxu Zhuang

    Leshem Choshen, Ryan Cotterell, Michael Y. Hu, Tal Linzen, Aaron Mueller, Candace Ross, Alex Warstadt, Ethan Wilcox, Adina Williams, and Chengxu Zhuang. 2024. https://doi.org/10.48550/arXiv.2404.06214 [ Call for Papers ] The 2nd BabyLM Challenge : Sample -efficient pretraining on a developmentally plausible corpus . arXiv preprint. ArXiv:2404.06214

    Pith/arXiv arXiv doi:10.48550/arxiv.2404.06214 2024

  9. [9]

    B. N. C. Consortium. 2007. https://llds.ling-phil.ox.ac.uk/llds/xmlui/handle/20.500.14106/2554 British National Corpus , XML edition . Accepted: 2018-07-27 Artwork Medium: Digital bitstream Interview Medium: Digital bitstream Publisher: University of Oxford

    work page 2007

  10. [10]

    Covington, , and Joe D

    Michael A. Covington, , and Joe D. McFall. 2010. https://doi.org/10.1080/09296171003643098 Cutting the Gordian Knot : The Moving - Average Type – Token Ratio ( MATTR ) . Journal of Quantitative Linguistics, 17(2):94--100. Publisher: Routledge \_eprint: https://doi.org/10.1080/09296171003643098

    doi:10.1080/09296171003643098 2010

  11. [11]

    Jeffrey L. Elman. 1993. https://doi.org/10.1016/0010-0277(93)90058-4 Learning and development in neural networks: the importance of starting small . Cognition, 48(1):71--99

    doi:10.1016/0010-0277(93)90058-4 1993

  12. [12]

    Martin Gerlach and Francesc Font-Clos. 2018. https://doi.org/10.48550/arXiv.1812.08092 A standardized Project Gutenberg corpus for statistical analysis of natural language and quantitative linguistics . arXiv preprint. ArXiv:1812.08092 [cs]

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1812.08092 2018

  13. [13]

    Mohammad Amin Ghanizadeh and Mohammad Javad Dousti. 2024. https://aclanthology.org/2024.conll-babylm.2/ Towards Data - Efficient Language Models : A Child - Inspired Approach to Language Learning . In The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning , pages 22--27, Miami, FL, USA. Association for Computational Linguistics

    work page 2024

  14. [14]

    Richards, Steven F

    Jill Gilkerson, Jeffrey A. Richards, Steven F. Warren, Judith K. Montgomery, Charles R. Greenwood, D. Kimbrough Oller, John H. L. Hansen, and Terrance D. Paul. 2017. https://doi.org/10.1044/2016_AJSLP-15-0169 Mapping the early language environment using all-day recordings and automated analysis . American Journal of Speech-Language Pathology, 26(2):248--265

    doi:10.1044/2016_ajslp-15-0169 2017

  15. [15]

    Guy Hacohen and Daphna Weinshall. 2019. https://doi.org/10.48550/arXiv.1904.03626 On The Power of Curriculum Learning in Training Deep Networks . arXiv preprint. ArXiv:1904.03626 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.1904.03626 2019

  16. [16]

    Zayd Hammoudeh and Daniel Lowd. 2022. https://doi.org/10.1145/3548606.3559335 Identifying a Training - Set Attack 's Target Using Renormalized Influence Estimation . In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security , CCS '22, pages 1367--1381, New York, NY, USA. Association for Computing Machinery

    work page arXiv 2022

  17. [17]

    Zayd Hammoudeh and Daniel Lowd. 2024. https://doi.org/10.1007/s10994-023-06495-7 Training data influence analysis and estimation: a survey . Machine Learning, 113(5):2351--2403

    doi:10.1007/s10994-023-06495-7 2024

  18. [18]

    Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2016. https://doi.org/10.48550/arXiv.1511.02301 The Goldilocks Principle : Reading Children 's Books with Explicit Memory Representations . arXiv preprint. ArXiv:1511.02301 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.1511.02301 2016

  19. [19]

    Valentin Hofmann, Leonie Weissweiler, David Mortensen, Hinrich Schütze, and Janet Pierrehumbert. 2024. https://doi.org/10.48550/arXiv.2411.07990 Derivational Morphology Reveals Analogical Generalization in Large Language Models . arXiv preprint. ArXiv:2411.07990 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2411.07990 2024

  20. [20]

    Hu, Aaron Mueller, Candace Ross, Adina Williams, Tal Linzen, Chengxu Zhuang, Ryan Cotterell, Leshem Choshen, Alex Warstadt, and Ethan Gotlieb Wilcox

    Michael Y. Hu, Aaron Mueller, Candace Ross, Adina Williams, Tal Linzen, Chengxu Zhuang, Ryan Cotterell, Leshem Choshen, Alex Warstadt, and Ethan Gotlieb Wilcox. 2024. https://doi.org/10.48550/arXiv.2412.05149 Findings of the Second BabyLM Challenge : Sample - Efficient Pretraining on Developmentally Plausible Corpora . arXiv preprint. ArXiv:2412.05149 [cs...

    Pith/arXiv arXiv doi:10.48550/arxiv.2412.05149 2024

  21. [21]

    Huebner, Elior Sulem, Fisher Cynthia, and Dan Roth

    Philip A. Huebner, Elior Sulem, Fisher Cynthia, and Dan Roth. 2021. https://doi.org/10.18653/v1/2021.conll-1.49 BabyBERTa : Learning More Grammar With Small - Scale Child - Directed Language . In Proceedings of the 25th Conference on Computational Natural Language Learning , pages 624--646, Online. Association for Computational Linguistics

    doi:10.18653/v1/2021.conll-1.49 2021

  22. [22]

    Ivanova, Aalok Sathe, Benjamin Lipkin, Unnathi Kumar, Setayesh Radkani, Thomas H

    Anna A. Ivanova, Aalok Sathe, Benjamin Lipkin, Unnathi Kumar, Setayesh Radkani, Thomas H. Clark, Carina Kauf, Jennifer Hu, R. T. Pramod, Gabriel Grand, Vivian Paulun, Maria Ryskina, Ekin Akyürek, Ethan Wilcox, Nafisa Rashid, Leshem Choshen, Roger Levy, Evelina Fedorenko, Joshua Tenenbaum, and Jacob Andreas. 2024. https://doi.org/10.48550/arXiv.2405.09605 ...

    Pith/arXiv arXiv doi:10.48550/arxiv.2405.09605 2024

  23. [23]

    Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C. Mozer. 2021. https://proceedings.mlr.press/v139/jiang21k.html Characterizing Structural Regularities of Labeled Data in Overparameterized Models . In Proceedings of the 38th International Conference on Machine Learning , pages 5034--5044. PMLR. ISSN: 2640-3498

    work page 2021

  24. [24]

    M. G. Kendall. 1945. https://doi.org/10.1093/biomet/33.3.239 THE TREATMENT OF TIES IN RANKING PROBLEMS . Biometrika, 33(3):239--251

    doi:10.1093/biomet/33.3.239 1945

  25. [25]

    Najoung Kim and Sebastian Schuster. 2023. https://doi.org/10.18653/v1/2023.acl-long.213 Entity Tracking in Language Models . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( Volume 1: Long Papers ) , pages 3835--3855, Toronto, Canada. Association for Computational Linguistics

    doi:10.18653/v1/2023.acl-long.213 2023

  26. [26]

    Kumar, Benjamin Packer, and Daphne Koller

    M. Kumar, Benjamin Packer, and Daphne Koller. 2010. https://papers.nips.cc/paper_files/paper/2010/hash/e57c6b956a6521b28495f2886ca0977a-Abstract.html Self- Paced Learning for Latent Variable Models . In Advances in Neural Information Processing Systems , volume 23. Curran Associates, Inc

    work page 2010

  27. [27]

    Conglong Li, Minjia Zhang, and Yuxiong He. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/hash/aac02401755a65904cf977a33136af4a-Abstract-Conference.html The Stability - Efficiency Dilemma : Investigating Sequence Length Warmup for Training GPT Models . Advances in Neural Information Processing Systems, 35:26736--26750

    work page 2022

  28. [28]

    Pierre Lison and Jörg Tiedemann. 2016. https://aclanthology.org/L16-1147/ OpenSubtitles2016 : Extracting Large Parallel Corpora from Movie and TV Subtitles . In Proceedings of the Tenth International Conference on Language Resources and Evaluation ( LREC `16) , pages 923--929, Portorož, Slovenia. European Language Resources Association (ELRA)

    work page 2016

  29. [29]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. https://doi.org/10.48550/arXiv.1907.11692 RoBERTa : A Robustly Optimized BERT Pretraining Approach . arXiv preprint. ArXiv:1907.11692 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.1907.11692 2019

  30. [30]

    Brian MacWhinney. 2014. https://doi.org/10.4324/9781315805641 The Childes Project , 0 edition. Psychology Press

    work page doi:10.4324/9781315805641 2014

  31. [31]

    Richard Diehl Martinez, Hope McGovern, Zebulon Goriely, Christopher Davis, Andrew Caines, Paula Buttery, and Lisa Beinborn. 2023. https://doi.org/10.18653/v1/2023.conll-babylm.10 CLIMB – Curriculum Learning for Infant -inspired Model Building . In Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning , pages...

    work page doi:10.18653/v1/2023.conll-babylm.10 2023

  32. [32]

    Maggie Mi. 2023. https://doi.org/10.18653/v1/2023.conll-babylm.23 Mmi01 at The BabyLM Challenge : Linguistically Motivated Curriculum Learning for Pretraining in Low - Resource Settings . In Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning , pages 269--278, Singapore. Association for Computational Linguistics

    doi:10.18653/v1/2023.conll-babylm.23 2023

  33. [33]

    Miyu Oba, Akari Haga, Akiyo Fukatsu, and Yohei Oseki. 2023. https://doi.org/10.18653/v1/2023.conll-babylm.25 BabyLM Challenge : Curriculum learning based on sentence complexity approximating language acquisition . In Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning , pages 290--297, Singapore. Associati...

    work page doi:10.18653/v1/2023.conll-babylm.25 2023

  34. [34]

    Morrison, and N

    Mattia Opper, J. Morrison, and N. Siddharth. 2023. https://doi.org/10.18653/v1/2023.conll-babylm.31 On the effect of curriculum learning with developmental data for grammar acquisition . In Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning , pages 346--355, Singapore. Association for Computational Linguistics

    work page doi:10.18653/v1/2023.conll-babylm.31 2023

  35. [35]

    Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. 2023. https://proceedings.mlr.press/v202/park23c.html TRAK : Attributing Model Behavior at Scale . In Proceedings of the 40th International Conference on Machine Learning , pages 27074--27113. PMLR. ISSN: 2640-3498

    work page 2023

  36. [36]

    Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnabas Poczos, and Tom Mitchell. 2019. https://doi.org/10.18653/v1/N19-1119 Competence-based Curriculum Learning for Neural Machine Translation . In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics : Human Language Technologies ,...

    doi:10.18653/v1/n19-1119 2019

  37. [37]

    Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. 2020. https://proceedings.neurips.cc/paper/2020/hash/e6385d39ec9394f2f3a354d9d2b88eec-Abstract.html Estimating Training Data Influence by Tracing Gradient Descent . In Advances in Neural Information Processing Systems , volume 33, pages 19920--19930. Curran Associates, Inc

    work page 2020

  38. [38]

    Anastasiia Sedova, Lena Zellinger, and Benjamin Roth. 2023. https://doi.org/10.1007/978-3-031-43412-9_14 Learning with Noisy Labels by Adaptive Gradient - Based Outlier Removal . In Machine Learning and Knowledge Discovery in Databases : Research Track , pages 237--253. Springer, Cham. ISSN: 1611-3349

    work page doi:10.1007/978-3-031-43412-9_14 2023

  39. [39]

    less is more

    Valentin I. Spitkovsky, Hiyan Alshawi, and Dan Jurafsky. 2010. https://aclanthology.org/N10-1116.pdf From baby steps to leapfrog: How “less is more” in unsupervised dependency parsing . In Human Language Technologies : The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics , pages 751--759

    work page 2010

  40. [40]

    Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. https://aclanthology.org/J00-3003/ Dialogue act modeling for automatic tagging and recognition of conversational speech . Computational Linguistics, 26(3):339--374. Place: Cambridge, MA Pu...

    work page 2000

  41. [41]

    Lukas Thoma, Ivonne Weyers, Erion Çano, Stefan Schweter, Jutta L Mueller, and Benjamin Roth. 2023. https://doi.org/10.18653/v1/2023.conll-babylm.15 CogMemLM : Human - Like Memory Mechanisms Improve Performance and Cognitive Plausibility of LLMs . In Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning , pag...

    work page doi:10.18653/v1/2023.conll-babylm.15 2023

  42. [42]

    Inar Timiryasov and Jean-Loup Tastet. 2023. https://doi.org/10.18653/v1/2023.conll-babylm.24 Baby Llama : knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty . In Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning , pages 279--289, Singapore. Associat...

    doi:10.18653/v1/2023.conll-babylm.24 2023

  43. [43]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. https://doi.org/10.48550/arXiv.2302.13971 LLaMA : Open and Efficient Foundation Language Models . arXiv preprint. ArXiv...

    Pith/arXiv arXiv doi:10.48550/arxiv.2302.13971 2023

  44. [44]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. https://doi.org/10.48550/arXiv.1804.07461 GLUE : A Multi - Task Benchmark and Analysis Platform for Natural Language Understanding . arXiv preprint. ArXiv:1804.07461

    Pith/arXiv arXiv doi:10.48550/arxiv.1804.07461 2019

  45. [45]

    Alex Warstadt, Leshem Choshen, Aaron Mueller, Adina Williams, Ethan Wilcox, and Chengxu Zhuang. 2023 a . https://doi.org/10.48550/arXiv.2301.11796 Call for Papers -- The BabyLM Challenge : Sample -efficient pretraining on a developmentally plausible corpus . arXiv preprint. ArXiv:2301.11796

    Pith/arXiv arXiv doi:10.48550/arxiv.2301.11796 2023

  46. [46]

    Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell. 2023 b . https://doi.org/10.18653/v1/2023.conll-babylm.1 Findings of the BabyLM Challenge : Sample - Efficient Pretraining on Developmentally Plausible Corpora . In Proceedings of the B...

    doi:10.18653/v1/2023.conll-babylm.1 2023

  47. [47]

    Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. https://doi.org/10.1162/tacl_a_00321 BLiMP : The Benchmark of Linguistic Minimal Pairs for English . Transactions of the Association for Computational Linguistics, 8:377--392

    doi:10.1162/tacl_a_00321 2020

  48. [48]

    Xiaoxia Wu, Ethan Dyer, and Behnam Neyshabur. 2020. https://openreview.net/forum?id=tW4QEInpni When Do Curricula Work ? In International Conference on Learning Representations

    work page 2020

  49. [49]

    Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. 2024. https://doi.org/10.48550/arXiv.2402.04333 LESS : Selecting Influential Data for Targeted Instruction Tuning . arXiv preprint. ArXiv:2402.04333 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2402.04333 2024

  50. [50]

    Chih-Kuan Yeh, Ankur Taly, Mukund Sundararajan, Frederick Liu, and Pradeep Ravikumar. 2022. https://doi.org/10.48550/arXiv.2202.11844 First is Better Than Last for Language Data Influence . arXiv preprint. ArXiv:2202.11844 [cs]

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2202.11844 2022

  51. [51]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  52. [52]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.