Pith. sign in

REVIEW 5 cited by

Call for Papers -- The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2301.11796 v1 pith:RUPCTBMR submitted 2023-01-27 cs.CL

Alex Warstadt , Leshem Choshen , Aaron Mueller , Adina Williams , Ethan Wilcox , Chengxu Zhuang This is my paper · ORCID
classification cs.CL
keywords datacorpuslanguagepretrainingapproachesbabylmcallchallenge
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We present the call for papers for the BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus. This shared task is intended for participants with an interest in small scale language modeling, human language acquisition, low-resource NLP, and cognitive modeling. In partnership with CoNLL and CMCL, we provide a platform for approaches to pretraining with a limited-size corpus sourced from data inspired by the input to children. The task has three tracks, two of which restrict the training data to pre-released datasets of 10M and 100M words and are dedicated to explorations of approaches such as architectural variations, self-supervised objectives, or curriculum learning. The final track only restricts the amount of text used, allowing innovation in the choice of the data, its domain, and even its modality (i.e., data from sources other than text is welcome). We will release a shared evaluation pipeline which scores models on a variety of benchmarks and tasks, including targeted syntactic evaluations and natural language understanding.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 16 citations worldwide. Full citation record

  1. Influence-driven Curriculum Learning for Pre-training on Limited Data

    cs.CL 2025-08 unverdicted novelty 6.0 of 10

    Sorting pre-training examples by training-data influence instead of human-judged difficulty reportedly gives over 10 percentage point benchmark gains over random order in limited-data language model pre-training.

  2. Representing Speech Through Autoregressive Prediction of Cochlear Tokens

    cs.CL 2025-08 conditional novelty 6.0 of 10

    Autoregressive prediction over discrete cochlear tokens yields a speech representation that beats prior self-supervised models on lexical-semantic similarity and is competitive on SUPERB tasks.

  3. Basic Reading Distillation

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A 564M-parameter model trained on teacher-generated NER and QA annotations over generic text matches or beats a 13B teacher on some benchmarks, though the comparison is not always clean.

  4. Spontaneous Speech Variables for Evaluating LLMs Cognitive Plausibility

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Language models fine-tuned to predict speech reductions and prosodic prominences from text perform above random baselines, and models pretrained on conversational data outperform those pretrained on written data in En...

  5. Masked Diffusion Language Models with Frequency-Informed Training

    cs.CL 2025-09 conditional novelty 4.0 of 10

    Masked diffusion language models trained on 100M words match a hybrid GPT-BERT baseline on BabyLM tests, with a rare-word-focused masking variant.

Pith tools