REVIEW 17 cited by
Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Information about pretraining corpora used to train the current best-performing language models is seldom discussed: commercial models rarely detail their data, and even open models are often released without accompanying training data or recipes to reproduce them. As a result, it is challenging to conduct and advance scientific research on language modeling, such as understanding how training data impacts model capabilities and limitations. To facilitate scientific research on language model pretraining, we curate and release Dolma, a three-trillion-token English corpus, built from a diverse mixture of web content, scientific papers, code, public-domain books, social media, and encyclopedic materials. We extensively document Dolma, including its design principles, details about its construction, and a summary of its contents. We present analyses and experimental results on intermediate states of Dolma to share what we have learned about important data curation practices. Finally, we open-source our data curation toolkit to enable reproduction of our work as well as support further research in large-scale data curation.
Forward citations
Cited by 17 Pith papers
-
Domain-Aware Scaling Laws Uncover Data Synergy
Domain-aware scaling laws with fitted γ and σ synergy terms recover stable code-math interactions from observational LLM mixtures and correctly predict mixture rankings in controlled small-scale trainings.
-
Small edits, large models: How Wikipedia advocacy shapes LLM values
Wikipedia edits by animal welfare advocates measurably influence LLM outputs on animal welfare topics, shown via retrieval and gradient attribution plus fine-tuning experiments.
-
OLMoASR: Open Models and Data for Training Robust Speech Recognition Models
An open 1M-hour English speech dataset plus Whisper-architecture models trained on it match Whisper's word error rates on short and long-form benchmarks.
-
Moir: Let the Model Direct Its Own Story for Robust Cross-Domain Knowledge Editing
MOIR: estimating the preservation covariance from a model's own random-token generations reduces collapse of math/code capabilities in some knowledge-editing settings, but the claimed consistency is not supported by t...
-
From Context Shift to Stylistic Collapse: Why Training Objectives Matter More Than Scale
Stylistic collapse in LLMs is largely pre-alignment and scale-independent; strong entropy regularization reduces it far more than weak smoothing or RLHF.
-
LLM generation novelty through the lens of semantic similarity
An output is called novel when no semantically similar pretraining context can be retrieved; using this embedding-based measure, SmolLM models show longer-range reuse and instruction tuning increases novelty.
-
User Privacy and Large Language Models: An Analysis of Frontier Developers' Privacy Policies
All six leading U.S. AI chatbot developers, as of May 2025, appear to train their models on users' chat data by default, often without clear opt-out options.
-
BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity
A ReLU-routed MoE with chunk-level sparsity training objectives and custom kernels combining activation sparsity with speculative decoding achieves over 70% 8-token chunk sparsity and up to 3.67x end-side speedup.
-
FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language
An adaptive, per-language data filtering and deduplication pipeline produces multilingual LLM pre-training corpora that beat prior public datasets on 11 of 14 evaluated languages, and a 20TB, 1,868 language-script dat...
-
Thunder-LLM: Efficiently Adapting LLMs to Korean with Minimal Resources
A cost-effective recipe consisting of tokenizer extension, continual pretraining, FP8 training, and SFT/DPO post-training yields Korean-English bilingual 8B models with top Korean benchmark scores.
-
Essential-Web v1.0: 24T tokens of organized web data
A 24T-token web corpus with automatic document-level taxonomy labels enables competitive domain-specific datasets via simple filters.
-
BehaviorBox: Automated Discovery of Fine-Grained Performance Differences Between Language Models
A pipeline that combines contextual embeddings with two LMs' per-word probabilities and sparse autoencoders to automatically find interpretable slices where one model outperforms another.
-
Using Scaling Laws for Data Source Utility Estimation in Domain-Specific Pre-Training
Running multiple short annealing runs at different token scales can reveal per-source utility scaling curves that change data-source rankings compared with single point estimates.
-
A New Pair of GloVes
New 2024 GloVe embeddings, trained on updated Wikipedia, Gigaword, and a Dolma subset, match the 2014 models on structural benchmarks while improving NER on temporally recent and non-Western data.
-
FEVO: Financial Knowledge Expansion and Reasoning Evolution for Large Language Models
FEVO applies continued pretraining, supervised fine-tuning, and reinforcement learning to a 32B model, claiming state-of-the-art financial benchmark scores, but the evaluation appears contaminated by training data.
-
CCI4.0: A Bilingual Pretraining Dataset for Enhancing Reasoning in Large Language Models
A new 35 TB bilingual pretraining dataset with 4.5 billion chain-of-thought templates is described, but the evidence for its benefits is marginal, confounded, and contradicted by the paper's own tables.
-
From Data to Device: ELMOD An Efficient German-First 2.7B Language Model for Mobile Inference
A 2.7B German-first LLM trained cheaply on public data with language-specific quality filtering matches larger 7B models on German reasoning benchmarks and runs on-device.
Discussion (0). Sign in to comment.