REVIEW 7 cited by
Improving Pretraining Data Using Perplexity Correlations
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Quality pretraining data is often seen as the key to high-performance language models. However, progress in understanding pretraining data has been slow due to the costly pretraining runs required for data selection experiments. We present a framework that avoids these costs and selects high-quality pretraining data without any LLM training of our own. Our work is based on a simple observation: LLM losses on many pretraining texts are correlated with downstream benchmark performance, and selecting high-correlation documents is an effective pretraining data selection method. We build a new statistical framework for data selection centered around estimates of perplexity-benchmark correlations and perform data selection using a sample of 90 LLMs taken from the Open LLM Leaderboard on texts from tens of thousands of web domains. In controlled pretraining experiments at the 160M parameter scale on 8 benchmarks, our approach outperforms DSIR on every benchmark, while matching the best data selector found in DataComp-LM, a hand-engineered bigram classifier. We have now also updated this paper to include results from preregistered experiments with new pretraining data on an aggregation of 22 benchmarks up to the 1.4B scale, showing increasing improvements of our method over others with more scale. A pip package with full documentation can be found here: https://github.com/TristanThrush/perplexity-correlations.
Forward citations
Cited by 7 Pith papers
-
Predicting Emergent Capabilities by Finetuning
Finetuning small models shifts the point where capability emerges, and extrapolating this shift to the low-data limit predicts few-shot emergence up to about 4x the compute in advance.
-
Next-Latent Prediction Transformers Learn Compact World Models
NextLat augments next-token prediction with latent next-state prediction, theoretically converging latents to belief states and showing empirical gains in world modeling, reasoning, planning, and faster inference via ...
-
Language Models Improve When Pretraining Data Matches Target Tasks
Ranking pretraining documents by similarity to benchmark training examples (BETR) yields consistent benchmark gains and a 2.1x compute multiplier over DCLM-Baseline.
-
Energy-Based Transformers are Scalable Learners and Thinkers
Energy-Based Transformers learn to predict by gradient-descent minimization of a learned energy function, and the paper reports faster pretraining scaling and inference-time thinking gains over Transformer++ and Diffu...
-
Evaluating Sample Utility for Efficient Data Selection by Mimicking Model Weights
A gradient alignment score against a reference model's weights selects and reweights training samples, improving data efficiency in image classification and CLIP pretraining.
-
Universal Model Routing for Efficient LLM Inference
UniRoute represents each language model by its error rates on a few prompt clusters, letting a router choose among models it has never seen during training.
-
Optimizing Pretraining Data Mixtures with LLM-Estimated Utility
Simple size-diversity heuristics outperform learned data mixing, while an LLM-based utility estimator (MEDU) matches expensive ablations at roughly 200x lower compute.
Discussion (0). Continue with ORCID to comment.