Pith. sign in

REVIEW 2 cited by

Paloma: A Benchmark for Evaluating Language Model Fit

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2312.10523 v2 pith:2EY3EL4N submitted 2023-12-16 cs.CL cs.AIcs.LG

Ian Magnusson , Akshita Bhagia , Valentin Hofmann , Luca Soldaini , Ananya Harsh Jha , Oyvind Tafjord , Dustin Schwenk , Evan Pete Walsh
show 8 more authors
Yanai Elazar Kyle Lo Dirk Groeneveld Iz Beltagy Hannaneh Hajishirzi Noah A. Smith Kyle Richardson Jesse Dodge
This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords languagebenchmarkdatapalomaperplexitycodecommondomains
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Evaluations of language models (LMs) commonly report perplexity on monolithic data held out from training. Implicitly or explicitly, this data is composed of domains--varying distributions of language. We introduce Perplexity Analysis for Language Model Assessment (Paloma), a benchmark to measure LM fit to 546 English and code domains, instead of assuming perplexity on one distribution extrapolates to others. We include two new datasets of the top 100 subreddits (e.g., r/depression on Reddit) and programming languages (e.g., Java on GitHub), both sources common in contemporary LMs. With our benchmark, we release 6 baseline 1B LMs carefully controlled to provide fair comparisons about which pretraining corpus is best and code for others to apply those controls to their own experiments. Our case studies demonstrate how the fine-grained results from Paloma surface findings such as that models pretrained without data beyond Common Crawl exhibit anomalous gaps in LM fit to many domains or that loss is dominated by the most frequently occurring strings in the vocabulary.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Training Dynamics Underlying Language Model Scaling Laws: Loss Deceleration and Zero-Sum Learning

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Loss deceleration, a piecewise-linear break in log-log loss curves, is attributed to zero-sum learning where per-example gradients oppose one another, and scaling helps by mitigating it.

  2. BehaviorBox: Automated Discovery of Fine-Grained Performance Differences Between Language Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A pipeline that combines contextual embeddings with two LMs' per-word probabilities and sparse autoencoders to automatically find interpretable slices where one model outperforms another.

Pith tools