REVIEW 6 cited by
How predictable is language model benchmark performance?
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We investigate large language model performance across five orders of magnitude of compute scaling in eleven recent model architectures. We show that average benchmark performance, aggregating over many individual tasks and evaluations as in the commonly-used BIG-Bench dataset, is decently predictable as a function of training compute scale. Specifically, when extrapolating BIG-Bench Hard performance across one order of magnitude in compute, we observe average absolute errors of 6 percentage points (pp). By contrast, extrapolation for individual BIG-Bench tasks across an order of magnitude in compute yields higher average errors of 18pp. Nonetheless, individual task performance remains significantly more predictable than chance. Overall, our work suggests compute scaling provides a promising basis to forecast AI capabilities in diverse benchmarks, though predicting performance in specific tasks poses challenges.
Forward citations
Cited by 6 Pith papers
-
EdgeBench: Unveiling Scaling Laws of Learning from Real-World Environments
Across ~38,000 hours on 134 ultra-long real-world tasks, aggregate agent performance follows a log-sigmoid of interaction time, and measured learning speed doubles about every three months.
-
Inference Compute-Optimal Video Vision Language Models
For video vision-language models under a fixed inference compute budget, the optimal setup scales language model size, frame count, and tokens per frame jointly, and shifts toward smaller LMs with more frames and toke...
-
A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Search
A skill-graph random-walk model gives closed-form accuracy-versus-compute formulas for four reasoning strategies and connects them to training scaling.
-
How Benchmark Prediction from Fewer Data Misses the Mark
Benchmark prediction methods mostly work by interpolation among similar models and fail on better, unfamiliar models, where random sampling with an AIPW-style correction is the only consistent improvement.
-
Meek Models Shall Inherit the Earth
Under fixed-distribution neural scaling laws, the capability gap between state-of-the-art and low-compute AI models shrinks over time toward zero.
-
Capability Salience Vector: Fine-grained Alignment of Loss and Capabilities for Downstream Task Scaling Law
A fitted token-level loss weighting, optimized against known model accuracies, predicts held-out downstream task performance more accurately than mean validation loss on five of six benchmarks.
Discussion (0). Continue with ORCID to comment.