Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A benchmark's signal-to-noise ratio, measured from small models, predicts whether its rankings survive scaling to larger models and how accurate scaling-law forecasts will be.

desk verdict A serious, data-rich paper on benchmark SNR for LLM evaluation; the decision-accuracy result is solid, but the noise–scaling-law-error link is likely inflated by a normalization artifact and needs out-of-sample fixing. read the letter →

arxiv 2508.13144 v1 pith:M5BXKIC5 submitted 2025-08-18 cs.CL cs.LG

David Heineman , Valentin Hofmann , Ian Magnusson , Yuling Gu , Noah A. Smith , Hannaneh Hajishirzi , Kyle Lo , Jesse Dodge This is my paper · ORCID
classification cs.CLcs.LG
keywords signal-to-noiseratiolanguagemodelevaluationscalinglawsdecisionaccuracybenchmarkreliabilitycheckpointaveragingbits-per-bytenoisereduction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that evaluation benchmarks have measurable intrinsic properties—signal, how spread out model scores are, and noise, how much a single model's score wobbles between training checkpoints—and that their ratio predicts whether small-scale experiments will transfer to large models. If correct, benchmark builders can diagnose and repair evaluation suites cheaply, before spending large compute on risky decisions. The claim is supported by evaluations of hundreds of open-weight models on thirty benchmarks, and by three interventions that directly target signal or noise: filtering noisy subtasks, averaging checkpoints, and scoring by bits-per-byte instead of accuracy. The value of the framework is that both signal and noise can be measured at small scale, without waiting for expensive large-model runs.

What carries the argument

The signal-to-noise ratio is defined as $$\text{SNR}(B) = \frac{\text{Rel. Dispersion(final checkpoints)}}{\text{Rel. Std.}(\text{final } n \text{ checkpoints)}},$$ where relative dispersion is the maximum difference between any two small models' final scores divided by the mean score, and relative noise is the standard deviation of a single model's final checkpoints divided by its mean. This ratio is the load-bearing quantity: neither signal nor noise alone correlates with decision accuracy, but their ratio does. It is cheap to compute because it uses only final checkpoints already produced during development, and it is the quantity that all three interventions are designed to improve.

What would settle it

One could settle this by taking a new set of tasks outside the validated set, training 1B models with varied random seeds and data orders, and comparing seed noise and data-order noise against checkpoint-to-checkpoint noise; a task with large seed noise but small checkpoint noise would disprove the claim that final-checkpoint wobble captures modeling noise, and would break the argument that reducing checkpoint noise improves scaling-law prediction.

Watch

Extended reading notes

Core claim

The central discovery is a two-number definition of benchmark quality: signal is the relative dispersion of final-checkpoint scores across a population of small models trained under a similar compute budget, and noise is the relative standard deviation of the final checkpoints of a single training run. The paper shows that their ratio, the signal-to-noise ratio, is strongly correlated with decision accuracy—whether ranking small 60M-to-750M models predicts the ranking of 1B models—with a correlation of 0.79 across the OLMES benchmarks. It further shows that the noise of the large prediction target correlates with scaling-law prediction error, with a correlation of 0.65 across 30 tasks. These correlations motivate three interventions, each of which improves both decision accuracy and scaling-law error: ranking subtasks by signal-to-noise ratio to filter noisy subtasks, averaging or smoothing checkpoint scores to reduce noise, and replacing discontinuous metrics like accuracy with bits-per-byte to increase signal.

Load-bearing premise

The load-bearing premise is that checkpoint-to-checkpoint wobble in the final steps of a single training run is a faithful stand-in for all modeling noise; if that proxy fails on tasks beyond the small set used to validate it, the claimed correlations and the interventions that target checkpoint noise do not follow.

Editorial extensions

If this is right

  • Benchmarks with higher signal-to-noise ratio at small scale will rank training choices at large scale more reliably, so developers can select evaluation suites by measuring SNR on small models they already train.
  • Lower-noise benchmarks should yield tighter scaling-law predictions, and a task's checkpoint noise around the prediction target can serve as a sanity-check lower bound on the minimum scaling-law error.
  • Filtering subtasks by SNR can produce smaller but more reliable evaluation sets—the paper reports MMLU subsets with 16 subtasks and AutoBencher subsets with 6 subtasks that outperform the full suites on decision accuracy—so bigger benchmarks are not necessarily better for development decisions.
  • Averaging the final checkpoints of small prediction models and target models improves decision accuracy and scaling-law prediction error, and the same smoothing helps when choosing a model by early stopping.
  • Switching from accuracy, exact match, or pass@1 to bits-per-byte raises SNR on most benchmarks, improving decision accuracy on 90% of benchmarks and reducing scaling-law prediction error on 73.3% of them.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the noise-to-scaling-law-error link is causal rather than merely correlational, then measuring checkpoint noise during training could guide stopping criteria: a practitioner could train until the target's noise drops below the tolerated prediction error.
  • The same signal-to-noise machinery could serve as a routine quality metric for newly released benchmarks, computed once on a standard population of open-weight models, giving developers a cheap way to compare evaluation suites before adopting them.
  • A natural extension is instance-level rather than subtask-level SNR: greedily adding individual instances sorted by marginal SNR could yield much smaller test sets than subtask filtering, and would test whether the MMLU and AutoBencher subset results generalize.
  • The framework makes a testable prediction for debates about emergent abilities: tasks where small models perform at chance will have very low SNR at small scale, and their SNR should rise across model sizes exactly when small-to-large rankings become reliable.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript proposes a framework for characterizing evaluation benchmarks by two quantities: signal (relative dispersion of scores across a population of small models) and noise (relative standard deviation of a single model's final checkpoints). The signal-to-noise ratio (SNR) is shown to correlate with decision accuracy for small-to-large model ranking (Fig. 2, R = 0.791), and noise is shown to correlate with scaling-law prediction error (Fig. 3, R = 0.653, R² = 0.426). The authors then introduce three interventions motivated by this framework: filtering subtasks by SNR, averaging final checkpoints to reduce noise, and replacing primary task metrics with bits-per-byte. The experiments use 30 benchmarks and a large collection of open-weight models, and the authors release a dataset of about 900K evaluation results.

Significance. If the proposed framework holds, it offers a cheap and actionable diagnostic for benchmark construction and selection: compute SNR at small scale and use it to anticipate whether small experiments will transfer to larger models. The paper is unusually transparent about its methodology, including an appendix comparing 20 signal measures and an appendix validating checkpoint-to-checkpoint noise against seed noise and data-order noise. The public release of evaluation results, trained models, and intermediate checkpoints is a substantial reproducibility contribution. The central risk is that the second main claim, that lower noise causes lower scaling-law prediction error, is supported by a cross-task correlation whose two variables are both normalized by score magnitude, so the correlation may partly reflect task difficulty rather than a genuine predictive relation. The intervention results in Sections 5.1 and 5.3 are also evaluated in-sample or under the same normalization, so their interpretation depends on resolving this confound.

major comments (4)
  1. [§4.2, Eq. (4), Fig. 3] The reported R² = 0.426 between noise and scaling-law prediction error is potentially inflated by shared score-magnitude normalization. Noise is defined as σ/μ and prediction error as |predicted − true|/|true|; for hard tasks with low 13B scores, both denominators are small, so a task-difficulty gradient can produce a positive correlation even if checkpoint noise is causally unrelated to extrapolation error. The paper should report partial correlations controlling for mean score, absolute (unnormalized) errors, and within-task analyses across multiple prediction targets. The bits-per-byte results in §5.3 are subject to the same concern, because switching from floor-saturated accuracy to bits-per-byte changes both relative quantities simultaneously. Without these controls, the claim that benchmarks with less noise have lower scaling-law prediction error is not cleanly separated from the claim that hard tasks are hard to predict.
  2. [§3.2, App. A.4, Table 3] The signal measure was selected from 20 candidates as the one maximizing correlation with decision accuracy on the same OLMES benchmarks used to report Fig. 2, so the headline R² = 0.626 for SNR versus decision accuracy is an in-sample fit. Many alternatives in Table 3 have nearly identical R², which mitigates the concern, but the reported number remains optimistically biased. The authors should provide a holdout evaluation (for example, splitting benchmarks by category or using the non-OLMES tasks) or apply a selection-bias correction before presenting this correlation as the central evidence for the framework.
  3. [§5.1, Fig. 4] The subtask-filtering intervention is evaluated in-sample. Subtask SNR is computed from the 1B DataDecide models, and the same 150M-to-1B DataDecide ranking is then used to measure the improvement in decision accuracy, so the greedy ordering can exploit idiosyncrasies of the evaluation population. The random-shuffle baseline does not cure this. The authors should report a holdout evaluation, such as using 7B or 13B rankings from the OLMo 2 family or external open-weight models, before claiming that SNR-based filtering improves multi-task evaluations.
  4. [§3.1, App. A.3.1, Fig. 7] The choice of checkpoint-to-checkpoint noise as the representative modeling-noise measure is validated on only 7 OLMES tasks with 20 1B models (R² between 0.82 and 0.95). The framework is then applied to math and code tasks and to 13B and 32B targets, where the transfer of this proxy is an assumption. The authors should either provide a small-scale check for generative math/code tasks or larger models, or explicitly restrict the scope of the noise-reduction interventions to settings where the proxy has been validated.
minor comments (5)
  1. [Throughout] There are several typos and inconsistent spellings: 'wether' in the Figure 2 caption, 'influce' in §5.1, 'illustraed' in §4.2, 'recepies' in App. A.5.1, 'dimishing' in App. B.2, 'A veraging' in the §5.2 heading, and inconsistent capitalization of 'AutoBencher' versus 'Autobencher'.
  2. [Abstract vs. §2] The abstract states that 375 open-weight models are used, while Section 2 and App. A.5.1 report 465 models; the relationship between these numbers should be clarified.
  3. [§3.3 and §4.2] The noise definition in Eq. (2) leaves the number of checkpoints n unspecified; the main text uses n = 5 for the small models and n = 30 for the 13B target. The authors should state the chosen n values at the point of definition and note that the noise estimates are not directly comparable across different n.
  4. [§5.3] The definition of bits-per-byte says it is the negative log likelihood of the correct answer divided by the number of UTF-8 bytes, but this is dimensionally not 'bits' unless the logarithm base is specified as 2; please clarify the base and the exact byte-counting convention.
  5. [App. A.3.2] The chi-squared guidance for selecting n relies on the assumption that checkpoint-to-checkpoint scores are independent and normally distributed. This assumption should be stated in the main text alongside the recommendation of n = 20, since checkpoint scores within a single run are serially correlated in practice.

Circularity Check

1 steps flagged · score 5.0 of 10

The central SNR-decision-accuracy correlation is partly in-sample: the signal statistic was chosen from 20 candidates to maximize that same correlation; the interventions provide independent support, so circularity is moderate.

  1. fitted input called prediction [Section 3.2, Appendix A.4 / Table 3; reported as evidence in Section 4.1 / Figure 2]
    "We also considered 20 different measures of spread, including variance, mean pairwise distance, Gini coefficient, etc., in Appendix A.4. In the following section we introduce signal-to-noise ratio, and find that this definition of signal leads to signal-to-noise ratio with the highest correlation with decision accuracy. ... Table 3 caption: 'We use the measure which is most predictive of decision accuracy as our measure of signal.'"

    The definition of 'signal' is selected, among 20 candidate spread measures, for maximizing the very SNR-decision-accuracy correlation that Section 4.1 then reports as evidence (R=0.791, R2=0.626) on the same OLMES tasks. This is an in-sample maximum over a family of statistics, so the headline correlation is partly a fitted value rather than an out-of-sample prediction; by construction at least one of the 20 candidates will appear predictive even if none is. The interventions in Section 5 were not used in this selection and supply independent corroboration, but the Section 4.1 correlation itself is not independently estimated.

full rationale

Most of the paper's derivation is non-circular: SNR, decision accuracy, and scaling-law error are measured separately on overlapping but different evaluations; noise is validated against seed and data-order noise in Appendix A.3.1; and the three interventions are genuine out-of-sample tests of the framework. The scaling-law-error correlation in Figure 3 (R2=0.426) is not definitional, although both noise and error are normalized by task score magnitude, leaving a plausible task-difficulty confound; the paper itself labels the lower-bound interpretation a conjecture. The subtask-filtering intervention selects subtasks by SNR and re-evaluates on the same 25 DataDecide models and benchmark evaluations, so selection noise is shared, but the greedy SNR ranking is not fit to decision accuracy, making this a limitation rather than circularity. The one concrete circular step is the choice of the signal measure: the paper explicitly picks the spread statistic that maximizes the correlation later reported as evidence, using the same OLMES tasks and the same decision-accuracy targets as Section 4.1, so the headline R2 is partly in-sample. The interventions keep the central claim from being fully forced, so the paper does not reduce entirely to its own definitions.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The framework introduces two new statistics (signal and noise) but no new physical or model entities. The central claims rest on several domain assumptions: the checkpoint noise proxy, the representativeness of the 25 pretraining corpora, the validity of decision accuracy as a gold standard, the unproven noise-error bound, and the preservation of capability ordering by bits-per-byte. The only hand-chosen numeric input is the checkpoint window size n.

free parameters (1)
  • n_checkpoints (noise window) = 5 for decision accuracy noise, 30 for scaling law target noise
    The number of final checkpoints used to estimate checkpoint-to-checkpoint noise is chosen by hand. The paper justifies n=9 via a chi-squared argument for a loose bound and uses n=20 for a tight bound in practice, but the core experiments use n=5 and n=30, which affect SNR values and correlations.
assumptions (5)
  • domain assumption Checkpoint-to-checkpoint noise of a single training run estimates all modeling noise, including seed and data order noise.
    Validated on 7 OLMES tasks with 20 1B models (Figure 7, R² between 0.82 and 0.95); may not hold for all tasks or scales.
  • domain assumption Relative dispersion over 25 pretraining corpora measures a benchmark's ability to separate better from worse models.
    Assumes the 25 DataDecide recipes span the relevant model population; signal is measured on these models only.
  • domain assumption Decision accuracy on 25 corpora pairwise comparisons is a valid proxy for benchmark usefulness in model development.
    Borrowed from Magnusson et al. [38]; one of many possible operationalizations of small-to-large rank agreement.
  • ad hoc to paper Noise of the prediction target bounds minimum scaling law error.
    Stated as a conjecture in Section 4.2; supported only by cross-task correlation, not by a derivation.
  • domain assumption Bits-per-byte on correct continuations preserves the capability ordering measured by primary metrics.
    The paper does not report correlation between BPB and primary metric rankings, yet uses BPB to rank models in decision accuracy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation." pith.science (2026). https://pith.science/paper/M5BXKIC5

@misc{pith2026250813144,
  author       = {Pith},
  title        = {Pith review of: Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M5BXKIC5}},
  note         = {Machine review of arXiv:2508.13144}
}
read the original abstract

Developing large language models is expensive and involves making decisions with small experiments, typically by evaluating on large, multi-task evaluation suites. In this work, we analyze specific properties which make a benchmark more reliable for such decisions, and interventions to design higher-quality evaluation benchmarks. We introduce two key metrics that show differences in current benchmarks: signal, a benchmark's ability to separate better models from worse models, and noise, a benchmark's sensitivity to random variability between training steps. We demonstrate that benchmarks with a better signal-to-noise ratio are more reliable when making decisions at small scale, and those with less noise have lower scaling law prediction error. These results suggest that improving signal or noise will lead to more useful benchmarks, so we introduce three interventions designed to directly affect signal or noise. For example, we propose that switching to a metric that has better signal and noise (e.g., perplexity rather than accuracy) leads to better reliability and improved scaling law error. We also find that filtering noisy subtasks, to improve an aggregate signal-to-noise ratio, leads to more reliable multi-task evaluations. We also find that averaging the output of a model's intermediate checkpoints to reduce noise leads to consistent improvements. We conclude by recommending that those creating new benchmarks, or selecting which existing benchmarks to use, aim for high signal and low noise. We use 30 benchmarks for these experiments, and 375 open-weight language models from 60M to 32B parameters, resulting in a new, publicly available dataset of 900K evaluation benchmark results, totaling 200M instances.

Figures

Figures reproduced from arXiv: 2508.13144 by the authors.

Figure 1
Figure 1. Training curves for the 25 pretraining corpora in DataDecide [ [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Signal, noise, and signal-to-noise ratio (x-axis) vs. decision accuracy (y-axis), (see Section 2 for definitions). The signal alone (left) and noise alone (center) have low correlation with decision accuracy, while the signal-to-noise ratio (right) is correlated with decision accuracy. The signal-to-noise ratio gives us information about wether a benchmark is useful during development, as high decision accuracy (and… view at source ↗
Figure 3
Figure 3. Left: Correlation between the noise and scaling law prediction error (see Section 2 for definitions). We observe benchmarks with a lower noise around the scaling law target (x-axis) also exhibit lower error (y-axis). Right: Example of scaling law for one benchmark (SocialIQA), with examples on all benchmarks in [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: Evaluating an intervention designed to increase [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: When stopping a training run early, averaging the checkpoint-to-checkpoint noise improves [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Impact of changing benchmark metric to bits-per-byte (BPB) from the primary score (e.g., [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Top: 10 different training runs (1B-5×C scale) varying random seed initialization and data order, plotting ARC-C accuracy smoothed across a window of 20 checkpoints. Bottom: Total variation or the relative standard deviation (STD normalized by average performance; §3) …
Figure 8
Figure 8. Figure 8: As the benchmark’s signal-to-noise ratio increases (across histograms), decision accuracy (from 300M to 1B scale) not only increases but becomes more consistent. We test this by resampling decision accuracy for combinations among last 5 checkpoints of the small and lar…
Figure 9
Figure 9. Figure 9: Signal-to-noise ratio, decision accuracy, and scaling law prediction error for randomly sampled subsets of instances for 6 development benchmarks. A large sample size alone does not improve signal-to-noise ratio. For example, a 1000 question subset of ARC Easy has a hi…
Figure 10
Figure 10. Figure 10: Correlation between decision accuracy and variants of [PITH_FULL_IMAGE:figures/full_fig_p024_10.png]
Figure 11
Figure 11. Figure 11: Scaled-up version of the Figure 3 in §4.2 with labels on each task. [PITH_FULL_IMAGE:figures/full_fig_p025_11.png]
Figure 12
Figure 12. Figure 12: Performance of language models from 60M parameters to 32B parameters, which we [PITH_FULL_IMAGE:figures/full_fig_p026_12.png]
Figure 13
Figure 13. Figure 13: Final 30 checkpoints, each spaced 1000 training steps, for OLMo 2 1B, 7B, 13B and 32B [PITH_FULL_IMAGE:figures/full_fig_p027_13.png]
Figure 14
Figure 14. Figure 14: 1B-5xC training curves and final checkpoints for DataDecide models across tasks, sorted [PITH_FULL_IMAGE:figures/full_fig_p028_14.png]
Figure 15
Figure 15. Figure 15: Scaling law fits for all tasks using the OLMo 2 13B-5T prediction target. [PITH_FULL_IMAGE:figures/full_fig_p029_15.png]
Figure 16
Figure 16. Figure 16: Larger version of Figure 4, showing the names of each subtask, sorted by SNR from [PITH_FULL_IMAGE:figures/full_fig_p030_16.png]
Figure 17
Figure 17. Figure 17: Bits-per-byte vs. primary metric on the full suite of tasks shown in Figure 6. [PITH_FULL_IMAGE:figures/full_fig_p033_17.png]
Figure 18
Figure 18. Figure 18: When stopping a training run early, averaging the checkpoint-to-checkpoint noise improves [PITH_FULL_IMAGE:figures/full_fig_p034_18.png]
Figure 19
Figure 19. Figure 19: Visualization for the seed noise, data order noise and total variation for all OLMES tasks. 35 [PITH_FULL_IMAGE:figures/full_fig_p035_19.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Deployment Decision Reliability: A Generalizability-Theory Framework for Sizing Long-Horizon Agent Evaluations

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Across three enterprise agent benchmarks, agent main effects are under 3% of total score variance, so leaderboard order reflects task specialization rather than a general capability advantage.

Reference graph

Works this paper leans on

70 extracted references · 18 canonical work pages · cited by 1 Pith paper

  1. [1]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

    arXiv 2021

  2. [2]

    An empirical investigation of statistical significance in NLP

    Taylor Berg-Kirkpatrick, David Burkett, and Dan Klein. An empirical investigation of statistical significance in NLP. In Jun’ichi Tsujii, James Henderson, and Marius Pa¸ sca, editors,Proceed- ings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 995–1005, Jeju Island, Korea...

    work page 2012

  3. [3]

    Estab- lishing task scaling laws via compute-efficient model ladders

    Akshita Bhagia, Jiacheng Liu, Alexander Wettig, David Heineman, Oyvind Tafjord, Ananya Harsh Jha, Luca Soldaini, Noah A Smith, Dirk Groeneveld, Pang Wei Koh, et al. Estab- lishing task scaling laws via compute-efficient model ladders. arXiv preprint arXiv:2412.04403, 2024

    arXiv 2024

  4. [4]

    Piqa: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 7432–7439, 2020

    work page 2020

  5. [5]

    Larsen, Sean Owen, and Jonathan Frankle

    Cody Blakeney, Mansheej Paul, Brett W. Larsen, Sean Owen, and Jonathan Frankle. Does your data spark joy? performance gains from domain upsampling at the end of training, 2024. URL https://arxiv.org/abs/2406.03476

    arXiv 2024

  6. [6]

    Position: Don’t use the clt in llm evals with fewer than a few hundred datapoints

    Sam Bowyer, Laurence Aitchison, and Desi R Ivanova. Position: Don’t use the clt in llm evals with fewer than a few hundred datapoints. arXiv preprint arXiv:2503.01747, 2025

    arXiv 2025

  7. [7]

    With little power comes great responsibility

    Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, and Dan Jurafsky. With little power comes great responsibility. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors,Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 9263–9274, Online, November 2020. Association for C...

    doi:10.18653/v1/2020.emnlp-main.745 2020

  8. [8]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    arXiv 2021

Show all 70 references
  1. [9]

    A hitchhiker’s guide to scaling law estimation

    Leshem Choshen, Yang Zhang, and Jacob Andreas. A hitchhiker’s guide to scaling law estimation. arXiv preprint arXiv:2410.11840, 2024

    2024 arXiv

  2. [10]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computati...

    2019

  3. [11]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  4. [12]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  5. [13]

    Un- derspecification presents challenges for credibility in modern machine learning

    Alexander D’Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, et al. Un- derspecification presents challenges for credibility in modern machine learning. Journal of Machine Lear...

    2022

  6. [14]

    Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping

    Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305, 2020. 11

    2002 arXiv

  7. [15]

    Understanding emergent abilities of language models from the loss perspective

    Zhengxiao Du, Aohan Zeng, Yuxiao Dong, and Jie Tang. Understanding emergent abilities of language models from the loss perspective. arXiv preprint arXiv:2403.15796, 2024

    2024 arXiv

  8. [16]

    Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gard- ner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Compu...

    2019

  9. [17]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  10. [18]

    Open llm leaderboard v2

    Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open llm leaderboard v2. https://huggingface.co/spaces/open-llm-leaderboard/ open_llm_leaderboard, 2024

    2024

  11. [19]

    Language models scale reliably with over-training and on downstream tasks

    Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, et al. Language models scale reliably with over-training and on downstream tasks. arXiv preprint arXiv:2403.08540, 2024

    2024 arXiv

  12. [20]

    The pile: An 800gb dataset of diverse text for language modeling

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020

    2020 arXiv

  13. [21]

    Are we done with mmlu? arXiv preprint arXiv:2406.04127, 2024

    Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, et al. Are we done with mmlu? arXiv preprint arXiv:2406.04127, 2024

    2024 arXiv

  14. [22]

    Olmes: A standard for language model evaluations

    Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hannaneh Hajishirzi. Olmes: A standard for language model evaluations. arXiv preprint arXiv:2406.08446, 2024

    2024 arXiv

  15. [23]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2021

    2009 arXiv

  16. [24]

    Training compute-optimal large language models

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022

    2022 arXiv

  17. [25]

    Compression represents intelligence linearly

    Yuzhen Huang, Jinghan Zhang, Zifei Shan, and Junxian He. Compression represents intelligence linearly. arXiv preprint arXiv:2404.09937, 2024

    2024 arXiv

  18. [26]

    Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension

    Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1601–1611, 2017

    2017

  19. [27]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  20. [28]

    Natural questions: A benchmark for question answering research

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: A benchmark for question answering research. Transactions of the Association for Computa...

    2019

  21. [29]

    Finetasks: Finding signal in a haystack of 200+ multilingual tasks, 2024

    Hynek Kydlíˇcek, Guilherme Penedo, Clémentine Fourier, Nathan Habib, and Thomas Wolf. Finetasks: Finding signal in a haystack of 200+ multilingual tasks, 2024. URL https:// huggingface.co/spaces/HuggingFaceFW/blogpost-fine-tasks

    2024

  22. [30]

    Solving quantitative reasoning problems with language models

    Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206.14858, 2022. 12

    2022 arXiv

  23. [31]

    Datacomp-lm: In search of the next generation of training sets for language models

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Keh, Kushal Arora, et al. Datacomp-lm: In search of the next generation of training sets for language models. Advances in Neural Information Processin...

    2024

  24. [32]

    Autobencher: Creating salient, novel, difficult datasets for language models

    Xiang Lisa Li, Evan Zheran Liu, Percy Liang, and Tatsunori Hashimoto. Autobencher: Creating salient, novel, difficult datasets for language models. arXiv preprint arXiv:2407.08351, 2024

    2024 arXiv

  25. [33]

    Holistic evaluation of language models

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022

    2022 arXiv

  26. [34]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36:21558–21572, 2023

    2023

  27. [35]

    Regmix: Data mixture as regression for language model pre-training

    Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. Regmix: Data mixture as regression for language model pre-training. arXiv preprint arXiv:2407.01492, 2024

    2024 arXiv

  28. [36]

    Quantifying variance in evaluation bench- marks

    Lovish Madaan, Aaditya K Singh, Rylan Schaeffer, Andrew Poulton, Sanmi Koyejo, Pontus Stenetorp, Sharan Narang, and Dieuwke Hupkes. Quantifying variance in evaluation bench- marks. arXiv preprint arXiv:2406.10229, 2024

    2024 arXiv

  29. [37]

    Paloma: A benchmark for evaluating language model fit

    Ian Magnusson, Akshita Bhagia, Valentin Hofmann, Luca Soldaini, Ananya Harsh Jha, Oyvind Tafjord, Dustin Schwenk, Evan Pete Walsh, Yanai Elazar, Kyle Lo, et al. Paloma: A benchmark for evaluating language model fit. arXiv preprint arXiv:2312.10523, 2024

    2024 arXiv

  30. [38]

    Hwang, Luca Soldaini, Akshita Bhagia, Jiacheng Liu, Dirk Groeneveld, Oyvind Tafjord, Noah A

    Ian Magnusson, Tai Nguyen, David Heineman, Jena D. Hwang, Luca Soldaini, Akshita Bhagia, Jiacheng Liu, Dirk Groeneveld, Oyvind Tafjord, Noah A. Smith, Pang Wei Koh, Ben Bogin, and Jesse Dodge. Datadecide: How to predict best pretraining data with small experiments. under submi...

    2025

  31. [39]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, 2018

    2018

  32. [40]

    Adding error bars to evals: A statistical approach to language model evaluations

    Evan Miller. Adding error bars to evals: A statistical approach to language model evaluations. arXiv preprint arXiv:2411.00640, 2024

    2024 arXiv

  33. [41]

    Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models

    Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint arXiv:2410.05229, 2024

    arXiv

  34. [42]

    2 olmo 2 furious

    Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious. arXiv preprint arXiv:2501.00656, 2024

    2024 arXiv

  35. [43]

    Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Proceedings of the Conference on Health, Inference, and Learning (CHIL), pages 248–260, 2022

    2022

  36. [44]

    Reconciling kaplan and chinchilla scaling laws

    Tim Pearce and Jinyeop Song. Reconciling kaplan and chinchilla scaling laws. arXiv preprint arXiv:2406.12907, 2024

    2024 arXiv

  37. [45]

    tinybenchmarks: evaluating llms with fewer examples

    Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. tinybenchmarks: evaluating llms with fewer examples. arXiv preprint arXiv:2402.14992, 2024

    2024 arXiv

  38. [46]

    Varbench: Robust language model benchmarking through dynamic variable perturbation

    Kun Qian, Shunji Wan, Claudia Tang, Youzhi Wang, Xuanming Zhang, Maximillian Chen, and Zhou Yu. Varbench: Robust language model benchmarking through dynamic variable perturbation. arXiv preprint arXiv:2406.17681, 2024. 13

    2024 arXiv

  39. [47]

    Squad: 100,000+ questions for machine comprehension of text

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, 2016

    2016

  40. [48]

    Siva Reddy, Danqi Chen, and Christopher D. Manning. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266, 2019

    2019

  41. [49]

    Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R

    David Rein, Betty Li Hou, Asa C. Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023

    2023 arXiv

  42. [50]

    Com- pute optimal scaling of skills: Knowledge vs reasoning

    Nicholas Roberts, Niladri Chatterji, Sharan Narang, Mike Lewis, and Dieuwke Hupkes. Com- pute optimal scaling of skills: Knowledge vs reasoning. arXiv preprint arXiv:2503.10061, 2025

    2025 arXiv

  43. [51]

    Observational scaling laws and the predictability of langauge model performance

    Yangjun Ruan, Chris J Maddison, and Tatsunori B Hashimoto. Observational scaling laws and the predictability of langauge model performance. Advances in Neural Information Processing Systems, 37:15841–15892, 2025

    2025

  44. [52]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 8732–8740, 2020

    2020

  45. [53]

    Social iqa: Commonsense reasoning about social interactions

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social iqa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, pages 4463–4473, 2019

    2019

  46. [54]

    Why has predicting downstream capabilities of frontier ai models with scale remained elusive? arXiv preprint arXiv:2406.04391, 2024

    Rylan Schaeffer, Hailey Schoelkopf, Brando Miranda, Gabriel Mukobi, Varun Madan, Adam Ibrahim, Herbie Bradley, Stella Biderman, and Sanmi Koyejo. Why has predicting downstream capabilities of frontier ai models with scale remained elusive? arXiv preprint arXiv:2406.04391, 2024

    2024 arXiv

  47. [55]

    Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting, 2024

    Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting, 2024. URL https://arxiv.org/abs/2310.11324

    2024 arXiv

  48. [56]

    Predictive data selection: The data that predicts is the data that teaches

    Kashun Shum, Yuzhen Huang, Hongjian Zou, Ding Qi, Yixuan Liao, Xiaoxin Chen, Qian Liu, and Junxian He. Predictive data selection: The data that predicts is the data that teaches. arXiv preprint arXiv:2503.00808, 2025

    2025 arXiv

  49. [57]

    Predicting emergent capabilities by finetuning, 2024

    Charlie Snell, Eric Wallace, Dan Klein, and Sergey Levine. Predicting emergent capabilities by finetuning, 2024. URL https://arxiv.org/abs/2411.16035

    2024 arXiv

  50. [58]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint a...

    2022 arXiv

  51. [59]

    Commonsenseqa: A question answering challenge targeting commonsense knowledge

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua...

    2019

  52. [60]

    Llama 2: Open foundation and fine-tuned chat models, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  53. [61]

    200,000+ jeopardy! questions

    (Kaggle Datasets) Tunguz. 200,000+ jeopardy! questions. https://www.kaggle.com/ datasets/tunguz/200000-jeopardy-questions, 2019

    2019

  54. [62]

    Do large language model benchmarks test reliability? arXiv preprint arXiv:2502.03461, 2025

    Joshua Vendrow, Edward Vendrow, Sara Beery, and Aleksander Madry. Do large language model benchmarks test reliability? arXiv preprint arXiv:2502.03461, 2025

    2025 arXiv

  55. [63]

    Wang, Alex Gu, Lovish Madaan, Dieuwke Hupkes, Jiawei Liu, Yuxiang Wei, Naman Jain, Yuhang Lai, Sten Sootla, Ofir Press, Baptiste Rozière, and Gabriel Synnaeve

    Sida I. Wang, Alex Gu, Lovish Madaan, Dieuwke Hupkes, Jiawei Liu, Yuxiang Wei, Naman Jain, Yuhang Lai, Sten Sootla, Ofir Press, Baptiste Rozière, and Gabriel Synnaeve. Eval-Arena: noise and errors on llm evaluations. https://github.com/crux-eval/eval-arena, 2024

    2024

  56. [64]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Pr...

    2024

  57. [65]

    Emergent abilities of large language models

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022

    2022 arXiv

  58. [66]

    Organize the web: Constructing domains enhances pre-training data curation

    Alexander Wettig, Kyle Lo, Sewon Min, Hannaneh Hajishirzi, Danqi Chen, and Luca Soldaini. Organize the web: Constructing domains enhances pre-training data curation. arXiv preprint arXiv:2502.10341, 2025

    2025 arXiv

  59. [67]

    Answer, assemble, ace: Understanding how transformers answer multiple choice questions

    Sarah Wiegreffe, Oyvind Tafjord, Yonatan Belinkov, Hannaneh Hajishirzi, and Ashish Sabhar- wal. Answer, assemble, ace: Understanding how transformers answer multiple choice questions. arXiv preprint arXiv:2407.15018, 2024

    2024 arXiv

  60. [68]

    Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao

    Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer, 2022. URL https://arxiv.org/abs/ 2203.03466

    2022 arXiv

  61. [69]

    Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, 2019

    2019

  62. [70]

    Agieval: A human-centric benchmark for evaluating foundation models

    Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364, 2023. 15 A Methodology Details A.1 Scaling Law Details Hoffma...

    2023 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.