{"id":"d30679bf-d4b5-458d-8ce2-8e53889a33fc","arxiv_id":"2508.13144","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Benchmark signal-to-noise ratio, measured as score spread across models divided by checkpoint-to-checkpoint variability, predicts small-to-large model decision accuracy and can be improved by subtask filtering, checkpoint averaging, or switching to bits-per-byte.","lead":"Researchers introduce two statistics, signal and noise, to rate how useful a benchmark is for choosing between language models at small scale. They show that benchmarks with higher signal-to-noise ratio support more reliable small-to-large model decisions and demonstrate three interventions, filtering subtasks, averaging checkpoints, and using bits-per-byte, that improve these decisions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Noise-error correlation may be inflated by normalizing both variables by task score magnitude; test with absolute/partial correlations.","rationale":"I read the paper as proposing a practical framework for benchmark selection, with the strongest independent assets being the large released dataset and direct interventions (checkpoint averaging, BPB) that do not rely solely on the cross-task correlation. However, the noise-error correlation is the keystone for the recommendation that lowering noise lowers scaling-law error. The normalization artifact is concrete and testable: both variables divide by a score-magnitude term, so hard tasks with low scores can drive the correlation mechanically. If the concern lands, the 'noise bounds scaling error' conjecture in §4.2 is unsupported, and the BPB gains in §5.3 may be explained by floor removal rather than by improved signal-to-noise ratio. This does not warrant rejection because the released data allow the proposed test, and the decision-accuracy findings rest on separate evidence. A conditional verdict with this added analytical requirement remains appropriate.","tokens_in":52038,"tokens_out":9112,"duration_ms":101006,"concrete_test":"Recompute the Fig. 3 analysis with absolute checkpoint standard deviation (not divided by mean) against absolute scaling-law error (not divided by true value), and compute the partial Spearman correlation between log relative noise and log relative error controlling for log mean score (or log true score) across the same 30 tasks. If the partial correlation drops to near zero or changes sign, the headline noise-error claim is largely a task-difficulty artifact; if it remains substantial (e.g., ρ≥0.3), the concern is answered.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is that the central evidence for noise predicting scaling-law error (Fig. 3, §4.2) may be inflated by a normalization artifact. Noise is defined as Rel. Std. = σ/mean (§3.1), and prediction error is defined as |predicted − true|/|true| (§2.1). Because both denominators shrink for hard tasks with low scores, a task-difficulty gradient can produce a positive correlation even if checkpoint noise is causally unrelated to scaling-law fit quality. The reported R²=0.426 is a cross-task correlation over 30 tasks with no control for mean score or task difficulty. The lower-bound interpretation is explicitly a conjecture (§4.2), and the same normalization likely drives the BPB improvements in §5.3: switching from accuracy (near floor for many math/code tasks) to bits-per-byte removes the floor and reduces both relative noise and relative error simultaneously. Thus the paper's second central claim—'those with less noise have lower scaling law prediction error'—is not yet cleanly separated from 'tasks where small models score near floor are hard to predict.'","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a framework for characterizing evaluation benchmarks by two quantities: signal (relative dispersion of scores across a population of small models) and noise (relative standard deviation of a single model's final checkpoints). The signal-to-noise ratio (SNR) is shown to correlate with decision accuracy for small-to-large model ranking (Fig. 2, R = 0.791), and noise is shown to correlate with scaling-law prediction error (Fig. 3, R = 0.653, R² = 0.426). The authors then introduce three interventions motivated by this framework: filtering subtasks by SNR, averaging final checkpoints to reduce noise, and replacing primary task metrics with bits-per-byte. The experiments use 30 benchmarks and a large collection of open-weight models, and the authors release a dataset of about 900K evaluation results.","tokens_in":52223,"tokens_out":4685,"duration_ms":51783,"significance":"If the proposed framework holds, it offers a cheap and actionable diagnostic for benchmark construction and selection: compute SNR at small scale and use it to anticipate whether small experiments will transfer to larger models. The paper is unusually transparent about its methodology, including an appendix comparing 20 signal measures and an appendix validating checkpoint-to-checkpoint noise against seed noise and data-order noise. The public release of evaluation results, trained models, and intermediate checkpoints is a substantial reproducibility contribution. The central risk is that the second main claim, that lower noise causes lower scaling-law prediction error, is supported by a cross-task correlation whose two variables are both normalized by score magnitude, so the correlation may partly reflect task difficulty rather than a genuine predictive relation. The intervention results in Sections 5.1 and 5.3 are also evaluated in-sample or under the same normalization, so their interpretation depends on resolving this confound.","major_comments":[{"comment":"The reported R² = 0.426 between noise and scaling-law prediction error is potentially inflated by shared score-magnitude normalization. Noise is defined as σ/μ and prediction error as |predicted − true|/|true|; for hard tasks with low 13B scores, both denominators are small, so a task-difficulty gradient can produce a positive correlation even if checkpoint noise is causally unrelated to extrapolation error. The paper should report partial correlations controlling for mean score, absolute (unnormalized) errors, and within-task analyses across multiple prediction targets. The bits-per-byte results in §5.3 are subject to the same concern, because switching from floor-saturated accuracy to bits-per-byte changes both relative quantities simultaneously. Without these controls, the claim that benchmarks with less noise have lower scaling-law prediction error is not cleanly separated from the claim that hard tasks are hard to predict.","section":"§4.2, Eq. (4), Fig. 3"},{"comment":"The signal measure was selected from 20 candidates as the one maximizing correlation with decision accuracy on the same OLMES benchmarks used to report Fig. 2, so the headline R² = 0.626 for SNR versus decision accuracy is an in-sample fit. Many alternatives in Table 3 have nearly identical R², which mitigates the concern, but the reported number remains optimistically biased. The authors should provide a holdout evaluation (for example, splitting benchmarks by category or using the non-OLMES tasks) or apply a selection-bias correction before presenting this correlation as the central evidence for the framework.","section":"§3.2, App. A.4, Table 3"},{"comment":"The subtask-filtering intervention is evaluated in-sample. Subtask SNR is computed from the 1B DataDecide models, and the same 150M-to-1B DataDecide ranking is then used to measure the improvement in decision accuracy, so the greedy ordering can exploit idiosyncrasies of the evaluation population. The random-shuffle baseline does not cure this. The authors should report a holdout evaluation, such as using 7B or 13B rankings from the OLMo 2 family or external open-weight models, before claiming that SNR-based filtering improves multi-task evaluations.","section":"§5.1, Fig. 4"},{"comment":"The choice of checkpoint-to-checkpoint noise as the representative modeling-noise measure is validated on only 7 OLMES tasks with 20 1B models (R² between 0.82 and 0.95). The framework is then applied to math and code tasks and to 13B and 32B targets, where the transfer of this proxy is an assumption. The authors should either provide a small-scale check for generative math/code tasks or larger models, or explicitly restrict the scope of the noise-reduction interventions to settings where the proxy has been validated.","section":"§3.1, App. A.3.1, Fig. 7"}],"minor_comments":[{"comment":"There are several typos and inconsistent spellings: 'wether' in the Figure 2 caption, 'influce' in §5.1, 'illustraed' in §4.2, 'recepies' in App. A.5.1, 'dimishing' in App. B.2, 'A veraging' in the §5.2 heading, and inconsistent capitalization of 'AutoBencher' versus 'Autobencher'.","section":"Throughout"},{"comment":"The abstract states that 375 open-weight models are used, while Section 2 and App. A.5.1 report 465 models; the relationship between these numbers should be clarified.","section":"Abstract vs. §2"},{"comment":"The noise definition in Eq. (2) leaves the number of checkpoints n unspecified; the main text uses n = 5 for the small models and n = 30 for the 13B target. The authors should state the chosen n values at the point of definition and note that the noise estimates are not directly comparable across different n.","section":"§3.3 and §4.2"},{"comment":"The definition of bits-per-byte says it is the negative log likelihood of the correct answer divided by the number of UTF-8 bytes, but this is dimensionally not 'bits' unless the logarithm base is specified as 2; please clarify the base and the exact byte-counting convention.","section":"§5.3"},{"comment":"The chi-squared guidance for selecting n relies on the assumption that checkpoint-to-checkpoint scores are independent and normally distributed. This assumption should be stated in the main text alongside the recommendation of n = 20, since checkpoint scores within a single run are serially correlated in practice.","section":"App. A.3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper's first contribution, SNR as a benchmark diagnostic for decision accuracy, is plausible despite the in-sample selection issue because many signal variants in Table 3 yield similar correlations. The more serious problem is the normalization confound in §4.2, which affects the paper's second central claim and the interpretation of the BPB intervention. I do not think this warrants rejection, because the interventions can be tested independently and the required control analyses are within the scope of a revision, but the current manuscript does not yet cleanly establish the noise-to-error relation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth your time if you work on evaluation or small-scale model selection. It defines signal and noise for benchmarks, shows SNR correlates with small-to-large decision accuracy, and tests three cheap interventions (subtask filtering, checkpoint averaging, swapping to bits-per-byte). The scale is impressive: 465 models, 30 benchmarks, 900K evaluation results, all released. The writing is honest, and the authors explicitly label the noise-as-bound idea a conjecture, which I appreciate.\n\nThe decision-accuracy result (Fig. 2) is the more robust half. SNR built from relative dispersion over a population of data recipes divided by checkpoint-to-checkpoint noise predicts rank agreement between 150M and 1B models fairly well (R²=0.626 over the OLMES tasks). That is not circular: decision accuracy is a rank-based measurement, not divided by score magnitude. The checkpoint-averaging intervention also gives consistent improvements across many tasks, which is independent evidence that step-to-step noise matters.\n\nThe soft spots are real and mostly in the second half. First, the signal metric was selected from 20 candidates by maximizing correlation with decision accuracy on the same benchmarks, so the headline R² is in-sample and optimistic. Second, the subtask filtering results (Fig. 4) are evaluated on the same models used to pick the subsets; a held-out recipe or new models would be needed to trust the +2.6%/+5% claims. Third, and most important, the noise–scaling-law-error correlation (Fig. 3, R²=0.426) is vulnerable to a normalization confound: both relative noise (σ/mean) and relative prediction error (|pred−true|/|true|) shrink as task scores grow, so tasks where small models sit near floor are exactly the ones with inflated values on both axes. The stress-test note is on target here. Without partial correlations controlling for mean score or task difficulty, the claim that checkpoint noise lowers scaling law error is not cleanly separated from the claim that hard tasks are just hard to predict. The same artifact likely contributes to the BPB improvements in §5.3, since switching from accuracy to BPB removes the floor for math and code tasks.\n\nThese are addressable problems: hold out some benchmarks when selecting the signal metric, validate subtask filtering on unseen models, and redo Fig. 3 with absolute errors and a difficulty control. The paper deserves a serious referee, but it should not be accepted in its current form. I would bring it to our reading group and cite the dataset and the SNR framing when I next write about evaluation design.","headline":"A serious, data-rich paper on benchmark SNR for LLM evaluation; the decision-accuracy result is solid, but the noise–scaling-law-error link is likely inflated by a normalization artifact and needs out-of-sample fixing.","tokens_in":52758,"tokens_out":1638,"would_cite":true,"duration_ms":21183,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A benchmark's signal-to-noise ratio, measured from small models, predicts whether its rankings survive scaling to larger models and how accurate scaling-law forecasts will be.","keywords":["signal-to-noise ratio","language model evaluation","scaling laws","decision accuracy","benchmark reliability","checkpoint averaging","bits-per-byte","noise reduction"],"falsifier":"One could settle this by taking a new set of tasks outside the validated set, training 1B models with varied random seeds and data orders, and comparing seed noise and data-order noise against checkpoint-to-checkpoint noise; a task with large seed noise but small checkpoint noise would disprove the claim that final-checkpoint wobble captures modeling noise, and would break the argument that reducing checkpoint noise improves scaling-law prediction.","tokens_in":1675,"feed_emoji":"📊","tokens_out":4191,"duration_ms":91229,"temperature":0.7,"pith_summary":"The paper tries to establish that evaluation benchmarks have measurable intrinsic properties—signal, how spread out model scores are, and noise, how much a single model's score wobbles between training checkpoints—and that their ratio predicts whether small-scale experiments will transfer to large models. If correct, benchmark builders can diagnose and repair evaluation suites cheaply, before spending large compute on risky decisions. The claim is supported by evaluations of hundreds of open-weight models on thirty benchmarks, and by three interventions that directly target signal or noise: filtering noisy subtasks, averaging checkpoints, and scoring by bits-per-byte instead of accuracy. The value of the framework is that both signal and noise can be measured at small scale, without waiting for expensive large-model runs.","feed_headline":"One ratio flags reliable LLM benchmarks","feed_subtitle":"Checkpoint noise plus model spread, measured at small scale, predicts whether benchmark rankings hold when models grow.","key_machinery":"The signal-to-noise ratio is defined as $$\\text{SNR}(B) = \\frac{\\text{Rel. Dispersion(final checkpoints)}}{\\text{Rel. Std.}(\\text{final } n \\text{ checkpoints)}},$$ where relative dispersion is the maximum difference between any two small models' final scores divided by the mean score, and relative noise is the standard deviation of a single model's final checkpoints divided by its mean. This ratio is the load-bearing quantity: neither signal nor noise alone correlates with decision accuracy, but their ratio does. It is cheap to compute because it uses only final checkpoints already produced during development, and it is the quantity that all three interventions are designed to improve.","core_discovery":"The central discovery is a two-number definition of benchmark quality: signal is the relative dispersion of final-checkpoint scores across a population of small models trained under a similar compute budget, and noise is the relative standard deviation of the final checkpoints of a single training run. The paper shows that their ratio, the signal-to-noise ratio, is strongly correlated with decision accuracy—whether ranking small 60M-to-750M models predicts the ranking of 1B models—with a correlation of 0.79 across the OLMES benchmarks. It further shows that the noise of the large prediction target correlates with scaling-law prediction error, with a correlation of 0.65 across 30 tasks. These correlations motivate three interventions, each of which improves both decision accuracy and scaling-law error: ranking subtasks by signal-to-noise ratio to filter noisy subtasks, averaging or smoothing checkpoint scores to reduce noise, and replacing discontinuous metrics like accuracy with bits-per-byte to increase signal.","pith_inferences":["If the noise-to-scaling-law-error link is causal rather than merely correlational, then measuring checkpoint noise during training could guide stopping criteria: a practitioner could train until the target's noise drops below the tolerated prediction error.","The same signal-to-noise machinery could serve as a routine quality metric for newly released benchmarks, computed once on a standard population of open-weight models, giving developers a cheap way to compare evaluation suites before adopting them.","A natural extension is instance-level rather than subtask-level SNR: greedily adding individual instances sorted by marginal SNR could yield much smaller test sets than subtask filtering, and would test whether the MMLU and AutoBencher subset results generalize.","The framework makes a testable prediction for debates about emergent abilities: tasks where small models perform at chance will have very low SNR at small scale, and their SNR should rise across model sizes exactly when small-to-large rankings become reliable."],"forward_implications":["Benchmarks with higher signal-to-noise ratio at small scale will rank training choices at large scale more reliably, so developers can select evaluation suites by measuring SNR on small models they already train.","Lower-noise benchmarks should yield tighter scaling-law predictions, and a task's checkpoint noise around the prediction target can serve as a sanity-check lower bound on the minimum scaling-law error.","Filtering subtasks by SNR can produce smaller but more reliable evaluation sets—the paper reports MMLU subsets with 16 subtasks and AutoBencher subsets with 6 subtasks that outperform the full suites on decision accuracy—so bigger benchmarks are not necessarily better for development decisions.","Averaging the final checkpoints of small prediction models and target models improves decision accuracy and scaling-law prediction error, and the same smoothing helps when choosing a model by early stopping.","Switching from accuracy, exact match, or pass@1 to bits-per-byte raises SNR on most benchmarks, improving decision accuracy on 90% of benchmarks and reducing scaling-law prediction error on 73.3% of them."],"supporting_citations":[{"why":"Supplies the 25 pretraining corpora and the small-to-large decision accuracy setup used throughout.","marker":"[38]"},{"why":"Provides the scaling-law model ladder and fitting methodology used for prediction error.","marker":"[3]"},{"why":"Establishes the multi-task scaling-law prediction setup that the paper extends.","marker":"[19]"},{"why":"Defines the OLMES evaluation standard used for many benchmarks.","marker":"[22]"},{"why":"Closest prior SNR work, which the paper contrasts with its cheaper checkpoint-based noise estimate.","marker":"[36]"},{"why":"Generates the synthetic AutoBencher benchmark used for the subtask-filtering intervention.","marker":"[32]"},{"why":"Identifies labeling errors in MMLU subtasks, linking low SNR with low task quality.","marker":"[21]"},{"why":"Supplies OLMo 2 checkpoints used to measure target noise at several scales.","marker":"[42]"}],"fun_headline_variants":["Signal-to-noise ratio predicts which LLM benchmarks are reliable","Benchmark quality: signal-to-noise ratio flags reliable LLMs","Signal beats noise for LLM benchmark picks","Benchmark fix: raise signal, lower noise","One ratio picks reliable LLM benchmarks"],"cache_read_input_tokens":55040,"weakest_assumption_plain":"The load-bearing premise is that checkpoint-to-checkpoint wobble in the final steps of a single training run is a faithful stand-in for all modeling noise; if that proxy fails on tasks beyond the small set used to validate it, the claimed correlations and the interventions that target checkpoint noise do not follow.","fun_headline_variants_meta":{"raw":{"variants":["Signal-to-noise ratio predicts which LLM benchmarks are reliable","Benchmark quality: signal-to-noise ratio flags reliable LLMs","Signal beats noise for LLM benchmark picks","Benchmark fix: raise signal, lower noise","One ratio picks reliable LLM benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00127,"raw_usage":{"total_tokens":5229,"prompt_tokens":1011,"completion_tokens":4218,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":627,"completion_tokens_details":{"reasoning_tokens":4145}},"tokens_in":627,"tokens_out":4218,"duration_ms":27952,"temperature":1.0,"reasoning_tokens":4145,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:15:07.679137+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"One could settle this by taking a new set of tasks outside the validated set, training 1B models with varied random seeds and data orders, and comparing seed noise and data-order noise against checkpoint-to-checkpoint noise; a task with large seed noise but small checkpoint noise would disprove the claim that final-checkpoint wobble captures modeling noise, and would break the argument that reducing checkpoint noise improves scaling-law prediction.","supporting_citations":[{"cited_title":"Hwang, Luca Soldaini, Akshita Bhagia, Jiacheng Liu, Dirk Groeneveld, Oyvind Tafjord, Noah A","cited_arxiv_id":null,"evidence_quote":"Supplies the 25 pretraining corpora and the small-to-large decision accuracy setup used throughout."}],"review_version":1}