The reviewed record of science sign in
Pith

arxiv: 2103.03098 · v1 · pith:JL5RKDIC · submitted 2021-03-01 · cs.LG · stat.ML

Accounting for Variance in Machine Learning Benchmarks

Xavier Bouthillier , Pierre Delaunay , Mirko Bronzi , Assya Trofimov , Brennan Nichyporuk , Justin Szeto , Naz Sepah , Edward Raff
show 9 more authors
Kanika Madan Vikram Voleti Samira Ebrahimi Kahou Vincent Michalski Dmitriy Serdyuk Tal Arbel Chris Pal Ga\"el Varoquaux Pascal Vincent
This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:JL5RKDICrecord.jsonopen to challenge →

classification cs.LG stat.ML
keywords datavarianceestimatorinitializationlearningparameterresultssampling
0
0 comments X
read the original abstract

Strong empirical evidence that one machine-learning algorithm A outperforms another one B ideally calls for multiple trials optimizing the learning pipeline over sources of variation such as data sampling, data augmentation, parameter initialization, and hyperparameters choices. This is prohibitively expensive, and corners are cut to reach conclusions. We model the whole benchmarking process, revealing that variance due to data sampling, parameter initialization and hyperparameter choice impact markedly the results. We analyze the predominant comparison methods used today in the light of this variance. We show a counter-intuitive result that adding more sources of variation to an imperfect estimator approaches better the ideal estimator at a 51 times reduction in compute cost. Building on these results, we study the error rate of detecting improvements, on five different deep-learning tasks/architectures. This study leads us to propose recommendations for performance comparisons.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. A Pre-Registered Causal Partition of Self-Consistency Elicitation and Reward Design in RLVR

    cs.AI 2026-06 conditional novelty 8.0

    Derives an exact telescoping decomposition of the naive RLVR reward-design estimator into null, elicitation, and reward-design terms on a tabular-GRPO simulator, measures the components across prior strengths, and val...

  2. From Forecasting Leaderboards to Deployment Decisions: A Fail-Closed Certification Protocol

    cs.LG 2026-06 unverdicted novelty 6.0

    Presents a fail-closed certification protocol for determining when forecasting leaderboard winners are deployment-actionable, using a traffic dataset to show friction-induced reversals and an audit to prevent overclaiming.

  3. SafetyRepro: Configuration-Conditional Rank Instability on Alignment Benchmarks

    cs.LG 2026-05 unverdicted novelty 6.0

    Configuration choices alone flip pairwise safety verdicts on every tested alignment benchmark, isolated via a finite-envelope proposition linking disagreement rate to strict ordering reversal.

  4. QuickScope: Certifying Hard Questions in Dynamic LLM Benchmarks

    cs.CL 2026-04 unverdicted novelty 6.0

    QuickScope uses modified COUP Bayesian optimization to find truly difficult questions in dynamic LLM benchmarks more sample-efficiently than baselines while cutting false positives.