A System for Massively Parallel Hyperparameter Tuning
Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 reserved pith:BBHKOZCJrecord.jsonopen to challenge →
read the original abstract
Modern learning models are characterized by large hyperparameter spaces and long training times. These properties, coupled with the rise of parallel computing and the growing demand to productionize machine learning workloads, motivate the need to develop mature hyperparameter optimization functionality in distributed computing settings. We address this challenge by first introducing a simple and robust hyperparameter optimization algorithm called ASHA, which exploits parallelism and aggressive early-stopping to tackle large-scale hyperparameter optimization problems. Our extensive empirical results show that ASHA outperforms existing state-of-the-art hyperparameter optimization methods; scales linearly with the number of workers in distributed settings; and is suitable for massive parallelism, as demonstrated on a task with 500 workers. We then describe several design decisions we encountered, along with our associated solutions, when integrating ASHA in Determined AI's end-to-end production-quality machine learning system that offers hyperparameter tuning as a service.
This paper has not been read by Pith yet.
Forward citations
Cited by 5 Pith papers
-
A Systematic Evaluation of Molecular Mixture Behavior Prediction
Strong absolute accuracy on mixture properties often masks poor recovery of non-ideal behavior, with large drops under strict molecule splits, making transfer to unseen molecules the central challenge.
-
QuickScope: Certifying Hard Questions in Dynamic LLM Benchmarks
QuickScope uses modified COUP Bayesian optimization to find truly difficult questions in dynamic LLM benchmarks more sample-efficiently than baselines while cutting false positives.
-
Unveiling Hidden Lyman Alpha Emitters in the DESI DR1 Data
A CNN detects 19,685 LAEs at z=2-3.5 in DESI DR1 spectra with 95% purity and completeness.
-
Quantifying the Carbon Emissions of Machine Learning
Presents a calculator tool for estimating carbon emissions from ML model training along with mitigation actions.
-
Staged Factorial Screening for Budget-Constrained Micro-Pretraining
Staged factorial screening recovers stable early penalties from total batch, depth, and width in 2-10 minute pretraining runs and supports a bridge-centered recommendation through 24-hour continuations on two hosts.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.