The reviewed record of science sign in
Pith

arxiv: 2109.06716 · v3 · pith:AWGFQFON · submitted 2021-09-14 · cs.LG

HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO

Katharina Eggensperger , Philipp M\"uller , Neeratyoy Mallik , Matthias Feurer , Ren\'e Sass , Aaron Klein , Noor Awad , Marius Lindauer
show 1 more author
Frank Hutter
This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:AWGFQFONrecord.jsonopen to challenge →

classification cs.LG
keywords hpobenchbenchmarksmulti-fidelitybenchmarkoptimizationtoolscomputationallyproblems
0
0 comments X
read the original abstract

To achieve peak predictive performance, hyperparameter optimization (HPO) is a crucial component of machine learning and its applications. Over the last years, the number of efficient algorithms and tools for HPO grew substantially. At the same time, the community is still lacking realistic, diverse, computationally cheap, and standardized benchmarks. This is especially the case for multi-fidelity HPO methods. To close this gap, we propose HPOBench, which includes 7 existing and 5 new benchmark families, with a total of more than 100 multi-fidelity benchmark problems. HPOBench allows to run this extendable set of multi-fidelity HPO benchmarks in a reproducible way by isolating and packaging the individual benchmarks in containers. It also provides surrogate and tabular benchmarks for computationally affordable yet statistically sound evaluations. To demonstrate HPOBench's broad compatibility with various optimization tools, as well as its usefulness, we conduct an exemplary large-scale study evaluating 13 optimizers from 6 optimization tools. We provide HPOBench here: https://github.com/automl/HPOBench.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. ASAP: Agent-System Co-Design for Wall-Clock-Centered Auto HPO Research for ML Experiments

    cs.LG 2026-06 unverdicted novelty 7.0

    ASAP integrates an LLM agent over a pool of HPO tools and adds system-level optimizations (prefix-stable prompts, speculation parallelism, Self-Tuner) to improve end-to-end wall-clock performance on diverse HPO tasks.

  2. RAISE: RAG Design as an Architecture Search Problem

    cs.AI 2026-05 unverdicted novelty 7.0

    RAISE is a standardized benchmark for RAG hyperparameter optimization that evaluates 13 search algorithms across seven datasets and finds performance is highly task-dependent.

  3. Regime-Conditioned Evaluation in Multi-Context Bayesian Optimization

    cs.LG 2026-05 unverdicted novelty 7.0

    The Portable Regime Score PRS=(B/|A|)(1-rho) captures and predicts acquisition function performance reversals in transfer Bayesian optimization, enabling a RegimePlanner that adapts and beats fixed baselines.

  4. BayMOTH: Bayesian optiMizatiOn with meTa-lookahead -- a simple approacH

    cs.LG 2026-04 unverdicted novelty 4.0

    BayMOTH unifies meta-Bayesian optimization with a usefulness-based fallback to lookahead, demonstrating competitive results on function optimization tasks even under low task relatedness.