pith. sign in

arxiv: 2606.10068 · v1 · pith:TMJ6ALU4new · submitted 2026-06-08 · 💻 cs.LG · cs.AI

Importance-Aware Scheduling for High-Dimensional Hyperparameter Optimization

Ruinan Wang , Ian Nabney , Mohammad Golbabaee This is my paper

Pith reviewed 2026-06-27 17:03 UTC · model grok-4.3

classification 💻 cs.LG cs.AI
keywords hyperparameter optimizationimportance estimationhigh-dimensional searchscheduling strategysample efficiencygreedy importance firstbayesmarkNAS-Bench
0
0 comments X

The pith

Greedy Importance First scheduling improves sample efficiency in high-dimensional hyperparameter optimization by prioritizing important variables.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces Greedy Importance First (GIF) as a scheduling strategy for hyperparameter optimization. It starts with a small warm-up phase to estimate which hyperparameters have the most impact, then groups them by importance and allocates evaluation trials proportionally to those groups while keeping a fallback to the full space. This approach aims to avoid wasting evaluations on low-impact dimensions in high-dimensional spaces. If effective, it would allow optimizers to reach better performing models with fewer trials when the search space has many variables of varying relevance.

Core claim

GIF uses a small-sample warm start to estimate hyperparameter importance, forms importance-based groups, allocates trials proportionally to group importance, and retains a full-space fallback. On higher-dimensional benchmarks including analytic functions, Bayesmark, and NAS-Bench-301, this leads to better incumbents and faster convergence compared to TPE, BOHB, Random Search, and Sequential Grouping, with ablations confirming the contribution of each component.

What carries the argument

The Greedy Importance First (GIF) scheduler, which estimates importance from warm-start samples, groups hyperparameters, and allocates trials proportionally while including a full-space fallback.

If this is right

  • Importance estimation from warm starts enables stable grouping for proportional trial allocation in high dimensions.
  • GIF outperforms standard methods on higher-dimensional benchmarks but shows smaller gains when effective dimensionality is lower.
  • Each of importance estimation, proportional allocation, and the fallback step contributes to the observed performance gains.
  • The method recovers intended anisotropy on analytic benchmarks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The approach could be combined with existing HPO algorithms like TPE or BOHB as a plug-in scheduler.
  • Similar importance-based allocation might apply to other expensive black-box optimization problems beyond machine learning.
  • Testing on even higher dimensional spaces or real-world DL tasks with thousands of hyperparameters would further validate the method.

Load-bearing premise

A small number of initial evaluations produces reliable estimates of which hyperparameters are most important for forming stable groups.

What would settle it

Running GIF on a high-dimensional benchmark where the hyperparameter importance ranking from the warm-start samples does not match the actual impact on the objective would show if performance gains vanish.

Figures

Figures reproduced from arXiv: 2606.10068 by Ian Nabney, Mohammad Golbabaee, Ruinan Wang.

Figure 1
Figure 1. Figure 1: GIF Pipeline: High-level workflow of the proposed Greedy Impor￾tance First strategy. estimates into concrete scheduling decisions. As a result, HIA methods are underutilized in practice. This paper introduces Greedy Importance-First (GIF), an importance-aware HPO strategy that turns HIA insights into an explicit, budgeted search plan. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Anisotropy verification on weighted Ackley and Griewank: nor [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Performance summary of GIF and baselines on weighted analytic benchmarks. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Convergence and Pareto analysis on NAS-Bench-301 (DARTS-XGB surrogate, 33D) [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Pareto trade-off between final score and time (lightweight methods). [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
read the original abstract

Hyperparameter Optimization (HPO) is essential for building high-performing ML/DL models, yet conventional optimizers often struggle in high-dimensional spaces where evaluations are costly and progress is diluted across many low-impact variables. We propose Greedy Importance First (GIF), an importance-aware scheduling strategy that uses a small-sample warm start to estimate hyperparameter importance, forms importance-based groups, allocates trials proportionally, and retains a full-space fallback. We evaluate GIF under fixed evaluation budgets on five anisotropic analytic functions, Bayesmark, and NAS-Bench-301. On the higher-dimensional benchmarks, GIF reaches better incumbents with faster convergence than TPE, BOHB, Random Search, and Sequential Grouping. On Bayesmark, where the effective dimensionality is smaller, GIF remains competitive but the margins are smaller. Ablation studies show that importance estimation, proportional allocation, and the fallback step all contribute to the gains. We also verify that the HIA component recovers the intended anisotropy on the analytic benchmarks. These results suggest that GIF is a simple and plug-compatible way to improve sample efficiency in high-dimensional HPO.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 1 minor

Summary. The paper proposes Greedy Importance First (GIF), an importance-aware scheduling strategy for high-dimensional hyperparameter optimization. It uses a small-sample warm start to estimate hyperparameter importance, forms importance-based groups, allocates trials proportionally, and includes a full-space fallback. Evaluations on five anisotropic analytic functions, Bayesmark, and NAS-Bench-301 show that on higher-dimensional benchmarks, GIF reaches better incumbents with faster convergence than TPE, BOHB, Random Search, and Sequential Grouping. Ablations confirm the contribution of each component, and HIA recovers intended anisotropy on analytic benchmarks.

Significance. If the results hold, GIF offers a simple, plug-compatible method to improve sample efficiency in high-dimensional HPO by focusing trials on important hyperparameters. The ablations isolating each component and the verification that HIA recovers intended anisotropy on analytic functions provide concrete empirical support for the approach.

major comments (1)
  1. [Abstract (HIA verification and ablation studies)] The central claim that GIF produces faster convergence on high-dimensional benchmarks rests on the assumption that small-sample warm-start importance estimates are sufficiently reliable to form stable groups and drive proportional allocation. The verification that HIA recovers intended anisotropy on analytic functions does not quantify estimator variance across random seeds or demonstrate stability on the transition to real ML spaces (Bayesmark, NAS-Bench-301); if importance rankings are noisy or flip, the scheduler may systematically mis-allocate trials and the reported gains would not be reproducible.
minor comments (1)
  1. The abstract does not report the exact dimensionality or number of trials in the warm-start phase, which would help readers assess the small-sample regime directly.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive comment on the reliability of the small-sample importance estimates. We address the concern regarding variance quantification and stability below, and commit to revisions that strengthen the supporting evidence without altering the core claims.

read point-by-point responses

  1. Referee: [Abstract (HIA verification and ablation studies)] The central claim that GIF produces faster convergence on high-dimensional benchmarks rests on the assumption that small-sample warm-start importance estimates are sufficiently reliable to form stable groups and drive proportional allocation. The verification that HIA recovers intended anisotropy on analytic functions does not quantify estimator variance across random seeds or demonstrate stability on the transition to real ML spaces (Bayesmark, NAS-Bench-301); if importance rankings are noisy or flip, the scheduler may systematically mis-allocate trials and the reported gains would not be reproducible.

    Authors: We agree that explicit quantification of estimator variance would strengthen the manuscript. The current verification shows HIA recovers intended anisotropy on analytic functions, but does not report variance across seeds. In revision we will add tables or plots quantifying the variance of importance rankings (and resulting group assignments) over multiple random seeds on the five analytic benchmarks. Regarding transition to real ML spaces, the main experimental results already include multiple independent runs on Bayesmark and NAS-Bench-301, with GIF showing consistent gains; unstable or flipping rankings would be expected to produce high variance or degraded performance, which is not observed. Nevertheless, we will add a supplementary analysis of ranking stability (e.g., Kendall-tau correlation of importance orderings across seeds) on these benchmarks where the underlying data permits, or explicitly note the limitation if additional computation is required. revision: yes

Circularity Check

0 steps flagged

No significant circularity; algorithmic procedure evaluated on external benchmarks

full rationale

The paper presents GIF as an algorithmic scheduling strategy (warm-start importance estimation, group formation, proportional allocation, fallback) and reports empirical results on fixed external benchmarks (anisotropic analytic functions, Bayesmark, NAS-Bench-301). No equations, fitted parameters renamed as predictions, or self-citation chains are shown that reduce the performance claims or group allocations to quantities defined by the same data or prior self-work. The method is a plug-compatible procedure whose claims rest on benchmark comparisons rather than any self-referential derivation. This is the normal non-circular case for an applied HPO paper.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The paper introduces an algorithmic procedure rather than new physical or mathematical entities. No free parameters are explicitly fitted to target data in the abstract; the method relies on standard assumptions from the HPO literature such as the existence of anisotropic response surfaces and the usefulness of ranking-based grouping.

axioms (2)
  • domain assumption A small number of initial evaluations suffices to produce a stable ranking of hyperparameter importance.
    Invoked by the warm-start importance estimation step described in the abstract.
  • domain assumption Hyperparameter response surfaces in the target domains are sufficiently anisotropic that importance-based grouping yields measurable gains.
    Central to the claim that proportional allocation improves sample efficiency on the higher-dimensional benchmarks.

pith-pipeline@v0.9.1-grok · 5720 in / 1415 out tokens · 18509 ms · 2026-06-27T17:03:11.753912+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

29 extracted references · 6 canonical work pages · 3 internal anchors

  1. [1]

    Algorithms for hyper- parameter optimization,

    J. Bergstra, R. Bardenet, Y . Bengio, and B. K ´egl, “Algorithms for hyper- parameter optimization,”Advances in neural information processing systems, vol. 24, 2011

    2011

  2. [2]

    Bohb: Robust and efficient hyperparameter optimization at scale,

    S. Falkner, A. Klein, and F. Hutter, “Bohb: Robust and efficient hyperparameter optimization at scale,” inInternational Conference on Machine Learning. PMLR, 2018, pp. 1437–1446

    2018

  3. [3]

    CMA-ES for Hyperparameter Optimization of Deep Neural Networks

    I. Loshchilov and F. Hutter, “Cma-es for hyperparameter optimization of deep neural networks,”arXiv preprint arXiv:1604.07269, 2016

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  4. [4]

    Hyperband: A novel bandit-based approach to hyperparameter opti- mization,

    L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar, “Hyperband: A novel bandit-based approach to hyperparameter opti- mization,”Journal of Machine Learning Research, vol. 18, no. 185, pp. 1–52, 2018

    2018

  5. [5]

    Hyperparameter optimization: Foundations, algorithms, best practices, and open chal- lenges,

    B. Bischl, M. Binder, M. Lang, T. Pielok, J. Richter, S. Coors, J. Thomas, T. Ullmann, M. Becker, A.-L. Boulesteixet al., “Hyperparameter optimization: Foundations, algorithms, best practices, and open chal- lenges,”Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, vol. 13, no. 2, p. e1484, 2023

    2023

  6. [6]

    Tunability: Importance of hyperparameters of machine learning algorithms,

    P. Probst, A.-L. Boulesteix, and B. Bischl, “Tunability: Importance of hyperparameters of machine learning algorithms,”Journal of Machine Learning Research, vol. 20, no. 53, pp. 1–32, 2019

    2019

  7. [7]

    Efficient hyperparameter importance assessment for cnns,

    R. Wang, I. Nabney, and M. Golbabaee, “Efficient hyperparameter importance assessment for cnns,” inInternational Conference on Neural Information Processing. Springer, 2024, pp. 16–31

    2024

  8. [8]

    An efficient approach for assessing hyperparameter importance,

    F. Hutter, H. Hoos, and K. Leyton-Brown, “An efficient approach for assessing hyperparameter importance,” inInternational conference on machine learning. PMLR, 2014, pp. 754–762

    2014

  9. [9]

    PED-ANOVA: Efficiently Quantifying Hyperparameter Importance in Arbitrary Subspaces

    S. Watanabe, A. Bansal, and F. Hutter, “Ped-anova: efficiently quantify- ing hyperparameter importance in arbitrary subspaces,”arXiv preprint arXiv:2304.10255, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  10. [10]

    Bayesmark: Benchmark framework to compare bayesian optimization methods on real ml tasks,

    R. D. Turner and D. Eriksson, “Bayesmark: Benchmark framework to compare bayesian optimization methods on real ml tasks,” https://github. com/uber/bayesmark, 2019, accessed 2025-09-14

    2019

  11. [11]

    Surrogate nas benchmarks: Going beyond the limited search spaces of tabular nas benchmarks,

    A. Zela, J. Siems, L. Zimmer, J. Lukasik, M. Keuper, and F. Hutter, “Surrogate nas benchmarks: Going beyond the limited search spaces of tabular nas benchmarks,”arXiv preprint arXiv:2008.09777, 2020

    work page arXiv 2008

  12. [12]

    Visualize sweep results,

    Weights & Biases, “Visualize sweep results,” https://docs.wandb.ai/ guides/sweeps/visualize-sweep-results/, 2025, accessed: 2025-09-18

    2025

  13. [13]

    Optuna: A next- generation hyperparameter optimization framework,

    T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, “Optuna: A next- generation hyperparameter optimization framework,” inProceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, 2019, pp. 2623–2631

    2019

  14. [14]

    Smac3: A versatile bayesian optimization package for hyperparameter optimization,

    M. Lindauer, K. Eggensperger, M. Feurer, A. Biedenkapp, D. Deng, C. Benjamins, T. Ruhkopf, R. Sass, and F. Hutter, “Smac3: A versatile bayesian optimization package for hyperparameter optimization,”Jour- nal of Machine Learning Research, vol. 23, no. 54, pp. 1–9, 2022

    2022

  15. [15]

    Uq-guided hyperparameter optimization for iterative learners,

    J. Liu, F. Zhang, J. Guan, and X. Shen, “Uq-guided hyperparameter optimization for iterative learners,”Advances in Neural Information Processing Systems, vol. 37, pp. 386–415, 2024

    2024

  16. [16]

    Improving hyperparameter optimization with checkpointed model weights,

    N. Mehta, J. Lorraine, S. Masson, R. Arunachalam, Z. P. Bhat, J. Lucas, and A. G. Zachariah, “Improving hyperparameter optimization with checkpointed model weights,” inEuropean Conference on Computer Vision. Springer, 2024, pp. 75–96

    2024

  17. [17]

    Scalable gaussian process-based transfer surrogates for hyperparameter optimization,

    M. Wistuba, N. Schilling, and L. Schmidt-Thieme, “Scalable gaussian process-based transfer surrogates for hyperparameter optimization,”Ma- chine Learning, vol. 107, no. 1, pp. 43–78, 2018

    2018

  18. [18]

    Multi-task bayesian optimiza- tion,

    K. Swersky, J. Snoek, and R. P. Adams, “Multi-task bayesian optimiza- tion,”Advances in neural information processing systems, vol. 26, 2013

    2013

  19. [19]

    arXiv preprint arXiv:2503.05106

    R. Wang, I. Nabney, and M. Golbabaee, “Grouped sequential optimiza- tion strategy–the application of hyperparameter importance assessment in deep learning,”arXiv preprint arXiv:2503.05106, 2025

    work page arXiv 2025

  20. [20]

    Ackley,A connectionist machine for genetic hillclimbing

    D. Ackley,A connectionist machine for genetic hillclimbing. Springer science & business media, 2012, vol. 28

    2012

  21. [21]

    Generalized descent for global optimization,

    A. Griewank, “Generalized descent for global optimization,”JOTA, vol. 34, p. 15, 1985

    1985

  22. [22]

    Systems of extremal control,

    L. A. Rastrigin, “Systems of extremal control,”Nauka, 1974

    1974

  23. [23]

    Opentuner: An extensible framework for program autotuning,

    J. Anselet al., “Opentuner: An extensible framework for program autotuning,” inInternational Conference on Parallel Architectures and Compilation Techniques. IEEE, 2014, pp. 303–315

    2014

  24. [24]

    Scalable global optimiza- tion via local bayesian optimization,

    D. Eriksson, D. Bindel, and C. A. Shoemaker, “Scalable global optimiza- tion via local bayesian optimization,” inAdvances in Neural Information Processing Systems, 2019

    2019

  25. [25]

    Scikit-optimize/scikit-optimize,

    T. Head, MechCoder, G. Louppe, I. Shcherbatyi, E. Fokoue et al., “Scikit-optimize/scikit-optimize,” https://scikit-optimize.github. io/, 2018

    2018

  26. [26]

    Nas-bench-101: Towards reproducible neural architecture search,

    C. Ying, A. Klein, E. Christiansen, E. Real, K. Murphy, and F. Hutter, “Nas-bench-101: Towards reproducible neural architecture search,” in International conference on machine learning. PMLR, 2019, pp. 7105– 7114

    2019

  27. [27]

    Nas-bench-201: Extending the scope of repro- ducible neural architecture search,

    X. Dong and Y . Yang, “Nas-bench-201: Extending the scope of repro- ducible neural architecture search,”arXiv preprint arXiv:2001.00326, 2020

    work page arXiv 2001

  28. [28]

    DARTS: Differentiable Architecture Search

    H. Liu, K. Simonyan, and Y . Yang, “Darts: Differentiable architecture search,”arXiv preprint arXiv:1806.09055, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  29. [29]

    Fast bayesian optimization of machine learning hyperparameters on large datasets,

    A. Klein, S. Falkner, J. T. Springenberg, and F. Hutter, “Fast bayesian optimization of machine learning hyperparameters on large datasets,” in Proceedings of the 20th International Conference on Artificial Intelli- gence and Statistics (AISTATS), 2017

    2017