Pith. sign in

REVIEW 4 major objections 5 minor 28 references

BOASF: A Unified Framework for Speeding up Automatic Machine Learning via Adaptive Successive Filtering

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Combining Bayesian optimization with adaptive successive filtering speeds up AutoML across 30 datasets.

desk verdict Plausible hybrid AutoML idea, but the empirical claims are undermined by a pseudocode bug, inconsistent table counts, and unexplained performance swings. read the letter →

arxiv 2507.20446 v2 pith:QXDA3PVY submitted 2025-07-28 cs.LG

Guanghui Zhu , Xin Fang , Feng Cheng , Lei Wang , Wenzhong Chen , Chunfeng Yuan , Yihua Huang This is my paper · ORCID
classification cs.LG
keywords automaticmachinelearningmodelselectionhyperparameteroptimizationBayesiansuccessivefilteringmulti-armedbanditGaussianupperconfidenceboundsubspacepartitioning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes BOASF, a single algorithm that handles both model selection and hyperparameter optimization by treating each candidate model (or each hyperparameter sub-space) as an arm in a multi-armed bandit. In each evaluation round, Bayesian optimization picks configurations for each arm, a Gaussian upper-confidence-bound score is converted into a probability of survival, and a softmax rule doles out the next round's time budget. The claim is that this combination finds good models faster than the standard AutoML pipelines, and the reported experiments support that claim: BOASF beats AutoSklearn on 20 of 30 datasets with a 2-hour budget and 23 of 30 with 4 hours, and beats Hyperopt-full on 23 of 30 for hyperparameter optimization. A practitioner with a fixed, tight time budget would get a better final model with the same wall-clock allowance.

What carries the argument

The machinery is the BOASF loop itself: TPE-style Bayesian optimization selects configurations for each arm; a Gaussian UCB score (mean plus c times standard deviation divided by sqrt(N)) summarizes each arm's reward history; MinMax scaling turns those scores into advancement probabilities; and a softmax over the same scores determines resource allocation. The crucial property is that the MinMaxScaler guarantees the worst arm is filtered and the best arm survives every round, while the softmax guarantees that surviving arms get resources proportional to their score. The subspace partitioner converts a continuous hyperparameter space into a finite set of bandit arms.

What would settle it

Run BOASF on a synthetic optimization problem whose optimum sits exactly on the midpoint of a hyperparameter interval (a partition boundary), with all other settings unchanged; if the final balanced accuracy trails Hyperopt-full's, the subspace-partitioning assumption is the load-bearing part.

Watch

Extended reading notes

Core claim

The central discovery is that adaptive successive filtering—probabilistic rather than fixed-ratio—combined with per-arm Bayesian optimization yields better anytime performance than current AutoML methods. Where Hyperband discards half the arms in a brute-force manner, BOASF computes a Gaussian UCB for each arm from the mean and variance of its evaluation rewards, scales those scores to advancement probabilities via MinMaxScaler, and allocates the next round's resources with a softmax over the same scores. For hyperparameter optimization the paper adds a subspace-partitioning trick: split each hyperparameter range into two intervals, treat each Cartesian product of intervals as an arm, and run the same bandit on those arms. The experiments report consistent wins over AutoSklearn, Hyperopt-full, and BOHB under 2- and 4-hour budgets.

Load-bearing premise

For hyperparameter optimization, the method cuts every hyperparameter range into two equal intervals and assumes the best configuration lies inside at least one of the resulting sub-spaces.

Editorial extensions

If this is right

  • Under a fixed wall-clock budget, a practitioner using BOASF should obtain a model with accuracy at least as high as AutoSklearn's on most of the 30 tested classification datasets.
  • BOASF's probabilistic filtering avoids Hyperband's fixed half-discard, so it can keep a promising arm alive even when its early evaluations are noisy.
  • The same bandit formulation covers both model selection and hyperparameter optimization, so one implementation can replace two separate search tools.
  • Because arms are evaluated independently, BOASF can be parallelized across cores with the total CPU time as the shared resource.
  • The subspace-partitioning variant for hyperparameter optimization suggests that searching disjoint regions of the space with separate Bayesian optimizers can beat a single Bayesian optimizer over the whole space.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the subspace-partitioning trick is what drives the hyperparameter gains, then the number of intervals per dimension is a sensitive knob; testing k=1,3,4 intervals on the same benchmarks would reveal whether the method's edge comes from partitioning or from the bandit filtering itself.
  • A natural extension the paper does not explore is applying the same adaptive filtering to the configurations themselves, which could make the method competitive on very high-dimensional hyperparameter spaces where a fixed partition explodes into 2^n arms.
  • The Gaussian UCB with c=2 assumes rewards are roughly Gaussian and bounded in [0,1]; on unbounded loss metrics the method would need a monotone rescaling, and testing that variant is a concrete next step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes BOASF, a multi-armed bandit framework for automated machine learning that combines per-arm Bayesian optimization (TPE) with adaptive successive filtering based on Gaussian UCB scores and softmax-based resource allocation. The method is applied to model selection, where each model is an arm, and to hyperparameter optimization, where each sub-space from a k-interval partition of the hyperparameter space is an arm. The authors report experiments on 30 classification datasets comparing BOASF with SelectBest, RandomForest&TPE, and AutoSklearn for model selection, and with Hyperopt-full and BOHB for hyperparameter optimization, concluding that BOASF achieves better anytime performance under 2-hour and 4-hour budgets.

Significance. If the empirical results are reliable, BOASF offers a simple and parallelizable alternative to existing AutoML systems, and the combination of Bayesian optimization with adaptive arm filtering and resource allocation is a reasonable design that could be useful in practice. The paper includes a sensitivity analysis of the method's own hyperparameters (Section 5.4) and evaluates on a broad set of datasets, which are strengths. However, the central claim is empirical, and the evidence as presented is weakened by the absence of error bars or significance tests, unexplained non-monotonic entries in the tables, and an algorithm pseudocode that is not executable as written. The 23/30 win count for the 4-hour model-selection comparison is consistent with Table 1, but other load-bearing details need correction before the contribution can be accepted.

major comments (4)
  1. [Algorithm 1 (Section 3.3)] In Algorithm 1, line 17 sets B = {} immediately before the loop 'for a ∈ B do' on line 18, so the loop body is never executed, and line 22 (B = B) is a tautology. This contradicts the prose in Section 3.2, where arms are filtered from the current set into a new set, and it makes the pseudocode non-executable. Because adaptive filtering is the core mechanism of the proposed method, this needs to be corrected (for example, by using a separate variable for the surviving arm set) before the method can be reproduced.
  2. [Section 5.2 and Tables 2-3] The anytime-performance claim is supported only by mean balanced-accuracy scores from 5 runs, with no standard deviations, per-seed results, or significance tests. Several entries in Tables 2 and 3 are non-monotonic in the time budget in ways that are hard to explain without high run-to-run variance: BOASF on openml279 drops from 0.823 at 2h to 0.097 at 4h for logistic regression (Table 2) yet rises from 0.000 at 2h to 0.802 at 4h for random forest (Table 3); BOHB on openml8 moves from 0.061 at 2h to 0.677 at 4h in Table 2 and from 0.743 at 2h to 0.082 at 4h in Table 3. I verified that the 23/30 and 3-tie count for the 4-hour model-selection comparison does match Table 1, so that specific count is not the issue; the lack of error bars and the anomalous entries are. Without seed-level data and a significance test, the conclusion that BOASF 'achieves better anytime performance under various time budgets' is not supported.
  3. [Section 4.2] The hyperparameter-optimization variant partitions each hyperparameter into k=2 equal-length intervals and treats each product sub-space as an arm. This silently assumes that at least one sub-space contains configurations close enough to the true optimum that the best arm can win the final evaluation. If the optimal region is small or straddles a partition boundary, all arms can be poor and the method cannot recover. No experiment varies k, no comparison with alternative partitionings is reported, and the sensitivity study in Section 5.4 varies only c and r. This assumption should be stated explicitly and tested, since it is load-bearing for the claim that BOASF outperforms Hyperopt-full in 23/30 cases.
  4. [Section 5.2 and Section 5.3] Several numerical values are identical between the 2-hour and 4-hour columns (for example, Table 2 openml714 is 0.602 for all methods at both budgets, and Table 2 openml764 is 0.500 for all methods), which is suspicious for stochastic optimization procedures and suggests either a table-generation error or that the underlying runs were not truly varying. Additionally, the text describes BOASF as 'statistically' outperforming baselines, but no statistical test is described anywhere and no variance information is given. The authors should provide per-seed results, standard deviations, and a clearly specified statistical comparison (for example, a Wilcoxon signed-rank test over datasets) to justify this wording.
minor comments (5)
  1. [Section 3.2, Eq. (7)] When all Gaussian UCB scores are equal, the denominator of Equation (7) is zero; a tie-breaking rule or a small regularization term is needed for this edge case.
  2. [Algorithm 1, line 22] The assignment 'B = B' on line 22 appears to be a typo; even if the filtering loop is corrected, this assignment should set the loop variable to the newly filtered arm set rather than to itself.
  3. [Section 5.3] There are several typos in this section, including 'demenstrate' and 'hyperparammeter', and 'RandomFrest&TPE' appears in Section 5.2; the manuscript should be proofread.
  4. [Section 5.1] The text says 'we performed 5 runs for each AutoML methods' but it is not clear whether the reported scores are averages over those 5 runs or over the cross-validation folds, and no confidence intervals are reported.
  5. [Tables 1-3] No dataset characteristics (number of samples, features, classes) are listed, which makes it harder to interpret the per-dataset differences; adding these in an appendix would help.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity found; the paper's claims are empirical comparisons against external baselines.

full rationale

The paper's central claims are empirical: BOASF is compared against AutoSklearn, Hyperopt-full, and BOHB on external benchmark datasets. The method's components (Bayesian optimization, Gaussian-UCB-based adaptive filtering, and Softmax resource allocation) are stated design choices, not derived from the reported outcomes. The hyperparameters c and r are fixed before the experiments and their sensitivity is examined in Section 5.4, so they are not fitted to the test scores. The sub-space partitioning in Section 4.2 is a heuristic design decision rather than a fitted parameter, and it is evaluated against whole-space TPE and BOHB. No load-bearing result is justified by self-citation, and no prediction is equivalent to its inputs by construction. Concerns about table consistency, non-monotonic anytime results, and missing significance testing are correctness and reproducibility issues, not circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The method relies on several hand-set parameters (c, r, k) and four heuristic modeling choices. None of these are fitted to the test data, but they are not derived either, so the contribution is an empirical heuristic rather than a principled algorithm.

free parameters (3)
  • c (UCB exploration constant) = 2
    Set by hand in Section 5.1. It controls the weight of uncertainty in Equation 6. Sensitivity is checked in Section 5.4, but no per-dataset optimization is performed.
  • r (number of evaluation rounds) = 3
    Set by hand in Section 5.1. It divides the time budget into equal parts. Sensitivity is checked in Section 5.4.
  • k (intervals per hyperparameter) = 2
    Chosen in Section 5.3 to create sub-spaces. The paper provides no analysis of the impact of k on the search.
assumptions (5)
  • domain assumption Gaussian UCB (Equation 6) is an appropriate performance metric for arm quality.
    Assumes the mean and variance of a small number of evaluations capture an arm's potential. No distributional justification is given.
  • domain assumption TPE is a suitable Bayesian optimizer for selecting configurations within each arm.
    Adopted from Bergstra et al. (2011). The paper relies on its effectiveness without new analysis.
  • ad hoc to paper MinMaxScaler (Equation 7) converts UCB scores into valid advancing probabilities.
    This is the core filtering rule: the worst arm is always discarded and the best always retained. No theoretical support is provided.
  • ad hoc to paper Softmax (Equation 8) resource allocation improves search efficiency.
    The paper uses the softmax of UCB scores to split round budgets. The benefit is asserted, not derived.
  • ad hoc to paper Partitioning each hyperparameter into two equal intervals preserves the optimal region.
    Introduced in Section 4.2. No formal or empirical analysis of the risk of splitting the optimum is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BOASF: A Unified Framework for Speeding up Automatic Machine Learning via Adaptive Successive Filtering." pith.science (2026). https://pith.science/paper/QXDA3PVY

@misc{pith2026250720446,
  author       = {Pith},
  title        = {Pith review of: BOASF: A Unified Framework for Speeding up Automatic Machine Learning via Adaptive Successive Filtering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QXDA3PVY}},
  note         = {Machine review of arXiv:2507.20446}
}
read the original abstract

Machine learning has been making great success in many application areas. However, for the non-expert practitioners, it is always very challenging to address a machine learning task successfully and efficiently. Finding the optimal machine learning model or the hyperparameter combination set from a large number of possible alternatives usually requires considerable expert knowledge and experience. To tackle this problem, we propose a combined Bayesian Optimization and Adaptive Successive Filtering algorithm (BOASF) under a unified multi-armed bandit framework to automate the model selection or the hyperparameter optimization. Specifically, BOASF consists of multiple evaluation rounds in each of which we select promising configurations for each arm using the Bayesian optimization. Then, ASF can early discard the poor-performed arms adaptively using a Gaussian UCB-based probabilistic model. Furthermore, a Softmax model is employed to adaptively allocate available resources for each promising arm that advances to the next round. The arm with a higher probability of advancing will be allocated more resources. Experimental results show that BOASF is effective for speeding up the model selection and hyperparameter optimization processes while achieving robust and better prediction performance than the existing state-of-the-art automatic machine learning methods. Moreover, BOASF achieves better anytime performance under various time budgets.

Figures

Figures reproduced from arXiv: 2507.20446 by the authors.

Figure 1
Figure 1. Overall workflow of a 4-round ASF [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The process of arm evaluation in each round the performance of learning models. Thus, to judge the quality of an arm in each round, it is necessary to evaluate one arm multiple times under the allocated resources. As shown in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The ratio of datasets which BOASF outperformed or tied with Hyperopt-full under different combinations of hyperparameters (c and r) in BOASF [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 26 canonical work pages

  1. [1]

    Learn., 47(2-3), 235–256, (May 2002)

    Peter Auer, Nicol `o Cesa-Bianchi, and Paul Fischer, ‘Finite-time analy- sis of the multiarmed bandit problem’,Mach. Learn., 47(2-3), 235–256, (May 2002)

    work page 2002

  2. [2]

    Adithya Balaji and Alexander Allen, ‘Benchmarking automatic ma- chine learning frameworks’, ArXiv, abs/1808.06492, (2018)

    work page Pith review arXiv 2018

  3. [3]

    2546–2554, Granada, Spain, (2011)

    James Bergstra, R ´emi Bardenet, Yoshua Bengio, and Bal´azs K´egl, ‘Al- gorithms for hyper-parameter optimization’, in Advances in Neural In- formation Processing Systems, pp. 2546–2554, Granada, Spain, (2011)

    work page 2011

  4. [4]

    James Bergstra and Yoshua Bengio, ‘Random search for hyper- parameter optimization’, Journal of Machine Learning Research , 13, 281–305, (2012)

    work page 2012

  5. [5]

    James Bergstra, Brent Komer, Chris Eliasmith, Dan Yamins, and David D Cox, ‘Hyperopt: a python library for model selection and hy- perparameter optimization’, Computational Science & Discovery, 8(1), 014008, (2015)

    work page 2015

  6. [6]

    Eric Brochu, Vlad M. Cora, and Nando de Freitas, ‘A tutorial on bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning’, CoRR, abs/1012.2599, (2010)

    arXiv 2010

  7. [7]

    S ´ebastien Bubeck and Nicol `o Cesa-Bianchi, ‘Regret analysis of stochastic and nonstochastic multi-armed bandit problems’, CoRR, abs/1204.5721, (2012)

    arXiv 2012

  8. [8]

    S ´ebastien Bubeck, R´emi Munos, and Gilles Stoltz, ‘Pure exploration in multi-armed bandits problems’, in Proceedings of International Con- ference on Algorithmic Learning Theory, pp. 23–37. Springer, (2009)

    work page 2009

Show all 28 references
  1. [9]

    1133–1141, (2015)

    Alexandra Carpentier and Michal Valko, ‘Simple regret for infinitely many armed bandits’, in Proceedings of the 32nd International Confer- ence on Machine Learning, ICML’15, pp. 1133–1141, (2015)

    2015

  2. [10]

    Chih-Chung Chang and Chih-Jen Lin, ‘LIBSVM: A library for support vector machines’, ACM TIST, 2(3), 27:1–27:27, (2011)

    2011

  3. [11]

    Dy and Andreas Krause, volume 80 of Proceedings of Machine Learn- ing Research, pp

    Stefan Falkner, Aaron Klein, and Frank Hutter, ‘BOHB: robust and effi- cient hyperparameter optimization at scale’, in Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stock- holmsm¨assan, Stockholm, Sweden, July 10-15, 2018 , eds., Jennifer G. ...

    2018

  4. [12]

    2962–2970, Montr´eal, Canada, (2015)

    Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Tobias Springenberg, Manuel Blum, and Frank Hutter, ‘Efficient and robust automated machine learning’, in Advances in Neural Information Pro- cessing Systems, pp. 2962–2970, Montr´eal, Canada, (2015)

    2015

  5. [13]

    Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown, ‘Sequential model-based optimization for general algorithm configuration’, in Pro- ceedings of the 5th Learning and Intelligent Optimization Conference , pp. 507–523, Rome, Italy, (2011)

    2011

  6. [14]

    Kevin G. Jamieson and Ameet Talwalkar, ‘Non-stochastic best arm identification and hyperparameter optimization’, in International Con- ference on Artificial Intelligence and Statistics, (2015)

    2015

  7. [15]

    Lawrence and Mark A

    Emilie Kaufmann, Olivier Capp ´e, and Aur´elien Garivier, ‘On bayesian upper confidence bounds for bandit problems’, in Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statis- tics, AISTATS 2012, La Palma, Canary Islands, Spain, April 21-23...

    2012

  8. [16]

    1137–1145

    Ron Kohavi, ‘A study of cross-validation and bootstrap for accuracy es- timation and model selection’, in Proceedings of the Fourteenth Inter- national Joint Conference on Artificial Intelligence, IJCAI 95, Montr´eal Qu´ebec, Canada, August 20-25 1995, 2 Volumes, pp. 1137–1145...

    1995

  9. [17]

    T.L Lai and Herbert Robbins, ‘Asymptotically efficient adaptive alloca- tion rules’, Adv. Appl. Math., 6(1), 4–22, (March 1985)

    1985

  10. [18]

    Haifang Li and Yingce Xia, ‘Infinitely many-armed bandits with budget constraints’, in AAAI Conference on Artificial Intelligence, (2017)

    2017

  11. [19]

    Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar, ‘Hyperband: A novel bandit-based approach to hyperparameter optimization’, J

    Lisha Li, Kevin G. Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar, ‘Hyperband: A novel bandit-based approach to hyperparameter optimization’, J. Mach. Learn. Res., 18, 185:1–185:52, (2017)

    2017

  12. [20]

    Olson, Nathan Bartley, Ryan J

    Randal S. Olson, Nathan Bartley, Ryan J. Urbanowicz, and Jason H. Moore, ‘Evaluation of a tree-based pipeline optimization tool for au- tomating data science’, inProceedings of the 2016 on Genetic and Evo- lutionary Computation Conference, pp. 485–492, Denver, CO, (2016)

    2016

  13. [21]

    18–23, Aachen, Germany, Germany, (2014)

    Parker Ridd and Christophe Giraud-Carrier, ‘Using metalearning to predict when parameter optimization is likely to improve classifica- tion accuracy’, in Proceedings of the 2014 International Conference on Meta-learning and Algorithm Selection, MLAS’14, pp. 18–23, Aachen, Germ...

    2014

  14. [22]

    Adams, ‘Practical bayesian optimization of machine learning algorithms’, in Advances in Neural Information Processing Systems , pp

    Jasper Snoek, Hugo Larochelle, and Ryan P. Adams, ‘Practical bayesian optimization of machine learning algorithms’, in Advances in Neural Information Processing Systems , pp. 2960–2968, Lake Tahoe, Nevada, (2012)

    2012

  15. [23]

    1015–1022, USA, (2010)

    Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger, ‘Gaussian process optimization in the bandit setting: No re- gret and experimental design’, in Proceedings of the 27th Interna- tional Conference on International Conference on Machine Learning , ICML’10, pp....

    2010

  16. [24]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto, Introduction to Reinforcement Learning, MIT Press, Cambridge, MA, USA, 1st edn., 1998

    1998

  17. [25]

    Chris Thornton, Frank Hutter, Holger H. Hoos, and Kevin Leyton- Brown, ‘Auto-weka: combined selection and hyperparameter optimiza- tion of classification algorithms’, in Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 847...

    2013

  18. [26]

    van Rijn, Bernd Bischl, and Luis Torgo, ‘Openml: networked science in machine learning’, CoRR, abs/1407.7722, (2014)

    Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis Torgo, ‘Openml: networked science in machine learning’, CoRR, abs/1407.7722, (2014)

    2014 arXiv

  19. [27]

    Ga ¨el Varoquaux, Lars Buitinck, Gilles Louppe, Olivier Grisel, Fabian Pedregosa, and Andreas Mueller, ‘Scikit-learn: Machine learning with- out learning the machinery’, GetMobile, 19(1), 29–33, (2015)

    2015

  20. [28]

    Jiazhuo Wang, Jason Xu, and Xuejun Wang, ‘Combination of hyper- band and bayesian optimization for hyperparameter optimization in deep learning’, CoRR, abs/1801.01596, (2018)

    2018 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.