{"id":"bcc6c30d-8505-4d99-b815-aea82bc88703","arxiv_id":"2507.20446","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"BOASF combines Bayesian optimization with adaptive successive filtering and softmax resource allocation to speed up model selection and hyperparameter optimization in automatic machine learning.","lead":"This paper presents BOASF, an automatic machine learning algorithm that combines Bayesian optimization with adaptive successive filtering to select models and tune hyperparameters under a time budget. It reports that BOASF beats AutoSklearn on 23 of 30 datasets for model selection and outperforms Hyperopt-full and BOHB for hyperparameter optimization.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The empirical evidence for BOASF's superiority is internally inconsistent: the 4-hour Table 1 count does not match the stated 23/30, and Tables 2–3 show unexplained non-monotonic swings. The anytime-performance claim is not yet supported.","rationale":"The paper's contribution is a heuristic combination of Bayesian optimization, successive filtering, and softmax resource allocation, so its validity rests on the reported experiments. I looked for the weakest point in that empirical chain. The subspace-partitioning concern named by the reader is real but secondary: for a continuous hyperparameter, any configuration, including the global optimum, lies in one of the closed sub-spaces, so the claim that 'every sub-space is poor' is not immediate. The more pressing weakness is the internal consistency of the result tables. The 4-hour statement in Section 5.2 does not match Table 1, and the 2-hour/4-hour non-monotonic entries in Tables 2–3 are unexplained and would require very high variance or an error. A corrected table with per-seed logs or released code would settle the issue. This is not an accusation of misconduct; it is a specific, checkable defect in the evidence. The verdict remains conditional: if raw results and code are provided and the discrepancies disappear, the paper can be accepted; if not, the central empirical claim should be weakened. The reader already reached a conditional verdict, so I do not propose changing it.","tokens_in":14153,"tokens_out":15699,"duration_ms":159964,"concrete_test":"Obtain the authors' code and per-seed logs, or re-run BOASF and AutoSklearn on the same 30 datasets with R=4h, r=3, c=2, and 5 seeds. Then recompute the win/tie/loss count from the 4-hour columns in Table 1 using the paper's exact tie rule; if the count is 22/4 rather than 23/3, the text must be corrected. Also re-run the hyperparameter-optimization experiments on openml279 and openml8 to check whether the large Table 2/3 non-monotonic values reproduce. If they reproduce, per-seed error bars are required before any anytime-superiority claim can be assessed; if they do not, the published tables contain an error.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is an empirical one: BOASF provides better anytime performance than AutoSklearn and Hyperopt-full on 30 datasets. The load-bearing condition is that the result tables are accurate and self-consistent. That condition currently fails in two concrete places. First, Section 5.2 states that with R=4h BOASF outperforms AutoSklearn on 23/30 datasets and ties on 3, but row-by-row comparison of the 4-hour columns in Table 1 yields 22 BOASF wins, 4 ties, and 4 AutoSklearn wins. The 2-hour count (20 wins, 3 ties) matches the text, so the 4-hour claim appears to contain an error. Second, the anytime claim is contradicted by entries in Tables 2–3 unless the runs are extremely noisy: for logistic regression on openml279, BOASF drops from 0.823 at 2h to 0.097 at 4h, while for random forest on the same dataset it jumps from 0.000 at 2h to 0.802 at 4h; BOHB on openml8 moves from 0.061 at 2h to 0.677 at 4h in Table 2 and from 0.743 to 0.082 at 4h in Table 3. These swings are not explained, and no standard deviations or per-seed results are given. Because the superiority claim is entirely empirical, the missing code/data and the absence of significance testing make the central conclusion unsupported as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BOASF, a multi-armed bandit framework for automated machine learning that combines per-arm Bayesian optimization (TPE) with adaptive successive filtering based on Gaussian UCB scores and softmax-based resource allocation. The method is applied to model selection, where each model is an arm, and to hyperparameter optimization, where each sub-space from a k-interval partition of the hyperparameter space is an arm. The authors report experiments on 30 classification datasets comparing BOASF with SelectBest, RandomForest&TPE, and AutoSklearn for model selection, and with Hyperopt-full and BOHB for hyperparameter optimization, concluding that BOASF achieves better anytime performance under 2-hour and 4-hour budgets.","tokens_in":14557,"tokens_out":6349,"duration_ms":64273,"significance":"If the empirical results are reliable, BOASF offers a simple and parallelizable alternative to existing AutoML systems, and the combination of Bayesian optimization with adaptive arm filtering and resource allocation is a reasonable design that could be useful in practice. The paper includes a sensitivity analysis of the method's own hyperparameters (Section 5.4) and evaluates on a broad set of datasets, which are strengths. However, the central claim is empirical, and the evidence as presented is weakened by the absence of error bars or significance tests, unexplained non-monotonic entries in the tables, and an algorithm pseudocode that is not executable as written. The 23/30 win count for the 4-hour model-selection comparison is consistent with Table 1, but other load-bearing details need correction before the contribution can be accepted.","major_comments":[{"comment":"In Algorithm 1, line 17 sets B = {} immediately before the loop 'for a ∈ B do' on line 18, so the loop body is never executed, and line 22 (B = B) is a tautology. This contradicts the prose in Section 3.2, where arms are filtered from the current set into a new set, and it makes the pseudocode non-executable. Because adaptive filtering is the core mechanism of the proposed method, this needs to be corrected (for example, by using a separate variable for the surviving arm set) before the method can be reproduced.","section":"Algorithm 1 (Section 3.3)"},{"comment":"The anytime-performance claim is supported only by mean balanced-accuracy scores from 5 runs, with no standard deviations, per-seed results, or significance tests. Several entries in Tables 2 and 3 are non-monotonic in the time budget in ways that are hard to explain without high run-to-run variance: BOASF on openml279 drops from 0.823 at 2h to 0.097 at 4h for logistic regression (Table 2) yet rises from 0.000 at 2h to 0.802 at 4h for random forest (Table 3); BOHB on openml8 moves from 0.061 at 2h to 0.677 at 4h in Table 2 and from 0.743 at 2h to 0.082 at 4h in Table 3. I verified that the 23/30 and 3-tie count for the 4-hour model-selection comparison does match Table 1, so that specific count is not the issue; the lack of error bars and the anomalous entries are. Without seed-level data and a significance test, the conclusion that BOASF 'achieves better anytime performance under various time budgets' is not supported.","section":"Section 5.2 and Tables 2-3"},{"comment":"The hyperparameter-optimization variant partitions each hyperparameter into k=2 equal-length intervals and treats each product sub-space as an arm. This silently assumes that at least one sub-space contains configurations close enough to the true optimum that the best arm can win the final evaluation. If the optimal region is small or straddles a partition boundary, all arms can be poor and the method cannot recover. No experiment varies k, no comparison with alternative partitionings is reported, and the sensitivity study in Section 5.4 varies only c and r. This assumption should be stated explicitly and tested, since it is load-bearing for the claim that BOASF outperforms Hyperopt-full in 23/30 cases.","section":"Section 4.2"},{"comment":"Several numerical values are identical between the 2-hour and 4-hour columns (for example, Table 2 openml714 is 0.602 for all methods at both budgets, and Table 2 openml764 is 0.500 for all methods), which is suspicious for stochastic optimization procedures and suggests either a table-generation error or that the underlying runs were not truly varying. Additionally, the text describes BOASF as 'statistically' outperforming baselines, but no statistical test is described anywhere and no variance information is given. The authors should provide per-seed results, standard deviations, and a clearly specified statistical comparison (for example, a Wilcoxon signed-rank test over datasets) to justify this wording.","section":"Section 5.2 and Section 5.3"}],"minor_comments":[{"comment":"When all Gaussian UCB scores are equal, the denominator of Equation (7) is zero; a tie-breaking rule or a small regularization term is needed for this edge case.","section":"Section 3.2, Eq. (7)"},{"comment":"The assignment 'B = B' on line 22 appears to be a typo; even if the filtering loop is corrected, this assignment should set the loop variable to the newly filtered arm set rather than to itself.","section":"Algorithm 1, line 22"},{"comment":"There are several typos in this section, including 'demenstrate' and 'hyperparammeter', and 'RandomFrest&TPE' appears in Section 5.2; the manuscript should be proofread.","section":"Section 5.3"},{"comment":"The text says 'we performed 5 runs for each AutoML methods' but it is not clear whether the reported scores are averages over those 5 runs or over the cross-validation folds, and no confidence intervals are reported.","section":"Section 5.1"},{"comment":"No dataset characteristics (number of samples, features, classes) are listed, which makes it harder to interpret the per-dataset differences; adding these in an appendix would help.","section":"Tables 1-3"}],"recommendation":"major_revision","confidential_remarks":"The paper's central contribution is empirical, and the current manuscript does not provide enough evidence to support the strong anytime-performance claims. The algorithm pseudocode bug and the suspicious table entries can be fixed with a careful revision, and the authors should be asked to release code and per-seed results. I do not see a novelty disclosure issue, but the fit with a machine-learning journal is appropriate if the empirical claims can be substantiated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe core idea is a sensible hybrid: treat each model (or each hyperparameter sub-space) as an arm, use TPE per arm to propose configurations, score arms with a UCB mean-variance statistic, filter probabilistically with a MinMaxScaler, and allocate next-round resources by softmax of UCB scores. That combination is not in the cited prior work, and the sub-space partitioning for HPO is a genuinely new twist. As a heuristic pipeline for speeding up AutoML on tabular data, it is plausible and worth taking seriously.\n\nUnfortunately, the empirical case is not currently written up at a publishable standard. Two concrete problems. Algorithm 1 sets B to empty at line 17 and then iterates over B in lines 18-21; that is a bug in the pseudocode that contradicts the prose (the paragraph around Equation 7 says the worst arm is filtered, and the algorithm should be iterating over the pre-filter set). The table counts don't match the text: Section 5.2 says 23/30 wins at 4h, but Table 1 row-by-row gives 22 BOASF wins, 4 ties, 4 AutoSklearn wins. More worrying, Tables 2-3 contain non-monotonic swings that look like noise or implementation issues: logistic regression on openml279 goes from 0.823 at 2h to 0.097 at 4h, random forest on the same dataset from 0.000 to 0.802, and BOHB on openml8 swings wildly between tables. No per-seed results or standard deviations are given, and the paper claims \"statistically\" better without tests. Code and data are not provided.\n\nThese are not minor blemishes; the central claim is empirical superiority, and the numbers as presented do not support it. I would not take the superiority claims at face value. But the algorithmic idea is not fatally flawed, and the sensitivity analysis for c and r in Figure 3 at least shows the method is not brittle to its two main hyperparameters. The weakest assumption, which the paper does not analyze, is that the HPO sub-space partitioning (two equal intervals per dimension) preserves a good-enough region; that deserves at least a discussion or a comparison with other partition counts.\n\nWho is this for? Practitioners who want a solid baseline for model selection/HPO on tabular problems could find the method useful once it is cleaned up. Researchers working on bandit-based AutoML will want to know the combination, but they need to be able to trust the experiments. I would send this to peer review after a major revision that fixes Algorithm 1, reconciles the table counts, adds seeded runs with error bars or significance tests, and releases code. As it stands, it's not acceptable.","headline":"Plausible hybrid AutoML idea, but the empirical claims are undermined by a pseudocode bug, inconsistent table counts, and unexplained performance swings.","tokens_in":15026,"tokens_out":2088,"would_cite":false,"duration_ms":23214,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Combining Bayesian optimization with adaptive successive filtering speeds up AutoML across 30 datasets.","keywords":["automatic machine learning","model selection","hyperparameter optimization","Bayesian optimization","successive filtering","multi-armed bandit","Gaussian upper confidence bound","subspace partitioning"],"falsifier":"Run BOASF on a synthetic optimization problem whose optimum sits exactly on the midpoint of a hyperparameter interval (a partition boundary), with all other settings unchanged; if the final balanced accuracy trails Hyperopt-full's, the subspace-partitioning assumption is the load-bearing part.","tokens_in":13979,"feed_emoji":"⚙️","tokens_out":6716,"duration_ms":72843,"temperature":0.7,"pith_summary":"This paper proposes BOASF, a single algorithm that handles both model selection and hyperparameter optimization by treating each candidate model (or each hyperparameter sub-space) as an arm in a multi-armed bandit. In each evaluation round, Bayesian optimization picks configurations for each arm, a Gaussian upper-confidence-bound score is converted into a probability of survival, and a softmax rule doles out the next round's time budget. The claim is that this combination finds good models faster than the standard AutoML pipelines, and the reported experiments support that claim: BOASF beats AutoSklearn on 20 of 30 datasets with a 2-hour budget and 23 of 30 with 4 hours, and beats Hyperopt-full on 23 of 30 for hyperparameter optimization. A practitioner with a fixed, tight time budget would get a better final model with the same wall-clock allowance.","feed_headline":"Adaptive filtering beats AutoSklearn on 23 of 30 datasets","feed_subtitle":"Probabilistic filtering plus per-arm Bayesian optimization wins on most of 30 classification benchmarks.","key_machinery":"The machinery is the BOASF loop itself: TPE-style Bayesian optimization selects configurations for each arm; a Gaussian UCB score (mean plus c times standard deviation divided by sqrt(N)) summarizes each arm's reward history; MinMax scaling turns those scores into advancement probabilities; and a softmax over the same scores determines resource allocation. The crucial property is that the MinMaxScaler guarantees the worst arm is filtered and the best arm survives every round, while the softmax guarantees that surviving arms get resources proportional to their score. The subspace partitioner converts a continuous hyperparameter space into a finite set of bandit arms.","core_discovery":"The central discovery is that adaptive successive filtering—probabilistic rather than fixed-ratio—combined with per-arm Bayesian optimization yields better anytime performance than current AutoML methods. Where Hyperband discards half the arms in a brute-force manner, BOASF computes a Gaussian UCB for each arm from the mean and variance of its evaluation rewards, scales those scores to advancement probabilities via MinMaxScaler, and allocates the next round's resources with a softmax over the same scores. For hyperparameter optimization the paper adds a subspace-partitioning trick: split each hyperparameter range into two intervals, treat each Cartesian product of intervals as an arm, and run the same bandit on those arms. The experiments report consistent wins over AutoSklearn, Hyperopt-full, and BOHB under 2- and 4-hour budgets.","pith_inferences":["If the subspace-partitioning trick is what drives the hyperparameter gains, then the number of intervals per dimension is a sensitive knob; testing k=1,3,4 intervals on the same benchmarks would reveal whether the method's edge comes from partitioning or from the bandit filtering itself.","A natural extension the paper does not explore is applying the same adaptive filtering to the configurations themselves, which could make the method competitive on very high-dimensional hyperparameter spaces where a fixed partition explodes into 2^n arms.","The Gaussian UCB with c=2 assumes rewards are roughly Gaussian and bounded in [0,1]; on unbounded loss metrics the method would need a monotone rescaling, and testing that variant is a concrete next step."],"forward_implications":["Under a fixed wall-clock budget, a practitioner using BOASF should obtain a model with accuracy at least as high as AutoSklearn's on most of the 30 tested classification datasets.","BOASF's probabilistic filtering avoids Hyperband's fixed half-discard, so it can keep a promising arm alive even when its early evaluations are noisy.","The same bandit formulation covers both model selection and hyperparameter optimization, so one implementation can replace two separate search tools.","Because arms are evaluated independently, BOASF can be parallelized across cores with the total CPU time as the shared resource.","The subspace-partitioning variant for hyperparameter optimization suggests that searching disjoint regions of the space with separate Bayesian optimizers can beat a single Bayesian optimizer over the whole space."],"supporting_citations":[{"why":"Hyperband is the successive-halving baseline whose fixed half-discard motivates ASF's adaptive filtering.","marker":"[19]"},{"why":"TPE is the Bayesian optimization method used to choose configurations for each arm in each round.","marker":"[3]"},{"why":"AutoSklearn is the main model-selection baseline; BOASF is compared against it on 30 datasets.","marker":"[12]"},{"why":"BOHB is the hyperparameter-optimization baseline that combines Hyperband and Bayesian optimization; BOASF is compared against it.","marker":"[11]"},{"why":"Hyperopt provides the TPE implementation used as the full-space baseline (Hyperopt-full).","marker":"[5]"},{"why":"Gaussian UCB supplies the upper-confidence-bound construction that BOASF uses to score and filter arms.","marker":"[23]"},{"why":"Random search is the standard baseline that motivates the use of Bayesian optimization over grid or random search.","marker":"[4]"}],"fun_headline_variants":["Adaptive filtering speeds up AutoML and beats AutoSklearn","Fast AutoML: Bayesian optimization plus adaptive filtering","BOASF: adaptive filtering cuts AutoML time","Probabilistic filtering wins on 23 of 30 datasets","Bayesian bandits for faster AutoML"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"For hyperparameter optimization, the method cuts every hyperparameter range into two equal intervals and assumes the best configuration lies inside at least one of the resulting sub-spaces.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive filtering speeds up AutoML and beats AutoSklearn","Fast AutoML: Bayesian optimization plus adaptive filtering","BOASF: adaptive filtering cuts AutoML time","Probabilistic filtering wins on 23 of 30 datasets","Bayesian bandits for faster AutoML"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00062,"raw_usage":{"total_tokens":2867,"prompt_tokens":932,"completion_tokens":1935,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":1868}},"tokens_in":548,"tokens_out":1935,"duration_ms":16382,"temperature":1.0,"reasoning_tokens":1868,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T13:33:04.976008+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run BOASF on a synthetic optimization problem whose optimum sits exactly on the midpoint of a hyperparameter interval (a partition boundary), with all other settings unchanged; if the final balanced accuracy trails Hyperopt-full's, the subspace-partitioning assumption is the load-bearing part.","supporting_citations":[{"cited_title":"Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar, ‘Hyperband: A novel bandit-based approach to hyperparameter optimization’, J","cited_arxiv_id":null,"evidence_quote":"Hyperband is the successive-halving baseline whose fixed half-discard motivates ASF's adaptive filtering."},{"cited_title":"2546–2554, Granada, Spain, (2011)","cited_arxiv_id":null,"evidence_quote":"TPE is the Bayesian optimization method used to choose configurations for each arm in each round."},{"cited_title":"2962–2970, Montr´eal, Canada, (2015)","cited_arxiv_id":null,"evidence_quote":"AutoSklearn is the main model-selection baseline; BOASF is compared against it on 30 datasets."},{"cited_title":"Dy and Andreas Krause, volume 80 of Proceedings of Machine Learn- ing Research, pp","cited_arxiv_id":null,"evidence_quote":"BOHB is the hyperparameter-optimization baseline that combines Hyperband and Bayesian optimization; BOASF is compared against it."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Hyperopt provides the TPE implementation used as the full-space baseline (Hyperopt-full)."},{"cited_title":"1015–1022, USA, (2010)","cited_arxiv_id":null,"evidence_quote":"Gaussian UCB supplies the upper-confidence-bound construction that BOASF uses to score and filter arms."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Random search is the standard baseline that motivates the use of Bayesian optimization over grid or random search."}],"review_version":1}