{"id":"15a3ca0a-e905-4c4f-a78b-50635c7ae79e","arxiv_id":"2501.11414","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"On BBOB trajectory-based algorithm selection, classifier choice strongly affects accuracy, and feature-based and interval-based models outperform kernel, deep learning, and shapelet alternatives.","lead":"Researchers compared 17 time-series classifiers for selecting an optimization algorithm from short performance trajectories on the BBOB benchmark. They found that the choice of classifier matters a lot, with feature-based and interval-based models such as Summary and Time Series Forest giving the best accuracy.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Tuned LOIO results may be inflated: irace selects hyperparameters on all CMA-ES best trajectories, including instances later used as test folds, with no nested split described in Section 4.2.","rationale":"The paper's central contribution is an empirical benchmark, and its strongest load-bearing assumption is that tuned hyperparameters were chosen without using the instances later scored in LOIO. The text gives no evidence of a nested split: Section 3 says irace tunes on all CMA-ES best 2-generation trajectories, and Section 4.2 says the result is transferred to all other trajectory types. Under the reported LOIO protocol, the held-out instance has therefore already influenced hyperparameter choice. This is a standard selection-on-test bias. It does not require any assumption about the authors' intent; it follows from the described protocol. I agree with the reader's weakest_assumption. The default-model comparison is a useful, independent result: it shows large accuracy differences across classifier families and identifies Summary and Time Series Forest as strong in LOIO without tuning. If the nested rerun confirms the tuned numbers, the quantitative claims stand; if not, the gains over ELA in the abstract and conclusion are provisional. The LOPO result (LSTM best on average) is a secondary caveat to the abstract's broad 'feature/interval best' phrasing, but I would not make it the primary attack. No ad hominem is intended; this is a protocol-transparency issue. The reader's CONDITIONAL verdict remains appropriate, so no verdict change is needed.","tokens_in":12626,"tokens_out":5682,"duration_ms":62869,"concrete_test":"Download the linked Zenodo artifact and inspect the irace configuration. If no explicit tuning fold is held out from the CMA-ES best-trajectory data, rerun Section 4.2 with a nested protocol: for each of the five LOIO folds, run irace on only the four training instances' CMA-ES best 2-generation trajectories with the same irace settings, then train the selected configuration on those four instances and evaluate on the held-out instance. Compare the mean LOIO accuracy for Summary and Time Series Forest with Figure 2 and the Section 6 ELA gains. If the nested mean drops by more than about 2 percentage points, the tuned gains should be re-reported with confidence intervals and the 'tuning improves performance' claim restricted to properly nested settings.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claims depend on the tuned-model results in Section 4.2. Section 3 ('Automated Configuration') says irace tunes using 'the best trajectory of CMA-ES for 2 generations' with no mention of a held-out partition. Section 4.2 then transfers those parameters to all trajectory types and reports LOIO accuracy. Under this protocol, each LOIO test instance's trajectories were part of the tuning set: irace could select hyperparameters that fit those very instances. The later train-on-480/test-on-120 step only avoids training on the test fold, not tuning on it. This directly biases the tuned accuracies in Figure 2 and the headline gains over ELA in Section 6 (3% to 7%, 2% at low budget, 6% with Summary). The default-model results in Section 4.1 are not affected by this leakage and still support the broad claim that classifier choice matters; the leakage specifically undermines the quantitative superiority of tuned Summary and Time Series Forest and the claim that tuning 'improves performance' in most settings. The limitation paragraph in Section 5 acknowledges tuning only on CMA-ES best trajectories, but not the absence of a tuning/test separation. A secondary caveat is that LOPO default results in Section 4.3 place LSTM ahead of Summary, so the abstract's 'feature-based and interval-based models are the best choices' is stronger for LOIO than for the harder LOPO setting; this does not weaken the main leakage point.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper benchmarks 17 time-series classifiers for algorithm selection on the BBOB suite, using probing trajectories (best, current, and concatenated all) as inputs, with leave-one-instance-out (LOIO) and leave-one-problem-out (LOPO) validation. The authors find that classifier choice strongly affects accuracy, that feature-based (Summary) and interval-based (Time Series Forest) classifiers consistently outperform other families, and that automated configuration of these models increases the reported accuracy gain over ELA features from 3% to 7%, with a 2% gain at a much lower evaluation budget. The paper recommends Summary or Time Series Forest as defaults for trajectory-based algorithm selectors.","tokens_in":12867,"tokens_out":2482,"duration_ms":28499,"significance":"If the results are valid, the paper provides a practically useful guideline for trajectory-based algorithm selection, correcting the implicit assumption that classifier choice is secondary when moving from tabular to time-series data. The study's strengths include its broad coverage of 17 classifiers from multiple families, the use of standard BBOB benchmarks, both LOIO and LOPO validation, and the public release of data and code. The default-model comparison in Section 4.1 is a clean and valuable benchmark, and the finding that several kernel- and deep-learning models perform no better than a dummy classifier is a useful caution. However, the quantitative headline claims about tuned gains over ELA are undermined by a tuning/test leakage issue, and the absence of statistical confidence intervals makes point-estimate comparisons difficult to assess. The paper is therefore likely to contribute to the field after the tuning protocol is corrected and the central claims are re-derived from unbiased results.","major_comments":[{"comment":"The tuning protocol leaks test information: irace is configured on the full set of CMA-ES best trajectories for 2 generations, with no held-out tuning partition, and Section 3 states that the tuned parameters are then transferred to all other trajectory types. Under the LOIO protocol of Section 3, every test instance's trajectories are part of the tuning set, so hyperparameters can be selected that incidentally fit the test folds. This directly biases the tuned accuracies in Figure 2 and the Section 6 claims of 3–7% gains over ELA, 2% at low budget, and 6% with Summary. The authors need to re-run tuning with a nested cross-validation or an explicit tuning/validation split (e.g., tuning on a subset of instances disjoint from LOIO test folds) and report the resulting accuracies and gains. The default-model results in Section 4.1 are not affected by this issue and still support the broader claim that classifier choice matters, but the tuned quantitative claims must be corrected or explicitly reframed as an upper bound.","section":"§3 (Automated Configuration) and §4.2"},{"comment":"The ELA baseline is not recomputed in the same experimental pipeline. The reported gains over ELA (3% to 7%, 2% at low budget, 6% with Summary) are comparisons to the previous paper [34] rather than to an ELA-feature classifier trained and evaluated under identical conditions in this study. Without a direct ELA baseline on the same trajectories, instances, and validation folds, the magnitude of the claimed improvement is not established. The authors should add an ELA-feature classifier to the benchmark, or at minimum clearly state that the gains are inherited from a different experimental setup and are not directly comparable.","section":"§6 and Introduction"},{"comment":"No confidence intervals, standard errors, or significance tests are reported for the accuracy figures. The LOIO test set has only 120 samples per fold, so differences of a few percentage points among the leading classifiers (e.g., Summary versus Time Series Forest in Figure 2) could be within sampling noise. The phrase 'significant impact' should be supported either by paired statistical tests across folds (e.g., Wilcoxon signed-rank tests or McNemar's test) or by reporting confidence intervals, especially in the default-model comparison in Section 4.1.","section":"§4 Results"},{"comment":"The LOPO results in Section 4.3 show that LSTM achieves the highest average accuracy (61.3%) and is tied for the highest number of functions with accuracy ≥ 90%, while Summary is second. This is in tension with the abstract and conclusion, which state that 'feature-based and interval-based models are the best choices' without qualifying the LOPO setting. The authors should either soften the global recommendation or provide an explicit discussion of why LOIO is the primary protocol for the headline claim and how the LOPO exception affects the practical guidance.","section":"§4.3 and Abstract/Conclusion"}],"minor_comments":[{"comment":"Figure 2 shows tuned versus default results only for DE and ALL trajectories, although the text claims that tuning improves performance 'in most settings' across all trajectory types. Clarify whether the remaining trajectory types are in the supplementary material and explicitly list the cases where tuning does not help.","section":"§4.2"},{"comment":"The description of trajectory types says 'four trajectories can therefore be obtained per instance,' but the preceding text describes one trajectory per algorithm plus one concatenated trajectory, which is four. This is correct, but the wording 'per instance' could be clarified as 'per run of the portfolio on an instance.'","section":"§3.1"},{"comment":"The tuned Summary parameters are listed inline as 'mean,min, max, kurtosis, variance, nb unique and count statistics, 0.25 quantile'; a more structured formatting (e.g., a list or sub-table) would improve readability and avoid ambiguity about whether 'nb unique' is a single statistic.","section":"Table 1"},{"comment":"The limitation paragraph mentions that tuning used only CMA-ES trajectory data, but it does not acknowledge the absence of a tuning/test separation, which is a more consequential methodological limitation. Please add an explicit statement about this and its effect on the reported tuned gains.","section":"§5 Discussion"}],"recommendation":"major_revision","confidential_remarks":"The tuning leakage is load-bearing for the quantitative claims but is fixable within the manuscript's scope: re-run the tuned experiments with a proper nested split and update the affected numbers and conclusions. The default-model comparison is a solid empirical contribution and should be retained. I would not recommend rejection because the core qualitative finding—classifier choice matters for trajectory-based selectors—is supported by the default results and is likely to hold after the tuning correction."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe useful part of this paper is the default-model comparison. It is the first to put 17 time-series classifiers side by side on probing-trajectory algorithm selection, and the result is clear: classifier choice matters a lot, some models (kernel-based, most deep nets) sit at or below the dummy baseline, and Summary and Time Series Forest are consistently strong. That part is solid and worth knowing.\n\nThe soft spot is the tuning protocol, and it is a real one. Section 3 says irace tunes on the CMA-ES best trajectory for 2 generations, with no held-out partition. Section 4.2 then transfers those settings to all trajectory types and reports LOIO accuracy. Since LOIO leaves out instances one at a time, every test fold's instances were part of the tuning set. The later train/test split avoids training on the test fold but does not avoid tuning on it. That means the tuned accuracies in Figure 2 and the headline gains over ELA in Section 6 (3-7%, 2% at low budget, 6% with Summary) are likely inflated. The limitation paragraph mentions tuning only on CMA-ES data, but it does not acknowledge the tuning/test overlap. This should be fixed with nested cross-validation or at minimum a separate tuning split, and then the tuned comparisons re-run.\n\nTwo smaller issues. The ELA baseline is taken from a previous paper rather than recomputed in the same experimental pipeline, so the gains over ELA are only as trustworthy as that comparison. And there are no confidence intervals or significance tests on the accuracy differences; with 600 trajectories and 120 test points per fold, some of the 2-7% gaps are within reasonable noise. Also note the LOPO results put LSTM ahead of Summary on average, which complicates the abstract's blanket statement that feature- and interval-based models are best; the claim is really about LOIO.\n\nWho is this for? People building algorithm selectors on trajectory data, and the BBOB community. It deserves a serious referee, but the version I read should not be accepted without addressing the tuning leakage. The default-model result can survive; the quantitative gains should be labeled provisional until the tuning protocol is cleaned up.","headline":"A useful first benchmark of 17 time-series classifiers for trajectory-based algorithm selection, with a solid default-model comparison but tuned results compromised by tuning on test-fold instances.","tokens_in":13458,"tokens_out":2468,"would_cite":true,"duration_ms":24644,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that for trajectory-based algorithm selection, the choice of time-series classifier has a significant impact, and that feature-based and interval-based models—especially Summary and Time Series Forest—consistently…","keywords":["algorithm selection","time series classification","probing trajectories","black-box optimisation","classifier benchmarking","BBOB","feature-based classifiers","interval-based classifiers"],"falsifier":"Re-run the benchmark with strict nested validation, tuning hyperparameters on a dedicated partition of CMA-ES best trajectories and evaluating on disjoint leave-one-instance-out folds; if the 2-7% accuracy gain over ELA features collapses or reverses, the central claim is falsified. A second decisive check is to repeat the 17-model comparison on a different continuous benchmark suite, where the claim would be weakened if kernel-based or deep models match Summary and Time Series Forest.","tokens_in":12329,"feed_emoji":"📈","tokens_out":5660,"duration_ms":53904,"temperature":0.7,"pith_summary":"This paper asks whether the machine-learning model inside an algorithm selector matters when the input is a probing trajectory—a short time series of objective values from a run of a solver. The authors benchmark 17 time-series classifiers on the BBOB suite with a three-solver portfolio, using leave-one-instance-out and leave-one-problem-out validation. They report that, unlike the tabular-data setting, the choice of classifier has a large effect: feature-based and interval-based models are consistently best, while kernel-based and most deep-learning models often do no better than predicting the majority class. Tuned Summary and Time Series Forest classifiers improve accuracy over ELA-feature selectors by 2 to 7 percent, sometimes using far fewer function evaluations. If this holds, practitioners should replace single default classifiers such as Rotation Forest with one of these two models.","feed_headline":"Classifier choice matters for trajectory-based algorithm selection","feed_subtitle":"A 17-model BBOB benchmark finds feature- and interval-based models gain 2-7% over ELA features.","key_machinery":"The central object is the probing trajectory: the sequence of objective values recorded at each function evaluation during a short run of CMA-ES, PSO, or DE, optionally concatenated across algorithms (ALL). Each instance is represented by such a series and labelled with the solver that achieves the best median value after 100,000 evaluations, turning algorithm selection into a time-series classification task. The benchmark spans 17 classifiers from seven families (deep learning, distance, feature, interval, kernel, shapelet, and a default scikit-learn ensemble), using default parameters first and then irace-tuned parameters for the strongest candidates. The comparison is what carries the argument: it isolates which classifier families extract useful signal from the ordered evaluation data.","core_discovery":"The central claim is that classifier choice is decisive for trajectory-based algorithm selection, and that two families dominate: a feature-based model (Summary) that extracts statistics from each trajectory and trains a Random Forest, and an interval-based model (Time Series Forest) that builds an ensemble of trees on random intervals of the series. Across all trajectory types and both validation schemes, these two are consistently ranked at or near the top; the Rotation Forest used in earlier work is never the best, and several kernel and deep models match the Dummy baseline. Tuned configurations selected on one trajectory type transfer to others and increase the gain over ELA-feature inputs from 3% to as much as 7% at similar budgets, with a 2% gain at more than seven times fewer evaluations in one setting. In the harder leave-one-problem-out setting, accuracy is generally lower and a few functions are nearly impossible for all models, yet LSTM and Summary stand out as the models that learn real predictions rather than echoing the majority class.","pith_inferences":["The authors do not test this, but the family-level pattern suggests the conclusion may extend beyond BBOB to other continuous benchmarks, while the specific accuracy magnitudes are probably suite-dependent.","Because LSTM was the only deep model that learned beyond the majority class in the harder validation setting, architecture choice within deep learning may matter more than the family label; the paper uses off-the-shelf architectures only.","A natural extension is to benchmark regressor-based selectors on the same trajectories, since the paper only considers classification.","The transferability of tuned parameters suggests a cheaper pipeline: tune once on a short CMA-ES trajectory and deploy on longer or multi-algorithm trajectories."],"forward_implications":["Practitioners building trajectory-based selectors should default to Summary or Time Series Forest rather than Rotation Forest.","Accuracy gains of 2 to 7 percent over ELA-feature selectors are available at similar or much lower evaluation budgets.","Classifier rankings are largely stable across trajectory types and validation settings, so model choice can be made once per pipeline.","Function-level difficulty in the leave-one-problem-out setting is robust across models, suggesting some BBOB functions are intrinsically hard to distinguish from trajectories.","Parameter tuning transfers between trajectory types, so expensive tuning can be done on a cheap trajectory and reused."],"supporting_citations":[{"why":"Supplies the probing-trajectory data, the classification task, and the 3% ELA-feature baseline that this paper extends.","marker":"[34]"},{"why":"Provides the tabular-data result that model choice matters little, the contrast this paper overturns for time-series input.","marker":"[20]"},{"why":"Defines Time Series Forest, one of the two recommended classifiers.","marker":"[7]"},{"why":"Defines Rotation Forest, the previous default classifier that this paper shows is never best.","marker":"[35]"},{"why":"Cited for the leave-one-problem-out validation procedure used in the harder evaluation setting.","marker":"[8]"},{"why":"Dataset source for the CMA-ES, PSO, and DE trajectories on BBOB used as training inputs.","marker":"[38]"},{"why":"Provides irace, the automated configuration tool used to tune the models' parameters.","marker":"[23]"},{"why":"Provides the sktime package, which supplies 16 of the 17 time-series classifiers benchmarked.","marker":"[22]"}],"fun_headline_variants":["Feature and interval classifiers dominate trajectory selection","Classifier choice matters: feature and interval models top BBOB","Benchmark: For probing trajectories, feature/interval models excel","Trajectory selectors: 17-model test crowns feature and interval classifiers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The tuned results assume that the hyperparameters were selected without using the instances later held out for validation, and that parameters tuned on one trajectory type transfer to the others.","fun_headline_variants_meta":{"raw":{"variants":["Feature and interval classifiers dominate trajectory selection","Classifier choice matters: feature and interval models top BBOB","Benchmark: For probing trajectories, feature/interval models excel","Trajectory selectors: 17-model test crowns feature and interval classifiers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00084,"raw_usage":{"total_tokens":3657,"prompt_tokens":937,"completion_tokens":2720,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":553,"completion_tokens_details":{"reasoning_tokens":2651}},"tokens_in":553,"tokens_out":2720,"duration_ms":21315,"temperature":1.0,"reasoning_tokens":2651,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:17:35.661338+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the benchmark with strict nested validation, tuning hyperparameters on a dedicated partition of CMA-ES best trajectories and evaluating on disjoint leave-one-instance-out folds; if the 2-7% accuracy gain over ELA features collapses or reverses, the central claim is falsified. A second decisive check is to repeat the 17-model comparison on a different continuous benchmark suite, where the claim would be weakened if kernel-based or deep models match Summary and Time Series Forest.","supporting_citations":[{"cited_title":"In: Proceedings of the Companion Conference on Genetic and Evolutionary Com- putation","cited_arxiv_id":null,"evidence_quote":"Provides the tabular-data result that model choice matters little, the contrast this paper overturns for time-series input."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines Time Series Forest, one of the two recommended classifiers."},{"cited_title":"IEEE Trans","cited_arxiv_id":null,"evidence_quote":"Defines Rotation Forest, the previous default classifier that this paper shows is never best."},{"cited_title":"https://doi.org/10.5281/zenodo.7249389","cited_arxiv_id":null,"evidence_quote":"Dataset source for the CMA-ES, PSO, and DE trajectories on BBOB used as training inputs."},{"cited_title":"Opera- tions Research Perspectives 3, 43 – 58 (2016)","cited_arxiv_id":null,"evidence_quote":"Provides irace, the automated configuration tool used to tune the models' parameters."}],"review_version":1}