Pith. sign in

REVIEW 4 major objections 4 minor 42 references

Algorithm Selection with Probing Trajectories: Benchmarking the Choice of Classifier Model

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that for trajectory-based algorithm selection, the choice of time-series classifier has a significant impact, and that feature-based and interval-based models—especially Summary and Time Series Forest—consistently…

desk verdict A useful first benchmark of 17 time-series classifiers for trajectory-based algorithm selection, with a solid default-model comparison but tuned results compromised by tuning on test-fold instances. read the letter →

arxiv 2501.11414 v1 pith:RUT3KKTN submitted 2025-01-20 cs.LG cs.NE

Quentin Renau , Emma Hart This is my paper · ORCID
classification cs.LGcs.NE
keywords algorithmselectiontimeseriesclassificationprobingtrajectoriesblack-boxoptimisationclassifierbenchmarkingBBOBfeature-basedclassifiersinterval-based
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the machine-learning model inside an algorithm selector matters when the input is a probing trajectory—a short time series of objective values from a run of a solver. The authors benchmark 17 time-series classifiers on the BBOB suite with a three-solver portfolio, using leave-one-instance-out and leave-one-problem-out validation. They report that, unlike the tabular-data setting, the choice of classifier has a large effect: feature-based and interval-based models are consistently best, while kernel-based and most deep-learning models often do no better than predicting the majority class. Tuned Summary and Time Series Forest classifiers improve accuracy over ELA-feature selectors by 2 to 7 percent, sometimes using far fewer function evaluations. If this holds, practitioners should replace single default classifiers such as Rotation Forest with one of these two models.

What carries the argument

The central object is the probing trajectory: the sequence of objective values recorded at each function evaluation during a short run of CMA-ES, PSO, or DE, optionally concatenated across algorithms (ALL). Each instance is represented by such a series and labelled with the solver that achieves the best median value after 100,000 evaluations, turning algorithm selection into a time-series classification task. The benchmark spans 17 classifiers from seven families (deep learning, distance, feature, interval, kernel, shapelet, and a default scikit-learn ensemble), using default parameters first and then irace-tuned parameters for the strongest candidates. The comparison is what carries the argument: it isolates which classifier families extract useful signal from the ordered evaluation data.

What would settle it

Re-run the benchmark with strict nested validation, tuning hyperparameters on a dedicated partition of CMA-ES best trajectories and evaluating on disjoint leave-one-instance-out folds; if the 2-7% accuracy gain over ELA features collapses or reverses, the central claim is falsified. A second decisive check is to repeat the 17-model comparison on a different continuous benchmark suite, where the claim would be weakened if kernel-based or deep models match Summary and Time Series Forest.

Watch

Extended reading notes

Core claim

The central claim is that classifier choice is decisive for trajectory-based algorithm selection, and that two families dominate: a feature-based model (Summary) that extracts statistics from each trajectory and trains a Random Forest, and an interval-based model (Time Series Forest) that builds an ensemble of trees on random intervals of the series. Across all trajectory types and both validation schemes, these two are consistently ranked at or near the top; the Rotation Forest used in earlier work is never the best, and several kernel and deep models match the Dummy baseline. Tuned configurations selected on one trajectory type transfer to others and increase the gain over ELA-feature inputs from 3% to as much as 7% at similar budgets, with a 2% gain at more than seven times fewer evaluations in one setting. In the harder leave-one-problem-out setting, accuracy is generally lower and a few functions are nearly impossible for all models, yet LSTM and Summary stand out as the models that learn real predictions rather than echoing the majority class.

Load-bearing premise

The tuned results assume that the hyperparameters were selected without using the instances later held out for validation, and that parameters tuned on one trajectory type transfer to the others.

Editorial extensions

If this is right

  • Practitioners building trajectory-based selectors should default to Summary or Time Series Forest rather than Rotation Forest.
  • Accuracy gains of 2 to 7 percent over ELA-feature selectors are available at similar or much lower evaluation budgets.
  • Classifier rankings are largely stable across trajectory types and validation settings, so model choice can be made once per pipeline.
  • Function-level difficulty in the leave-one-problem-out setting is robust across models, suggesting some BBOB functions are intrinsically hard to distinguish from trajectories.
  • Parameter tuning transfers between trajectory types, so expensive tuning can be done on a cheap trajectory and reused.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors do not test this, but the family-level pattern suggests the conclusion may extend beyond BBOB to other continuous benchmarks, while the specific accuracy magnitudes are probably suite-dependent.
  • Because LSTM was the only deep model that learned beyond the majority class in the harder validation setting, architecture choice within deep learning may matter more than the family label; the paper uses off-the-shelf architectures only.
  • A natural extension is to benchmark regressor-based selectors on the same trajectories, since the paper only considers classification.
  • The transferability of tuned parameters suggests a cheaper pipeline: tune once on a short CMA-ES trajectory and deploy on longer or multi-algorithm trajectories.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper benchmarks 17 time-series classifiers for algorithm selection on the BBOB suite, using probing trajectories (best, current, and concatenated all) as inputs, with leave-one-instance-out (LOIO) and leave-one-problem-out (LOPO) validation. The authors find that classifier choice strongly affects accuracy, that feature-based (Summary) and interval-based (Time Series Forest) classifiers consistently outperform other families, and that automated configuration of these models increases the reported accuracy gain over ELA features from 3% to 7%, with a 2% gain at a much lower evaluation budget. The paper recommends Summary or Time Series Forest as defaults for trajectory-based algorithm selectors.

Significance. If the results are valid, the paper provides a practically useful guideline for trajectory-based algorithm selection, correcting the implicit assumption that classifier choice is secondary when moving from tabular to time-series data. The study's strengths include its broad coverage of 17 classifiers from multiple families, the use of standard BBOB benchmarks, both LOIO and LOPO validation, and the public release of data and code. The default-model comparison in Section 4.1 is a clean and valuable benchmark, and the finding that several kernel- and deep-learning models perform no better than a dummy classifier is a useful caution. However, the quantitative headline claims about tuned gains over ELA are undermined by a tuning/test leakage issue, and the absence of statistical confidence intervals makes point-estimate comparisons difficult to assess. The paper is therefore likely to contribute to the field after the tuning protocol is corrected and the central claims are re-derived from unbiased results.

major comments (4)
  1. [§3 (Automated Configuration) and §4.2] The tuning protocol leaks test information: irace is configured on the full set of CMA-ES best trajectories for 2 generations, with no held-out tuning partition, and Section 3 states that the tuned parameters are then transferred to all other trajectory types. Under the LOIO protocol of Section 3, every test instance's trajectories are part of the tuning set, so hyperparameters can be selected that incidentally fit the test folds. This directly biases the tuned accuracies in Figure 2 and the Section 6 claims of 3–7% gains over ELA, 2% at low budget, and 6% with Summary. The authors need to re-run tuning with a nested cross-validation or an explicit tuning/validation split (e.g., tuning on a subset of instances disjoint from LOIO test folds) and report the resulting accuracies and gains. The default-model results in Section 4.1 are not affected by this issue and still support the broader claim that classifier choice matters, but the tuned quantitative claims must be corrected or explicitly reframed as an upper bound.
  2. [§6 and Introduction] The ELA baseline is not recomputed in the same experimental pipeline. The reported gains over ELA (3% to 7%, 2% at low budget, 6% with Summary) are comparisons to the previous paper [34] rather than to an ELA-feature classifier trained and evaluated under identical conditions in this study. Without a direct ELA baseline on the same trajectories, instances, and validation folds, the magnitude of the claimed improvement is not established. The authors should add an ELA-feature classifier to the benchmark, or at minimum clearly state that the gains are inherited from a different experimental setup and are not directly comparable.
  3. [§4 Results] No confidence intervals, standard errors, or significance tests are reported for the accuracy figures. The LOIO test set has only 120 samples per fold, so differences of a few percentage points among the leading classifiers (e.g., Summary versus Time Series Forest in Figure 2) could be within sampling noise. The phrase 'significant impact' should be supported either by paired statistical tests across folds (e.g., Wilcoxon signed-rank tests or McNemar's test) or by reporting confidence intervals, especially in the default-model comparison in Section 4.1.
  4. [§4.3 and Abstract/Conclusion] The LOPO results in Section 4.3 show that LSTM achieves the highest average accuracy (61.3%) and is tied for the highest number of functions with accuracy ≥ 90%, while Summary is second. This is in tension with the abstract and conclusion, which state that 'feature-based and interval-based models are the best choices' without qualifying the LOPO setting. The authors should either soften the global recommendation or provide an explicit discussion of why LOIO is the primary protocol for the headline claim and how the LOPO exception affects the practical guidance.
minor comments (4)
  1. [§4.2] Figure 2 shows tuned versus default results only for DE and ALL trajectories, although the text claims that tuning improves performance 'in most settings' across all trajectory types. Clarify whether the remaining trajectory types are in the supplementary material and explicitly list the cases where tuning does not help.
  2. [§3.1] The description of trajectory types says 'four trajectories can therefore be obtained per instance,' but the preceding text describes one trajectory per algorithm plus one concatenated trajectory, which is four. This is correct, but the wording 'per instance' could be clarified as 'per run of the portfolio on an instance.'
  3. [Table 1] The tuned Summary parameters are listed inline as 'mean,min, max, kurtosis, variance, nb unique and count statistics, 0.25 quantile'; a more structured formatting (e.g., a list or sub-table) would improve readability and avoid ambiguity about whether 'nb unique' is a single statistic.
  4. [§5 Discussion] The limitation paragraph mentions that tuning used only CMA-ES trajectory data, but it does not acknowledge the absence of a tuning/test separation, which is a more consequential methodological limitation. Please add an explicit statement about this and its effect on the reported tuned gains.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the classifier benchmark is evaluated against external LOIO/LOPO accuracy and the ELA comparison is an external baseline; the only caveat is a possible tuning/validation overlap, which is a statistical concern, not a circular derivation.

full rationale

The paper's claims are empirical benchmarking results, not derivations from the target quantity. The label (best algorithm after 100,000 evaluations) is intentionally built from the same runs that provide the early probing trajectories; this is the standard supervised setup and is not circular because the early trajectory does not contain the final label. Classifier rankings and the recommendation of Summary and Time Series Forest rest on LOIO and LOPO accuracy computed by training on held-in folds and testing on held-out instances/functions, and the comparison with ELA features uses an external prior result ([34]). The only notable weakness is the hyperparameter tuning in Sections 3.2 and 4.2: irace is described as tuning on the CMA-ES best 2-generation trajectories without an explicit held-out tuning partition, and the Section 5 limitation does not discuss a tuning/test split. If the tuning set overlaps the LOIO test folds, the tuned accuracies in Figure 2 and the Section 6 gains over ELA would be optimistically biased. That is a potential evaluation-protocol flaw, not a circularity: the tuned models' test predictions are still produced from training-fold data and are not equal to the tuning objective by construction. No self-citation chain or fitted-parameter-renamed-as-prediction is load-bearing here, so the circularity score is 0.

Assumptions & free parameters 9 free parameters · 5 assumptions · 0 invented entities

The paper is an empirical benchmark, so the ledger mostly lists dataset design choices and tuned hyperparameters rather than derived constants. The central claim depends on the BBOB dataset, the label budget, the short trajectories, and the assumption that hyperparameters tuned once on one trajectory type can be transferred without leaking test-fold information. No new entities are introduced.

free parameters (9)
  • Label budget for determining winning algorithm = 100,000 function evaluations
    The class label for each instance is defined by the algorithm with the best median objective value after this fixed budget, following reference [34]. Changing the budget would change labels and possibly the rankings.
  • Trajectory lengths = 2 and 7 generations
    The paper tests only two short trajectory lengths, chosen to match prior work, and all results are reported only for these values.
  • Number of BBOB instances and runs = 5 instances per function, 5 runs per instance
    The dataset contains 600 trajectories per input type. The small instance count makes accuracy estimates noisy and limits generalization claims.
  • Summary classifier tuned statistics = mean, min, max, kurtosis, variance, nb unique, count, 0.25 quantile
    Selected by irace on the CMA-ES best trajectory and transferred to all other trajectory types.
  • Time Series Forest tuned parameters = 460 estimators, minimum interval length 3
    Selected by irace on the CMA-ES best trajectory and transferred to all other trajectory types.
  • Rotation Forest tuned parameters = 367 estimators, min group 10, max group 19, remove proportion 0.2364
    Selected by irace on the CMA-ES best trajectory and transferred to all other trajectory types.
  • kNN tuned parameters = 4 neighbors, uniform weights, twe distance
    Selected by irace on the CMA-ES best trajectory and transferred to all other trajectory types.
  • ShapeDTW tuned parameters = 4 neighbors, raw descriptor
    Selected by irace on the CMA-ES best trajectory and transferred to all other trajectory types.
  • irace tuning budget = 5,000 evaluations for cheap models, 1,000 for expensive models
    The choice of tuning budget can affect which hyperparameters are selected and therefore the tuned results.
assumptions (5)
  • domain assumption BBOB is a representative benchmark for continuous black-box optimization algorithm selection
    The study relies on BBOB to make general claims about classifier choice. The authors acknowledge in Section 5 that BBOB is known not to generalize well.
  • domain assumption A 2 or 7 generation probing trajectory contains sufficient signal to predict the winning algorithm after 100,000 evaluations
    The entire predictive setup assumes short prefixes of algorithm runs are informative about final rankings. This is the premise of trajectory-based algorithm selection from reference [34].
  • domain assumption The winning algorithm label based on median target after 100,000 evaluations is the correct target for selection
    Labels are defined by one fixed budget. Different budgets could change which algorithm wins and therefore all accuracy numbers.
  • domain assumption Default implementations from sktime, scikit-learn, and irace faithfully realize the cited algorithms
    The benchmark compares implementations rather than conceptual algorithms. The sktime version is specified, but no package commit hashes are given.
  • domain assumption Hyperparameters tuned once on CMA-ES best trajectories transfer to other trajectory types without loss
    Section 4.2 transfers tuned configurations to DE, PSO, and ALL trajectories. The paper does not validate transfer per trajectory type with a separate tuning set.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Algorithm Selection with Probing Trajectories: Benchmarking the Choice of Classifier Model." pith.science (2026). https://pith.science/paper/RUT3KKTN

@misc{pith2026250111414,
  author       = {Pith},
  title        = {Pith review of: Algorithm Selection with Probing Trajectories: Benchmarking the Choice of Classifier Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RUT3KKTN}},
  note         = {Machine review of arXiv:2501.11414}
}
read the original abstract

Recent approaches to training algorithm selectors in the black-box optimisation domain have advocated for the use of training data that is algorithm-centric in order to encapsulate information about how an algorithm performs on an instance, rather than relying on information derived from features of the instance itself. Probing-trajectories that consist of a sequence of objective performance per function evaluation obtained from a short run of an algorithm have recently shown particular promise in training accurate selectors. However, training models on this type of data requires an appropriately chosen classifier given the sequential nature of the data. There are currently no clear guidelines for choosing the most appropriate classifier for algorithm selection using time-series data from the plethora of models available. To address this, we conduct a large benchmark study using 17 different classifiers and three types of trajectory on a classification task using the BBOB benchmark suite using both leave-one-instance out and leave-one-problem out cross-validation. In contrast to previous studies using tabular data, we find that the choice of classifier has a significant impact, showing that feature-based and interval-based models are the best choices.

Figures

Figures reproduced from arXiv: 2501.11414 by the authors.

Figure 1
Figure 1. Accuracy of classification on the LOIO cross-validation for best-so-far and [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Accuracy of classification on the LOIO cross-validation for best-so-far and [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Heatmap of classification accuracy on the LOPO cross-validation for [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Number of functions with accuracy above 90% or below 10% for each [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Number of models with accuracy above 90% or below 10% for each func [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 17 canonical work pages

  1. [34]

    In: Applications of Evolutionary Computation - 27th European Conference, EvoApplications 2024, Held as Part of EvoStar 2024, Aberystwyth, UK, April 3-5, 2024, Proceedings, Part I

    Renau, Q., Hart, E.: On the utility of probing trajectories for algorithm-selection. In: Applications of Evolutionary Computation - 27th European Conference, EvoApplications 2024, Held as Part of EvoStar 2024, Aberystwyth, UK, April 3-5, 2024, Proceedings, Part I. Lecture Notes in Computer Science, vol. 14634, pp. 98–114. Springer (2024). https://doi.org/...

    doi:10.1007/978-3-031-56852-7_7 2024

  2. [1]

    Alissa, M., Sim, K., Hart, E.: Automated algorithm selection: from feature-based to feature-free approaches. J. Heuristics 29(1), 1–38 (2023). https://doi.org/10. 1007/s10732-022-09505-4

    work page 2023

  3. [2]

    Baratchi, M., Wang, C., Limmer, S., van Rijn, J., Hoos, H., B¨ ack, T., Olhofer, M.: Automated machine learning: past, present and future. Artif. Intell. Rev. 57(5), 122 (2024). https://doi.org/10.1007/S10462-024-10726-1

    doi:10.1007/s10462-024-10726-1 2024

  4. [3]

    Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001). https://doi.org/ 10.1023/A:1010933404324

    doi:10.1023/a:1010933404324 2001

  5. [4]

    In: 20th IEEE International Conference on Data Mining, ICDM 2020, Sorrento, Italy, November 17-20, 2020

    Cabello, N., Naghizade, E., Qi, J., Kulik, L.: Fast and accurate time series classifi- cation through supervised interval search. In: 20th IEEE International Conference on Data Mining, ICDM 2020, Sorrento, Italy, November 17-20, 2020. pp. 948–953. IEEE (2020). https://doi.org/10.1109/ICDM50108.2020.00107

    arXiv 2020

  6. [5]

    In: Proceedings of the Genetic and Evolutionary Computation Confer- ence, GECCO 2023, Lisbon, Portugal, July 15-19, 2023

    Cenikj, G., Petelin, G., Doerr, C., Korosec, P., Eftimov, T.: Dynamorep: Trajectory-based population dynamics for classification of black-box optimization problems. In: Proceedings of the Genetic and Evolutionary Computation Confer- ence, GECCO 2023, Lisbon, Portugal, July 15-19, 2023. pp. 813–821. ACM (2023). https://doi.org/10.1145/3583131.3590401

    arXiv 2023

  7. [6]

    Data Min

    Dempster, A., Petitjean, F., Webb, G.I.: ROCKET: exceptionally fast and accurate time series classification using random convolutional kernels. Data Min. Knowl. Discov. 34(5), 1454–1495 (2020). https://doi.org/10.1007/ S10618-020-00701-Z

    work page 2020

  8. [7]

    Deng, H., Runger, G.C., Tuv, E., Martyanov, V.: A time series forest for classifica- tion and feature extraction. Inf. Sci. 239, 142–153 (2013). https://doi.org/10. 1016/J.INS.2013.02.030

    work page 2013

Show all 42 references
  1. [8]

    In: Proceedings of Foundations of Genetic Algorithms (FOGA) ’19

    Derbel, B., Liefooghe, A., V´ erel, S., Aguirre, H., Tanaka, K.: New features for continuous exploratory landscape analysis based on the SOO tree. In: Proceedings of Foundations of Genetic Algorithms (FOGA) ’19. pp. 72–86. ACM (2019).https: //doi.org/10.1145/3299904.3340308

    2019

  2. [9]

    Advances in Neural Information Processing Systems 34, 18932–18943 (2021)

    Gorishniy, Y., Rubachev, I., Khrulkov, V., Babenko, A.: Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems 34, 18932–18943 (2021)

    2021

  3. [10]

    Optimiza- tion Methods and Software 36(1), 114–144 (2021)

    Hansen, N., Auger, A., Ros, R., Mersmann, O., Tusar, T., Brockhoff, D.: COCO: a platform for comparing continuous optimizers in a black-box setting. Optimiza- tion Methods and Software 36(1), 114–144 (2021). https://doi.org/10.1080/ 10556788.2020.1808977

    2021

  4. [11]

    Evolutionary Computation 9(2), 159–195 (2001)

    Hansen, N., Ostermeier, A.: Completely derandomized self-adaptation in evolution strategies. Evolutionary Computation 9(2), 159–195 (2001). https://doi.org/10. 1162/106365601750190398

    2001

  5. [12]

    Data Min

    Hills, J., Lines, J., Baranauskas, E., Mapp, J., Bagnall, A.J.: Classification of time series by shapelet transformation. Data Min. Knowl. Discov.28(4), 851–881 (2014). https://doi.org/10.1007/S10618-013-0322-1

    2014 doi

  6. [13]

    Neural Comput

    Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (nov 1997). https://doi.org/10.1162/neco.1997.9.8.1735 16 Q. Renau and E. Hart

    1997 doi

  7. [14]

    In: Proceedings of the Genetic and Evolutionary Com- putation Conference, GECCO ’20 (2020)

    Jankovic, A., Doerr, C.: Landscape-aware fixed-budget performance regression for modular cma-es variants. In: Proceedings of the Genetic and Evolutionary Com- putation Conference, GECCO ’20 (2020). https://doi.org/10.1145/3377930. 3390183, https://doi.org/10.1145/3377930.33901...

    2020

  8. [15]

    In: Applications of Evolutionary Computation - 24th International Conference, EvoApplications 2021, Held as Part of EvoStar 2021, Proceedings

    Jankovic, A., Eftimov, T., Doerr, C.: Towards feature-based performance regres- sion using trajectory data. In: Applications of Evolutionary Computation - 24th International Conference, EvoApplications 2021, Held as Part of EvoStar 2021, Proceedings. Lecture Notes in Computer ...

    2021 doi

  9. [16]

    In: IEEE Congress on Evolutionary Computation, CEC 2022, Padua, Italy, July 18-23, 2022

    Jankovic, A., Vermetten, D., Kostovska, A., de Nobel, J., Eftimov, T., Doerr, C.: Trajectory-based algorithm selection with warm-starting. In: IEEE Congress on Evolutionary Computation, CEC 2022, Padua, Italy, July 18-23, 2022. pp. 1–8. IEEE (2022). https://doi.org/10.1109/CEC...

    2022

  10. [17]

    In: Proceedings of ICNN’95 - International Conference on Neural Networks

    Kennedy, J., Eberhart, R.: Particle swarm optimization. In: Proceedings of ICNN’95 - International Conference on Neural Networks. vol. 4, pp. 1942–1948 vol.4 (1995). https://doi.org/10.1109/ICNN.1995.488968

    1995

  11. [18]

    Evolutionary Computation 27(1), 3–45 (Mar 2019)

    Kerschke, P., Hoos, H., Neumann, F., Trautmann, H.: Automated Algorithm Selec- tion: Survey and Perspectives. Evolutionary Computation 27(1), 3–45 (Mar 2019)

    2019

  12. [19]

    In: Parallel Problem Solving from Nature - PPSN XVII - 17th Inter- national Conference, PPSN 2022, Dortmund, Germany, September 10-14, 2022, Proceedings, Part I

    Kostovska, A., Jankovic, A., Vermetten, D., de Nobel, J., Wang, H., Eftimov, T., Doerr, C.: Per-run algorithm selection with warm-starting using trajectory-based features. In: Parallel Problem Solving from Nature - PPSN XVII - 17th Inter- national Conference, PPSN 2022, Dortmu...

    2022 doi

  13. [20]

    In: Proceedings of the Companion Conference on Genetic and Evolutionary Com- putation

    Kostovska, A., Jankovic, A., Vermetten, D., Dˇ zeroski, S., Eftimov, T., Doerr, C.: Comparing algorithm selection approaches on black-box optimization problems. In: Proceedings of the Companion Conference on Genetic and Evolutionary Com- putation. pp. 495–498 (2023)

    2023

  14. [21]

    ACM Trans

    Lines, J., Taylor, S., Bagnall, A.: Time series classification with hive-cote: The hierarchical vote collective of transformation-based ensembles. ACM Trans. Knowl. Discov. Data 12(5) (Jul 2018). https://doi.org/10.1145/3182382

    2018 doi

  15. [22]

    CoRR abs/1909.07872 (2019), http://arxiv.org/abs/1909.07872

    L¨ oning, M., Bagnall, A.J., Ganesh, S., Kazakov, V., Lines, J., Kir´ aly, F.J.: sktime: A unified interface for machine learning with time series. CoRR abs/1909.07872 (2019), http://arxiv.org/abs/1909.07872

    2019 arXiv

  16. [23]

    Opera- tions Research Perspectives 3, 43 – 58 (2016)

    L´ opez-Ib´ a˜ nez, M., Dubois-Lacoste, J., P´ erez C´ aceres, L., Birattari, M., St¨ utzl, T.: The irace package: Iterated racing for automatic algorithm configuration. Opera- tions Research Perspectives 3, 43 – 58 (2016)

    2016

  17. [24]

    Data Min

    Lubba, C.H., Sethi, S.S., Knaute, P., Schultz, S.R., Fulcher, B.D., Jones, N.S.: catch22: Canonical time-series characteristics - selected through highly compar- ative time-series analysis. Data Min. Knowl. Discov. 33(6), 1821–1852 (2019). https://doi.org/10.1007/S10618-019-00647-X

    2019 doi

  18. [25]

    In: Proceedings of the Genetic and Evolutionary Computation Conference, GECCO ’11

    Mersmann, O., Bischl, B., Trautmann, H., Preuss, M., Weihs, C., Rudolph, G.: Exploratory Landscape Analysis. In: Proceedings of the Genetic and Evolutionary Computation Conference, GECCO ’11. pp. 829–836. ACM (2011). https://doi. org/10.1145/2001576.2001690

    2011

  19. [26]

    Middlehurst, M., Large, J., Flynn, M., Lines, J., Bostrom, A., Bagnall, A.J.: HIVE-COTE 2.0: a new meta ensemble for time series classification. Mach. Learn. 110(11), 3211–3243 (2021). https://doi.org/10.1007/S10994-021-06057-9

    2021 doi

  20. [27]

    Journal of Machine Learning Research 12, 2825–2830 (2011)

    Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Algorithm Selection with Probing-Trajectories 17 Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-l...

    2011

  21. [28]

    In: Proceedings of the Genetic and Evolutionary Computation Conference, GECCO ’19

    Pitra, Z., Repick´ y, J., Holena, M.: Landscape analysis of Gaussian process surro- gates for the covariance matrix adaptation evolution strategy. In: Proceedings of the Genetic and Evolutionary Computation Conference, GECCO ’19. pp. 691–699 (2019). https://doi.org/10.1145/332...

    2019

  22. [29]

    In: Ap- plications of Evolutionary Computation - 24th International Conference, EvoAp- plications 2021, Held as Part of EvoStar 2021, Proceedings

    Renau, Q., Dr´ eo, J., Doerr, C., Doerr, B.: Towards explainable exploratory land- scape analysis: Extreme feature selection for classifying BBOB functions. In: Ap- plications of Evolutionary Computation - 24th International Conference, EvoAp- plications 2021, Held as Part of ...

    2021

  23. [30]

    https://doi.org/10.5281/ zenodo.14163833, https://doi.org/10.5281/zenodo.14163833

    Renau, Q., Hart, E.: Algorithm Selection with Probing Trajectories: Benchmarking the Choice of Classifier Model - Data (Nov 2024). https://doi.org/10.5281/ zenodo.14163833, https://doi.org/10.5281/zenodo.14163833

    2024 doi

  24. [31]

    In: Parallel Problem Solving from Nature - PPSN XVIII - 18th International Conference, PPSN 2024, Hagenberg, Austria, September 14-18, 2024, Proceedings, Part II

    Renau, Q., Hart, E.: Identifying easy instances to improve efficiency of ML pipelines for algorithm-selection. In: Parallel Problem Solving from Nature - PPSN XVIII - 18th International Conference, PPSN 2024, Hagenberg, Austria, September 14-18, 2024, Proceedings, Part II. Lec...

    2024 doi

  25. [32]

    https://doi.org/10

    Renau, Q., Hart, E.: Identifying easy instances to improve efficiency of ml pipelines for algorithm-selection - code and data (2024). https://doi.org/10. 5281/zenodo.10590233

    2024

  26. [33]

    In: Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2024, Melbourne, VIC, Australia, July 14-18, 2024

    Renau, Q., Hart, E.: Improving algorithm-selectors and performance-predictors via learning discriminating training samples. In: Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2024, Melbourne, VIC, Australia, July 14-18, 2024. ACM (2024). https://doi....

    2024

  27. [35]

    IEEE Trans

    Rodr ´ ıguez, J., Kuncheva, L., Alonso, C.: Rotation forest: A new classifier ensemble method. IEEE Trans. Pattern Anal. Mach. Intell.28(10), 1619–1630 (2006). https: //doi.org/10.1109/TPAMI.2006.211

    2006 doi

  28. [36]

    In: International Conference on Parallel Problem Solving from Nature

    Seiler, M., Pohl, J., Bossek, J., Kerschke, P., Trautmann, H.: Deep learning as a competitive feature-free approach for automated algorithm selection on the travel- ing salesperson problem. In: International Conference on Parallel Problem Solving from Nature. pp. 48–64. Spring...

    2020

  29. [37]

    Journal of Global Optimization 11(4), 341–359 (1997)

    Storn, R., Price, K.: Differential evolution - A simple and efficient heuristic for global optimization over continuous spaces. Journal of Global Optimization 11(4), 341–359 (1997). https://doi.org/10.1023/A:1008202821328

    1997 doi

  30. [38]

    https://doi.org/10.5281/zenodo.7249389

    Vermetten, D., Hao, W., Sim, K., Hart, E.: To Switch or not to Switch: Predict- ing the Benefit of Switching between Algorithms based on Trajectory Features - Dataset (2022). https://doi.org/10.5281/zenodo.7249389

    2022 doi

  31. [39]

    In: Proceedings of the Second International Confer- ence on Automated Machine Learning

    Vermetten, D., Ye, F., B¨ ack, T., Doerr, C.: Ma-bbob: Many-affine combinations of bbob functions for evaluating automl approaches in noiseless numerical black- box optimization contexts. In: Proceedings of the Second International Confer- ence on Automated Machine Learning. P...

    2023

  32. [40]

    In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining

    Zerveas, G., Jayaraman, S., Patel, D., Bhamidipaty, A., Eickhoff, C.: A transformer-based framework for multivariate time series representation learning. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. p. 2114–2124. KDD ’21, Association ...

    2021

  33. [41]

    Journal of Systems Engineering and Electronics 28(1), 162–169 (2017)

    Zhao, B., Lu, H., Chen, S., Liu, J., Wu, D.: Convolutional neural networks for time series classification. Journal of Systems Engineering and Electronics 28(1), 162–169 (2017). https://doi.org/10.21629/JSEE.2017.01.18

    2017 doi

  34. [42]

    Pattern Recogni- tion 74, 171–184 (2018)

    Zhao, J., Itti, L.: shapedtw: Shape dynamic time warping. Pattern Recogni- tion 74, 171–184 (2018). https://doi.org/https://doi.org/10.1016/j.patcog. 2017.09.020

    2018 doi

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.