Pith. sign in

REVIEW 3 major objections 5 minor 90 references

The paper claims that FLASH, a greedy decision-tree optimiser, is the most reliable configuration-tuning choice across all budgets, winning on 20 of 22 systems.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 20:51 UTC pith:32ZHWROP

load-bearing objection Useful, reproducible benchmark, but the FLASH-consistency claim rests on a synthetic objective for 13 of 22 systems; needs sensitivity analysis before I'd trust the headline. the 3 major comments →

arxiv 2607.16476 v1 pith:32ZHWROP submitted 2026-07-17 cs.SE cs.AI

When to Use Which? Benchmarking Optimisers for Configurable Systems under Varying Budgets

Chao Jiang , Yulong Ye , Tao Chen , Miqing Li This is my paper
classification cs.SE cs.AI
keywords configuration tuningoptimiser benchmarkingFLASHgreedy searchfitness landscapebasins of attractionevaluation budgetsmodel-based optimisation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks which software-configuration optimiser a practitioner should use under a given measurement budget, and claims the answer is usually FLASH regardless of budget tightness. It benchmarks eight optimisers on 22 configurable systems under budgets from roughly 100 to 10,000 evaluations, finding that model-based optimisers dominate tight budgets, model-free ones catch up with generous budgets, but FLASH alone stays best on 20 of 22 systems at every budget. The paper explains this through fitness-landscape analysis: many configuration spaces have high-quality local optima with large basins of attraction, so a greedy exploitative search lands quickly in good regions. If correct, this gives a simple default choice for practitioners and a landscape-based explanation for optimiser behaviour that could guide new budget-aware designs.

Core claim

Under evaluation budgets from 100 to 10,000 configurations, FLASH—an optimiser that builds a classification-and-regression-tree surrogate and greedily selects the configuration with maximum predicted performance—is statistically the best or tied-best on 20 of 22 systems regardless of budget tightness, while SMAC and TPE lead only under tight budgets and GA and IRACE only under generous ones. Landscape analysis shows why: relative optimiser quality tracks not the size or dimensionality of the configuration space but the distribution of local optima, specifically how good the local optima are and how many configurations their basins attract. Systems where high-quality local optima have large b

What carries the argument

The central object is FLASH, a model-based optimiser that uses a CART surrogate and a Maximum Mean acquisition function, which produces a greedy, exploitation-heavy search. The explanatory machinery is the basin-of-attraction analysis: each configuration is assigned to the local optimum reached by steepest descent over one-option Hamming neighbours, and each basin is measured by its size and by the quality (regret) of its local optimum. These basin properties carry the argument, connecting greedy behaviour to tuning success and identifying when that greediness fails.

Load-bearing premise

For 13 of the 22 systems, configurations missing from the dataset are assigned the performance of their nearest measured neighbour, so the benchmark's objective function may not match the true system landscape; if this imputation distorts local optima or basins, the rankings—including FLASH's consistency—could change.

What would settle it

Measure performance for every configuration of the 13 incomplete systems (or at least for all configurations those optimisers actually query), rerun the eight optimisers under budgets from 100 to 10,000, and check whether FLASH remains statistically best on 20 of 22 systems; losing top rank on even a handful of systems would undermine the claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Practitioners with tight budgets (under about 500 evaluations) should use FLASH rather than uncertainty-aware Bayesian optimisation.
  • With budgets above a few thousand evaluations, GA or IRACE can rival or beat FLASH, so the best choice shifts with budget.
  • Tuning difficulty depends more on the distribution of local optima and their basins than on system attributes like dimensionality or search-space size.
  • A simple greedy optimiser can outperform more complex model-based and model-free alternatives across most systems, suggesting sophistication is not always necessary.
  • The basin-identification procedure gives a reusable way to characterise a new system's landscape and predict which optimiser class will suit it.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial: the 20-of-22 claim rests on nearest-neighbour imputation for 13 systems; if real measurements distort those landscapes, the exact count could change, so a fully measured re-benchmark is a natural test.
  • Editorial: the paper's own guidance implies an adaptive controller could start with FLASH and switch to GA or IRACE after roughly 500 evaluations, which could be tested on the same benchmark suite.
  • Editorial: the basin-quality view suggests a cheap sampling probe—estimate local optima and basin sizes from a small configuration sample—to predict whether a greedy optimiser will plateau, before spending a full budget.
  • Editorial: the claimed mechanism predicts that artificially constructed landscapes with many low-quality large basins should shift the crossover budget between FLASH and exploration-oriented optimisers in a measurable, repeatable way.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper benchmarks eight configuration-tuning optimisers (SMAC, TPE, FLASH, IRACE, RS, SWAY, SA, GA) on 22 configurable software systems under evaluation budgets from a few dozen to 10,000 configurations. It reports that model-based optimisers (especially FLASH) dominate under tight budgets, while model-free optimisers (GA, IRACE) catch up with larger budgets, and it argues that FLASH is consistently among the best on most systems regardless of budget. The paper then offers a fitness-landscape explanation for this result, claiming that many systems have high-quality local optima with large basins of attraction, which favours FLASH's greedy search. The study uses 30 independent runs, Wilcoxon rank-sum tests with Holm–Bonferroni correction, and publicly promised source code/data.

Significance. If the results are reliable, the paper would provide practically useful guidance for choosing a tuning optimiser under different budget constraints, a question that has received comparatively little systematic study. The breadth of the benchmark (22 systems, real workloads, budgets up to 10,000 evaluations) is a genuine strength, as is the use of repeated measurements and standard statistical testing. The landscape-analysis angle is a reasonable way to move beyond 'which optimiser wins' toward 'why it wins'. However, the central 'FLASH is best in 20/22 systems' claim rests on a partially synthetic objective: for 13 of 22 systems, unmeasured configurations are assigned the performance of their nearest measured neighbour, and regret is computed against the optimum of this imputed landscape. Without validation of that imputation, the headline ranking may be an artifact of the proxy objective rather than a property of the real systems. The paper also overstates the uniqueness of FLASH's 'best' status given the paper's own definition of statistical best.

major comments (3)
  1. [§3.2 and §6] For 13 of the 22 systems (all except Brotli, Apache, HSQLDB, PostgreSQL, x264, LRZIP, Spear, LLVM, and 7z), the paper states that any configuration not in the dataset is assigned the performance of its nearest measured neighbour. This means the objective function used to answer optimiser queries is a piecewise-constant surrogate, not the real system performance, for the majority of the benchmark. Since regret is also computed relative to the optimum of this imputed landscape, the RQ1/RQ2 rankings and the claim that FLASH is best in 20/22 systems may be properties of the imputation scheme. The threats-to-validity section does not mention this at all. I ask the authors to provide a sensitivity analysis: for example, compare rankings on the 9 fully measured systems with those on imputed systems, measure imputation error via cross-validation on the measured configurations, and state explicit
  2. [§3.6, Answer to RQ2, abstract] The definition of 'statistically the best' is that an optimiser is not statistically worse than any other optimiser, so multiple optimisers can be 'best' for the same system. The abstract and Answer to RQ2 say 'FLASH achieves the best in 20/22 systems regardless of the budget tightness', which implies uniqueness. Given the statistical definition, this should be rephrased as 'FLASH is among the statistically best in 20/22 systems', and the paper should report how often FLASH is uniquely best versus tied. The current wording overstates the strength of the finding and is directly tied to the paper's headline contribution.
  3. [§4.3, Algorithm 1] The RQ3 explanation defines basins of attraction via greedy best-improvement local search (Algorithm 1) and then attributes FLASH's success to the presence of large basins of high-quality local optima. Since FLASH itself is a greedy local search method, this explanation is partially self-confirming: a landscape with large basins under best-improvement descent is, by construction, one in which a greedy method should do well. To make the explanation non-circular, the paper should either (a) correlate the basin metrics (number, quality, size, distance) with FLASH's rank across all 22 systems, or (b) compare FLASH's behaviour with a random-restart local search baseline that also exploits basins but has no surrogate model. As presented, the four illustrative systems support the narrative but do not establish a general landscape-level explanation.
minor comments (5)
  1. [Figures 2, 3, 5, 6] Several figure captions contain stray axis labels (e.g., '0 100 200 300 400 500' and '10 2 100 BATLIK' in the caption of Figure 2), and the system name is misspelled as 'BATLIK' instead of 'BATIK' in multiple places. Please fix the captions and typographical errors.
  2. [§3.3 and §6] Many free hyperparameters (SA initial temperature, GA population size/crossover/mutation rates, FLASH's 1,000 surrogate evaluations, initial sampling budget of 30) are taken from previous work without any sensitivity analysis. The parameter-bias threat is acknowledged, but a small sensitivity study on a few systems would make the 'default settings' argument more convincing.
  3. [§3.4] The 2-hour wall-clock termination condition is described as a safeguard when optimisers get stuck, but the interaction between wall-clock time and the evaluation budget is not analysed. Please clarify how often the wall-clock limit was hit per optimiser/system and whether the reported regret trajectories exclude runs truncated by the time limit.
  4. [§3.5] The paper says 'we directly use performance values as a metric' but then reports regret; it would be clearer to state that performance values are used to compute regret, and that raw performance is shown only for representative examples.
  5. [General] The header 'Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009' appears to be a template leftover from the submission format and should be removed.

Circularity Check

1 steps flagged

The FLASH rankings are direct measurements, not derived values; the only partial self-consistency is the RQ3 basin-of-attraction explanation, which defines basins via greedy steepest descent, the same search family as FLASH, but this does not feed back into or force the measured ranking result.

specific steps
  1. other [Section 4.3.1 (Eq. 1, Algorithm 1) and Answer to RQ3 / Abstract]
    "FLASH is designed to favour configurations close to the current best, which results in a greedy search behaviour ... each local optimum has an associated basin of attraction, defined as the set of configurations that eventually converge to that optimum under a local search process (Algorithm 1: 'move to best neighbour') ... many systems possess good local optima (with large basins of attraction), allowing greedy optimisers (e.g., FLASH) to achieve strong performance."

    The RQ3 explanation's key construct, 'basin of attraction', is defined by greedy steepest descent: a configuration belongs to a basin if repeatedly moving to the best neighbour (Algorithm 1) terminates at that local optimum, and basin size is the count of such starting points. FLASH itself is characterised as greedy ('favour configurations close to the current best'). So the explanation 'FLASH performs well because good local optima have large basins' partially restates 'greedy descent from many starting points terminates at good optima' — the explanatory metric embeds the same search behaviour it is invoked to explain. This is only partial self-consistency, not a forced derivation: basins are computed over the full configuration space before any optimiser run, and the RQ1/RQ2 rankings are

full rationale

Walking the derivation chain: the central claims (Finding 3, Answer to RQ2 — 'FLASH achieves the best in 20/22 systems regardless of the budget tightness') are measured outcomes. Each optimiser is run on the benchmark objective and regret is computed directly from the best value found versus the stated optimum of that objective; no parameter is fitted to a subset and then 'predicted' on a close relative, so the fitted-input-called-prediction pattern does not apply. The landscape analysis (RQ3) is post-hoc and does not enter the ranking computation. The only genuine self-consistency is that basin sizes are defined via greedy local search, the same behaviour family as FLASH, making the RQ3 narrative partly a restatement in landscape coordinates; I flag this as a mild step but it does not force the measurement. On self-citation: the paper cites author-overlapping works for methodology — the nearest-neighbour imputation practice '[14, 26, 84]' (Tao Chen in [14]; Ye, Chen, Li in [84]), the 2-hour termination '[16, 45]', and the exclusion of ParamILS '[19], [74]'. These are not load-bearing circularity: each also rests on external sources ([26] Ganguly & Menzies, [74] Pushak & Hoos, external datasets [53, 65, 80]), and none is used to define the FLASH result. The nearer concern is validity, not circularity: for 13 of 22 systems the objective is a nearest-neighbour imputation, the 'known optimum' is the optimum of that imputed landscape, and the threats-to-validity section does not bound the imputation error — but per the review rules this is a correctness risk, not a reduction by construction. Overall: rankings are self-contained measurements; mild self-consistency in the RQ3 explanation; score 2.

Axiom & Free-Parameter Ledger

9 free parameters · 8 axioms · 0 invented entities

This is an empirical study, so the ledger captures experimental and modelling assumptions rather than derived free parameters. The most load-bearing items are the nearest-neighbour imputation and the feasibility of exhaustive basin analysis on massive configuration spaces.

free parameters (9)
  • SA initial temperature = 10
    Set following prior practice [18]; affects SA's exploration-exploitation balance.
  • GA population size = 50
    Standard setting from prior work [13,16,78].
  • GA crossover rate = 0.9
    Uniform crossover rate per [13,16,78].
  • GA mutation rate = 0.1
    Boundary mutation rate per [13,16,78].
  • Initial sampling budget for SMAC/TPE/FLASH = 30
    According to FLASH paper [66].
  • FLASH surrogate evaluations = 1000
    As recommended in [13,14].
  • SWAY initial samples / budget = N and sqrt(N)
    SWAY samples N candidates and measures sqrt(N) per authors' setting [11].
  • Wall-clock termination = 2 hours
    Empirically chosen; most optimisers finish earlier, but this caps slow ones.
  • Significance level = 0.05 (Holm-Bonferroni)
    Statistical test threshold for pairwise comparisons.
axioms (8)
  • domain assumption Dataset performance values (median/mean of repeated measurements) accurately represent true system performance.
    Relies on the quality of datasets from [53,65,80]; invoked in §3.2.
  • domain assumption Nearest-neighbour performance imputation is valid for unmeasured configurations.
    Used for 13/22 systems in §3.2; can alter the optimisation landscape.
  • domain assumption The 22 selected systems are representative of configurable software systems.
    Selection criteria in §3.2; acknowledged as a threat in §6.
  • domain assumption The 8 selected optimisers are representative of general-purpose configurable-system optimisers.
    Selection criteria in §2.2; exclusions (HC, TS, ParamILS, SPL, multi-fidelity) are justified but limit scope.
  • domain assumption Local optima defined via 1-Hamming-distance neighbourhoods and basins via greedy descent meaningfully capture landscape difficulty.
    Definition in §4.3.1; other definitions (e.g., ruggedness) might give different explanations.
  • domain assumption The best-known value in each dataset is the true global optimum.
    Regret computation in §3.5 assumes the optimum is known; for imputed configurations this optimum may be approximated.
  • domain assumption The 2-hour wall-clock cap does not materially change the evaluation-budget comparison.
    Authors state most optimisers reach the max budget within half an hour, but some (e.g., SMAC) stop earlier; this conflates time and evaluation budget.
  • domain assumption Exhaustive basin computation (Algorithm 1) is feasible for all 22 systems including those with 10^9 configurations.
    Algorithm 1 enumerates the full configuration space; the paper does not describe sampling or approximation for large spaces like XZ and VP9.

pith-pipeline@v1.3.0-alltime-deepseek · 20142 in / 13844 out tokens · 125735 ms · 2026-08-01T20:51:20.256012+00:00 · methodology

0 comments
read the original abstract

Software configuration tuning is crucial for optimising system performance, and various optimisers have emerged over the last decade. Yet, the time required during the tuning process may vary across systems. In some systems (e.g., PostgreSQL), it may take a few minutes to measure a configuration, whereas in some others (e.g., MariaDB), it can take several hours. Moreover, even within the same system, users may have varying budgets and preferred settings. This naturally raises a question -- Given a budget level, which optimiser is the best choice for SE practitioners? This matters because optimisers usually have their own ``comfort zone'' and may perform very differently under distinct budgets. In this paper, we aim to answer this question. We systematically evaluate eight well-established optimisers across 22 configurable systems under varying budget levels. We find that, unsurprisingly, model-based optimisers (e.g., SMAC) are well-suited under tight budgets, and model-free optimisers (e.g., GAs) become superior with more generous budgets. However, interestingly, there is one optimiser, FLASH, that performs consistently well on most systems regardless of budgets. We lastly investigate the reasons behind this phenomenon and find that many systems possess good local optima (with large basins of attraction), allowing greedy optimisers (e.g., FLASH) to achieve strong performance. Source code, data, and supplementary materials of this work are available at https://anonymous.4open.science/r/Config-W2W-98B2.

Figures

Figures reproduced from arXiv: 2607.16476 by Chao Jiang, Miqing Li, Tao Chen, Yulong Ye.

Figure 1
Figure 1. Figure 1: The workflow of our empirical study. • SA [51]: Inspired by the annealing process in metallurgy, SA explores the search space by probabilistically accepting worse configurations to escape local optima. • GA [38]: GA is a population-based optimiser that evolves configurations through the processes of selection, crossover, and mutation. • IRACE [62]: IRACE is designed to automatically set algorithmic paramet… view at source ↗
Figure 2
Figure 2. Figure 2: Trajectories of the number of systems where each optimiser is statistically the best out of the 22 systems over 30 independent [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Convergence trajectories of the 8 optimisers on the 4 representative systems ( [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Distribution of regrets for the 4 representative systems ( [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Trajectories of the number of systems where each optimiser is statistically the best out of the 22 systems over 30 independent [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Convergence trajectories of the 8 optimisers on the 4 representative systems ( [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Distributions of local optima (right plots) of the 4 representative systems ( [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

90 extracted references · 7 linked inside Pith

  1. [1]

    2019.Learning very large configuration spaces: What matters for Linux kernel sizes

    Mathieu Acher, Hugo Martin, Juliana Alves Pereira, Arnaud Blouin, Jean-Marc Jézéquel, Djamel Eddine Khelladi, Luc Lesoil, and Olivier Barais. 2019.Learning very large configuration spaces: What matters for Linux kernel sizes. Ph. D. Dissertation. Inria Rennes-Bretagne Atlantique

    2019

  2. [2]

    Apache Software Foundation. 2025. Apache Storm. Retrieved on August 1, 2025, from https://storm.apache.org/

    2025

  3. [3]

    Marvin A Arostegui Jr, Sukran N Kadipasaoglu, and Basheer M Khumawala. 2006. An empirical comparison of tabu search, simulated annealing, and genetic algorithms for facilities location problems.International Journal of Production Economics103, 2 (2006), 742–754. Manuscript submitted to ACM When to Use Which? Benchmarking Optimisers for Configurable System...

    2006

  4. [4]

    Noor Awad, Neeratyoy Mallik, and Frank Hutter. 2021. DEHB: Evolutionary hyperband for scalable, robust and efficient hyperparameter optimization. InIJCAI. 2147–2153

    2021

  5. [5]

    James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011. Algorithms for hyper-parameter optimization. InAdvances in Neural Information Processing Systems, Vol. 24

    2011

  6. [6]

    James Bergstra and Yoshua Bengio. 2012. Random search for hyper-parameter optimization.Journal of Machine Learning Research13, 2 (2012), 281–305

    2012

  7. [7]

    Leo Breiman, Jerome Friedman, R. A. Olshen, and Charles J. Stone. 1984.Classification and Regression Trees. Wadsworth & Brooks/Cole, Belmont, CA, USA

    1984

  8. [8]

    Rong Cao, Liang Bao, Chase Wu, Panpan Zhangsun, Yufei Li, and Zhe Zhang. 2023. CM-CASL: Comparison-based performance modeling of software systems via collaborative active and semisupervised learning.Journal of Systems and Software201 (2023), 111686

    2023

  9. [9]

    Rong Cao, Liang Bao, Panpan Zhangsun, Chase Wu, Shouxin Wei, Ren Sun, Ran Li, and Zhe Zhang. 2024. PTSSBench: a performance evaluation platform in support of automated parameter tuning of software systems.Automated Software Engineering31, 4 (2024)

    2024

  10. [10]

    Rong Cao, Liang Bao, Kaibi Zhao, and Panpan Zhangsun. 2024. ETune: Efficient configuration tuning for big-data software systems via configuration space reduction.Journal of Systems and Software209 (2024), 111936

    2024

  11. [11]

    Sampling

    Jianfeng Chen, Vivek Nair, Rahul Krishna, and Tim Menzies. 2018. “Sampling” as a baseline optimizer for search-based software engineering.IEEE Transactions on Software Engineering45, 6 (2018), 597–614

    2018

  12. [12]

    Junjie Chen, Ningxin Xu, Peiqi Chen, and Hongyu Zhang. 2021. Efficient compiler autotuning via Bayesian optimization. In2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 1198–1209

    2021

  13. [13]

    Pengzhou Chen, Tao Chen, and Miqing Li. 2024. MMO: Meta Multi-Objectivization for Software Configuration Tuning.IEEE Transactions on Software Engineering50, 6 (2024), 1478–1504

    2024

  14. [14]

    Pengzhou Chen, Jingzhi Gong, and Tao Chen. 2025. Accuracy can lie: On the impact of surrogate model in configuration tuning.IEEE Transactions on Software Engineering51, 2 (2025), 548–580

    2025

  15. [15]

    Tao Chen. 2022. Planning landscape analysis for self-adaptive systems. InProceedings of the 17th Symposium on Software Engineering for Adaptive and Self-Managing Systems. 84–90

    2022

  16. [16]

    Tao Chen and Miqing Li. 2021. Multi-objectivizing software configuration tuning. InProceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 453–465

    2021

  17. [17]

    Tao Chen and Miqing Li. 2023. Do performance aspirations matter for guiding software configuration tuning? an empirical investigation under dual performance objectives.ACM Transactions on Software Engineering and Methodology32, 3 (2023), 1–41

    2023

  18. [18]

    Tao Chen and Miqing Li. 2023. The weights can be harmful: Pareto search versus weighted search in multi-objective search-based software engineering.ACM Transactions on Software Engineering and Methodology32, 1 (2023), 1–40

    2023

  19. [19]

    Tao Chen and Miqing Li. 2024. Adapting Multi-objectivized Software Configuration Tuning.Proceedings of the ACM on Software Engineering1, FSE (2024), 539–561

    2024

  20. [20]

    Eugenia Díaz, Javier Tuya, and Raquel Blanco. 2003. Automated software testing using a metaheuristic technique based on tabu search. In18th IEEE International Conference on Automated Software Engineering, 2003. Proceedings.IEEE, 310–313

    2003

  21. [21]

    Wenlong Dong, Wei Liu, Rui Xi, Mengshu Hou, and Shuhuan Fan. 2024. MLETune: Streamlining Database Knob Tuning via Multi-LLMs Experts Guided Deep Reinforcement Learning. In2024 IEEE 30th International Conference on Parallel and Distributed Systems (ICPADS). IEEE, 226–235

    2024

  22. [22]

    Songyun Duan, Vamsidhar Thummala, and Shivnath Babu. 2009. Tuning database configuration parameters with ituned.Proceedings of the VLDB Endowment2, 1 (2009), 1246–1257

    2009

  23. [23]

    Stefan Falkner, Aaron Klein, and Frank Hutter. 2018. BOHB: Robust and efficient hyperparameter optimization at scale. InProceedings of the 35th International Conference on Machine Learning, Vol. 80. PMLR, 1437–1446

    2018

  24. [24]

    Chongjiong Fan, Zhicheng Pan, Wenwen Sun, Chengcheng Yang, and Wei-Neng Chen. 2024. LATuner: An LLM-enhanced database tuning system based on adaptive surrogate model. InECML PKDD. Springer, 372–388

    2024

  25. [25]

    Ayat Fekry, Lucian Carata, Thomas Pasquier, Andrew Rice, and Andy Hopper. 2020. To tune or not to tune? in search of optimal configurations for data analytics. InProceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2494–2504

    2020

  26. [26]

    Kishan Kumar Ganguly and Tim Menzies. 2025. BINGO! Simple optimizers win big if problems collapse to a few buckets.arXiv preprint arXiv:2506.04509(2025)

    Pith/arXiv arXiv 2025

  27. [27]

    Jian Geng, Hongzhi Wang, and Yu Yan. 2024. EMIT: Micro-Invasive Database Configuration Tuning.arXiv preprint arXiv:2406.00616(2024)

    Pith/arXiv arXiv 2024

  28. [28]

    Victor Giannakouris and Immanuel Trummer. 2024. Demonstrating 𝜆-tune: Exploiting large language models for workload-adaptive database system tuning. InCompanion of the 2024 International Conference on Management of Data. 508–511

    2024

  29. [29]

    1998.Tabu search

    Fred Glover and Manuel Laguna. 1998.Tabu search. Springer

    1998

  30. [30]

    Jingzhi Gong and Tao Chen. 2024. Predicting configuration performance in multiple environments with sequential meta-learning.Proceedings of the ACM on Software Engineering1, FSE (2024), 359–382

    2024

  31. [31]

    Edouard Guégain, Amir Taherkordi, and Clément Quinton. 2023. Configuration optimization with limited functional impact. InInternational Conference on Advanced Information Systems Engineering. Springer, 53–68

    2023

  32. [32]

    Bruno Guindani, Danilo Ardagna, Alessandra Guglielmi, Roberto Rocco, and Gianluca Palermo. 2024. Integrating Bayesian optimization and machine learning for the optimal configuration of cloud systems.IEEE transactions on cloud computing12, 1 (2024), 277–294. Manuscript submitted to ACM 20 Jiang et al

    2024

  33. [33]

    Huong Ha, Zongwen Fan, and Hongyu Zhang. 2022. Uncertainty-Aware Performance Prediction for Highly Configurable Software Systems via Bayesian Neural Networks.arXiv preprint arXiv:2212.13359(2022)

    Pith/arXiv arXiv 2022

  34. [34]

    Xue Han and Tingting Yu. 2016. An empirical study on performance bugs for highly configurable software systems. InProceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. 1–10

    2016

  35. [35]

    Nikolaus Hansen and Andreas Ostermeier. 2001. Completely derandomized self-adaptation in evolution strategies.Evolutionary Computation9, 2 (2001), 159–195

    2001

  36. [36]

    Herodotos Herodotou, Yuxing Chen, and Jiaheng Lu. 2020. A survey on automatic parameter tuning for big data processing systems.ACM Computing Surveys (CSUR)53, 2 (2020), 1–37

    2020

  37. [37]

    Robert M Hierons, Miqing Li, Xiaohui Liu, Sergio Segura, and Wei Zheng. 2016. SIP: Optimal product selection from feature models using many-objective evolutionary optimization.ACM Transactions on Software Engineering and Methodology (TOSEM)25, 2 (2016), 1–39

    2016

  38. [38]

    1992.Adaptation in natural and artificial systems: an introductory analysis with applications to biology, control, and artificial intelligence

    John H Holland. 1992.Adaptation in natural and artificial systems: an introductory analysis with applications to biology, control, and artificial intelligence. MIT press

    1992

  39. [39]

    Sture Holm. 1979. A simple sequentially rejective multiple test procedure.Scandinavian journal of statistics(1979), 65–70

    1979

  40. [40]

    Mingyu Huang, Peili Mao, and Ke Li. 2025. Rethinking Performance Analysis for Configurable Software Systems: A Case Study from a Fitness Landscape Perspective.Proceedings of the ACM on Software Engineering2, ISSTA (2025), 1748–1771

    2025

  41. [41]

    Xinmei Huang, Haoyang Li, Jing Zhang, Xinxin Zhao, Zhiming Yao, Yiyan Li, Zhuohao Yu, Tieying Zhang, Hong Chen, and Cuiping Li. 2024. Llmtune: Accelerate database knob tuning with large language models.arXiv preprint arXiv:2404.11581(2024)

    Pith/arXiv arXiv 2024

  42. [42]

    Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. 2011. Sequential model-based optimization for general algorithm configuration. InLearning and Intelligent Optimization. Springer, 507–523

    2011

  43. [43]

    Frank Hutter, Holger H Hoos, Kevin Leyton-Brown, and Thomas Stützle. 2009. ParamILS: an automatic algorithm configuration framework.Journal of Artificial Intelligence Research36 (2009), 267–306

    2009

  44. [44]

    Md Shahriar Iqbal, Ziyuan Zhong, Iftakhar Ahmad, Baishakhi Ray, and Pooyan Jamshidi. 2023. CAMEO: A Causal Transfer Learning Approach for Performance Optimization of Configurable Computer Systems. InProceedings of the 2023 ACM Symposium on Cloud Computing. 555–571

    2023

  45. [45]

    Pooyan Jamshidi and Giuliano Casale. 2016. An uncertainty-aware approach to optimal configuration of stream processing systems. In2016 IEEE 24th International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS). IEEE, 39–48

    2016

  46. [46]

    Pooyan Jamshidi, Norbert Siegmund, Miguel Velez, Christian Kästner, Akshay Patel, and Yuvraj Agarwal. 2017. Transfer learning for performance modeling of configurable systems: An exploratory analysis. In2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 497–508

    2017

  47. [47]

    Chao Jiang, Jingyu Huang, and Miqing Li. 2026. Do We Really Need to Approach the Entire Pareto Front in Many-Objective Bayesian Optimisation? Transactions on Machine Learning Research(2026)

    2026

  48. [48]

    Chao Jiang and Miqing Li. 2025. Multi-objectivising acquisition functions in Bayesian optimisation.ACM Transactions on Evolutionary Learning and Optimization5, 2 (2025)

    2025

  49. [49]

    Chao Jiang and Miqing Li. 2025. Trading off quality and uncertainty through multi-objective optimisation in batch Bayesian optimisation.Proceedings of the AAAI Conference on Artificial Intelligence39, 25 (2025), 27027–27035

    2025

  50. [50]

    Konstantinos Kanellis, Ramnatthan Alagappan, and Shivaram Venkataraman. 2020. Too many knobs to tune? towards faster database tuning by pre-selecting important knobs. In12th USENIX Workshop on Hot Topics in Storage and File Systems (HotStorage 20). USENIX Association

    2020

  51. [51]

    Scott Kirkpatrick, C Daniel Gelatt Jr, and Mario P Vecchi. 1983. Optimization by simulated annealing.Science220, 4598 (1983), 671–680

    1983

  52. [52]

    Ugur Koc, Austin Mordahl, Shiyi Wei, Jeffrey S Foster, and Adam A Porter. 2021. SATune: a study-driven auto-tuning approach for configurable software verification tools. In2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 330–342

    2021

  53. [53]

    Rahul Krishna, Vivek Nair, Pooyan Jamshidi, and Tim Menzies. 2020. Whence to learn? transferring knowledge in configurable systems using beetle. IEEE Transactions on Software Engineering47, 12 (2020), 2956–2972

    2020

  54. [54]

    Harold J Kushner. 1964. A new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise.Journal Basic Engineering86, 1 (1964), 97–106

    1964

  55. [55]

    Jiale Lao, Yibo Wang, Yufei Li, Jianping Wang, Yunjia Zhang, Zhiyuan Cheng, Wanghu Chen, Mingjie Tang, and Jianguo Wang. 2025. GPTuner: An LLM-Based Database Tuning System.ACM SIGMOD Record54, 1 (2025), 101–110

    2025

  56. [56]

    Jieun Lee, Sangmin Seo, Jonghwan Choi, and Sanghyun Park. 2024. K2vTune: A workload-aware configuration tuning for RocksDB.Information Processing & Management61, 1 (2024), 103567

    2024

  57. [57]

    Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. 2018. Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization.Journal of Machine Learning Research18, 185 (2018), 1–52

    2018

  58. [58]

    Yufei Li, Liang Bao, Kaipeng Huang, and Chase Wu. 2025. CSAT: Configuration structure-aware tuning for highly configurable software systems. Journal of Systems and Software222 (2025), 112316

    2025

  59. [59]

    Yiyan Li, Haoyang Li, Zhao Pu, Jing Zhang, Xinyi Zhang, Tao Ji, Luming Sun, Cuiping Li, and Hong Chen. 2024. Is Large Language Model Good at Database Knob Tuning? A Comprehensive Experimental Evaluation.arXiv preprint arXiv:2408.02213(2024)

    Pith/arXiv arXiv 2024

  60. [60]

    Lizhi Liao, Heng Li, Weiyi Shang, and Lei Ma. 2022. An empirical study of the impact of hyperparameter tuning and model optimization on the performance properties of deep neural networks.ACM Transactions on Software Engineering and Methodology (TOSEM)31, 3 (2022), 1–40. Manuscript submitted to ACM When to Use Which? Benchmarking Optimisers for Configurabl...

    2022

  61. [61]

    Marius Lindauer, Katharina Eggensperger, Matthias Feurer, André Biedenkapp, Difan Deng, Carolin Benjamins, Tim Ruhkopf, René Sass, and Frank Hutter. 2022. SMAC3: A versatile Bayesian optimization package for hyperparameter optimization.Journal of Machine Learning Research23, 54 (2022), 1–9

    2022

  62. [62]

    Manuel López-Ibáñez, Jérémie Dubois-Lacoste, Leslie Pérez Cáceres, Mauro Birattari, and Thomas Stützle. 2016. The irace package: Iterated racing for automatic algorithm configuration.Operations Research Perspectives3 (2016), 43–58

    2016

  63. [63]

    Andre Lustosa and Tim Menzies. 2024. Learning from Very Little Data: On the Value of Landscape Analysis for Predicting Software Project Health. ACM Transactions on Software Engineering and Methodology33, 3 (2024), 1–22

    2024

  64. [64]

    MariaDB Foundation. 2025. MariaDB. Retrieved on August 1, 2025, from https://mariadb.org/

    2025

  65. [65]

    Stefan Mühlbauer, Florian Sattler, Christian Kaltenecker, Johannes Dorn, Sven Apel, and Norbert Siegmund. 2023. Analysing the Impact of Workloads on Modeling the Performance of Configurable Software Systems. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2085–2097

    2023

  66. [66]

    Vivek Nair, Zhe Yu, Tim Menzies, Norbert Siegmund, and Sven Apel. 2018. Finding faster configurations using FLASH.IEEE Transactions on Software Engineering46, 7 (2018), 794–811

    2018

  67. [67]

    1972.Human Problem Solving

    Allen Newell and Herbert Simon. 1972.Human Problem Solving. Prentice-Hall, Inc., USA

    1972

  68. [68]

    Changhai Nie and Hareton Leung. 2011. A survey of combinatorial testing.ACM Computing Surveys (CSUR)43, 2 (2011), 1–29

    2011

  69. [69]

    Jeho Oh, Don Batory, and Rubén Heradio. 2023. Finding near-optimal configurations in colossal spaces with statistical guarantees.ACM Transactions on Software Engineering and Methodology33, 1 (2023), 1–36

    2023

  70. [70]

    Jeho Oh, Don Batory, Margaret Myers, and Norbert Siegmund. 2017. Finding near-optimal configurations in product lines by random sampling. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. 61–71

    2017

  71. [71]

    Juliana Alves Pereira, Mathieu Acher, Hugo Martin, Jean-Marc Jézéquel, Goetz Botterweck, and Anthony Ventresque. 2021. Learning software configuration spaces: A systematic literature review.Journal of Systems and Software182 (2021), 111044

    2021

  72. [72]

    Florian Pfisterer, Lennart Schneider, Julia Moosbauer, Martin Binder, and Bernd Bischl. 2022. Yahpo Gym-an efficient multi-objective multi-fidelity benchmark for hyperparameter optimization. InAutoML. PMLR, 3–1

    2022

  73. [73]

    PostgreSQL. 2025. Retrieved on August 1, 2025, from https://www.postgresql.org/

    2025

  74. [74]

    Yasha Pushak and Holger H Hoos. 2020. Golden parameter search: Exploiting structure to quickly configure parameters in parallel. InProceedings of the 2020 Genetic and Evolutionary Computation Conference. 245–253

    2020

  75. [75]

    LA Rastrigin. 1963. The convergence of the random search method in the extremal control of a many parameter system.Automaton & Remote Control24 (1963), 1337–1342

    1963

  76. [76]

    Lohith Senthilkumar and Tim Menzies. 2024. Can Large Language Models Improve SE Active Learning via Warm-Starts?arXiv preprint arXiv:2501.00125(2024)

    Pith/arXiv arXiv 2024

  77. [77]

    Urban Škvorc, Tome Eftimov, and Peter Korošec. 2020. Understanding the problem space in single-objective numerical optimization using exploratory landscape analysis.Applied Soft Computing90 (2020), 106138

    2020

  78. [78]

    Michael Trotter, Timothy Wood, and Jinho Hwang. 2019. Forecasting a storm: Divining optimal configurations using genetic algorithms and supervised learning. In2019 IEEE international conference on autonomic computing (ICAC). IEEE, 136–146

    2019

  79. [79]

    Dana Van Aken, Dongsheng Yang, Sebastien Brillard, Ari Fiorino, Bohan Zhang, Christian Bilien, and Andrew Pavlo. 2021. An inquiry into machine learning-based automatic configuration tuning services on real-world database management systems.Proceedings of the VLDB Endowment14, 7 (2021), 1241–1253

    2021

  80. [80]

    Max Weber, Christian Kaltenecker, Florian Sattler, Sven Apel, and Norbert Siegmund. 2023. Twins or false friends? a study on energy consumption and performance of configurable software. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2098–2110

    2023

Showing first 80 references.