Pith. sign in

REVIEW 2 major objections 7 minor 47 references

Rethinking Performance Analysis for Configurable Software Systems: A Case Study from a Fitness Landscape Perspective

T0 review · 2 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that modeling configurable software systems as fitness landscapes uncovers spatial structure—abundant local optima, scattered peaks, and high-order interactions—that distribution-only analysis misses.

desk verdict Real asset is the 86M-point dataset and GraphFLA framework; the ruggedness headline is not yet supported because local optima are identified without a noise threshold. read the letter →

arxiv 2412.16888 v2 pith:BF3FOZO3 submitted 2024-12-22 cs.PF cs.DCcs.LGcs.SE

Mingyu Huang , Peili Mao , Ke Li This is my paper · ORCID
classification cs.PFcs.DCcs.LGcs.SE
keywords configurablesoftwaresystemsfitnesslandscapeanalysisperformancemodelingconfigurationtuninglocaloptimafeatureinteractionsruggednessgraphmining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the usual way of analyzing configurable software—treating each measured configuration as an isolated data point and looking at the distribution of performance values—throws away the spatial structure of the configuration space. It proposes to view the configuration-performance mapping as a fitness landscape, where neighboring configurations differ by one option and performance is the elevation. Using a graph-based framework on 86 million measured configurations from three real systems, it reports six findings: the landscapes are highly skewed, the best configurations form scattered clusters, local optima are abundant and nearly uniformly distributed, few options matter but their effects change sign with context, higher-order interactions are prevalent, and the accuracy of performance models drops as landscape ruggedness rises. If true, this gives both configuration tuning and performance modeling a new set of quantities to work with—ruggedness, local-optimum density, and spatial distribution—rather than summary statistics alone.

What carries the argument

The carrying mechanism is the configuration landscape, defined as the triple (C, N, f): the set of configurations, a neighborhood structure connecting configurations that differ in exactly one option, and the performance function as elevation. GraphFLA encodes this as a directed attributed graph whose nodes are configurations, node attributes are measured performances, and edges point from worse to better neighbors; local optima are sink nodes. This graph representation turns landscape analysis into graph mining and scales to millions of nodes, and it is the device that lets the paper measure things like local-optimum abundance, autocorrelation along random walks, per-mutation fitness effects over all backgrounds, and pairwise and higher-order interactions.

What would settle it

Re-measure each configuration with many more repetitions, or apply a paired significance test between neighboring configurations, on one or two workloads and recount local optima; if the proportion of local optima drops far below the reported 2.56% to 4.24% or loses its near-uniform spatial distribution, the ruggedness and scattering claims would collapse.

Watch

Extended reading notes

Core claim

The central empirical discovery is that the configuration spaces of real-world software (LLVM, Apache, SQLite) are highly rugged fitness landscapes. Across 32 workloads, local optima make up 2.56% to 4.24% of all configurations, a proportion comparable to a maximally rugged NK landscape, and even nearest-neighbor configurations have only weakly correlated fitness values. The local optima are almost uniformly scattered, most lie far from the global optimum, and the global optimum moves to a different region when the workload changes. At the option level, only a small fraction of options have significant average effects, yet a single option's effect can flip sign depending on the other options, and interactions of order higher than two contribute substantially to performance variance. These observations, the paper argues, are invisible to distribution-based performance analysis and have direct consequences: optimizers must contend with many traps, and performance models trained on random samples lose accuracy as ruggedness increases.

Load-bearing premise

The results depend on the 10-repetition mean being an accurate enough measurement of true performance that declaring one configuration better than a neighbor is meaningful; the paper applies no statistical significance threshold, so if measurement noise is larger than the fitness gaps, some of the counted local optima and their spatial distribution would be artifacts.

Editorial extensions

If this is right

  • Tuning algorithms should expect abundant local optima; strategies like random restarting, multi-objectivization, or landscape smoothing respond to a measured property of the space rather than an assumed one.
  • Performance-model accuracy is not uniform across systems: a landscape's local-optima density predicts how well a random-forest or DNN surrogate will fit it.
  • Directly reusing top configurations across workloads is risky—top-1% sets barely overlap and ranks can shift by tens of percent—so transfer learning needs workload-aware selection.
  • Because only a few options carry most of the influence, search can focus on those options, but per-option effects should not be read as constant since they change sign with context.
  • High-order interactions are common, so models that assume only pairwise interactions will systematically miss part of the performance variance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One testable extension is to apply the same graph-based pipeline to other black-box tuning spaces such as hyperparameters, compiler flags, or database knobs, and compare local-optimum densities across domains; the paper's findings would gain strength if the abundance and scattering patterns replicate elsewhere.
  • The near-uniform scattering of local optima suggests that restart-based global search may be more robust than a single long run, since there is no single basin to escape from.
  • A natural stress test is to apply the framework to synthetic landscapes with known ruggedness, verifying that the graph-mining local-optimum counts are exact at the sampled resolution before trusting them on measured data.
  • The reported link between ruggedness and model error implies that sampling strategies for performance modeling could be adaptive: spend more evaluations in regions likely to contain local optima rather than sampling uniformly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 7 minor

Summary. The paper proposes a fitness landscape perspective for performance analysis of configurable software systems, implemented in an open-source graph-based framework called GraphFLA. The authors benchmark 86M configurations from 32 workloads of LLVM, Apache, and SQLite, using exhaustive enumeration of 15-20 selected options to construct full configuration landscapes. They report six findings (F1-F6) covering fitness distribution skew, scattered and transfer-unfriendly top-1% regions, abundant local optima and ruggedness, uniform spatial distribution of local optima, sparse individual fitness effects with strong context dependence, and prevalence of higher-order interactions. They also connect landscape ruggedness to the predictive performance of random forest and DNN models, and study surrogate-guided simulated annealing.

Significance. The paper's central proposition is that spatial/neighborhood information reveals landscape features invisible to distribution-level performance analysis. The scale of the data collection (86M configurations, 6 months of benchmarking) and the open-sourcing of both code and data are major strengths; the GraphFLA framework itself is a reusable contribution. If the ruggedness finding survives a noise-robustness check, the paper offers a new analytical lens for configurable systems and a testable claim that landscape ruggedness predicts model performance. The descriptive findings are mostly transparently derived, though the local-optima analysis currently lacks the statistical rigor needed to support the strongest claims.

major comments (2)
  1. [Sections 3.2.2, 4.3, and 5] Local optima are declared by comparing the mean performance of a configuration against all Hamming-distance-1 neighbors, with no significance threshold or effect-size criterion. Section 3.2.2 states that each configuration is evaluated with 10 runs and the mean is used; Section 5 defends the protocol only via 'the width of the 95% obtained confidence interval is typically < 10% of the measured mean.' This bound is not the relevant quantity: a 10% confidence interval on each mean translates to a much larger noise-to-signal ratio for the tiny pairwise differences that determine whether a node is a local optimum. With graph degrees of 16 to 20 and millions of configurations, even a small per-comparison false-positive rate produces tens of thousands of spurious local optima, exactly the 10^4-10^5 range reported in F3. The comparison to a maximally rugged NK landscape (k=19, ~4.43% local optima) is therefore not evidence of true ruggedness; a random-noise null model could produce similar counts. The paper's own limitation statement in Section 5 ('we do not expect such bias to significantly affect our results') is an assertion, not a demonstration. Please add a noise-robustness analysis: for example, perturb each measured mean with synthetic noise calibrated to the measured run-to-run variance, recompute local optima under several noise levels, and report how many detected local optima survive; alternatively, apply a paired significance test or a minimum effect size per edge before declaring a local optimum. Unless the counts are robust to such perturbations, F3, F4, and the Section 4.7 ruggedness-R^2 correlation are not established.
  2. [Section 4.7] The correlation between local optima abundance and RF R^2 in Figure 7 is based on only 12 (LLVM), 10 (SQLite), and 10 (Apache) workload-level data points. The Spearman coefficients (-0.73, -0.84, -0.48) are suggestive, but with these sample sizes the 95% confidence intervals are wide, and the analysis inherits the spurious-local-optima problem from the previous comment: if measurement noise inflates the apparent local-optima count more strongly in some workloads, the observed negative correlation could be an artifact of noise rather than a genuine landscape effect. Please report confidence intervals or bootstrap/permutation test results, and, if possible, re-run the correlation using noise-corrected local-optima counts.
minor comments (7)
  1. [Section 5] In the first paragraph, 'interal validity' should be 'internal validity'.
  2. [Conclusion] In the opening sentence, 'configuraiton' should be 'configuration'.
  3. [Table 1 and Section 3.2.1] Table 1 reports 10 workloads for Apache, but Section 3.2.1 says 'we created 9 workloads'; please reconcile the count.
  4. [General] The manuscript references an appendix multiple times (e.g., Section 3.2.1 option selection, Figure 6 caption) but no appendix appears in the provided text; please include the appendix or remove/redirect the references.
  5. [Footnote, page 1] The footnote stating that the paper is an ISSTA 2025 accepted paper and not the final camera-ready version is unnecessary in a journal submission and should be removed.
  6. [Running head] The running head 'Trovato et al.' appears to be a leftover template artifact and should be replaced with the correct running title.
  7. [Section 4.2, panel F description] The text refers to 'SQLlite' in the description of panel F; the correct spelling is 'SQLite'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the landscape findings are direct measurements of exhaustively enumerated configurations, and the Section 4.7 modeling experiments are observed correlations between independently measured quantities rather than fitted-parameter predictions.

full rationale

The central claims are descriptive measurements over exhaustively enumerated configuration landscapes; no fitted parameter is later relabeled as a prediction. Local optima in Section 4.3 are computed by direct Hamming-neighbor comparisons of the 10-run mean fitness values defined in Section 3.2, and the NK-landscape comparison (approximately 4.43% local optima for k=19, n=20) is an external reference from Kauffman (1993), not an output of a model fitted to the same data. The R-squared-versus-ruggedness analysis in Section 4.7 is an observed correlation between two independently measured quantities; the models are trained on 1% of configurations and tested on the remainder, so their accuracy is not constructed from the local-optima counts. Author self-citations (e.g., Huang and Li 2023a,b,c and Zhou et al. 2024) are used as related-work context and do not carry the load-bearing argument. No uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via citation. The one flagged weakness is in Section 5, where the authors state that 'the width of the 95% obtained confidence interval is typically < 10% of the measured mean' and argue that measurement bias 'should not significantly affect our results.' This is an unsupported robustness assertion about pairwise neighbor comparisons, but it is a threat-to-validity and correctness concern, not a circular derivation. No equation or definition in the paper reduces a claimed result to its own input; the paper is self-contained against its own benchmarks and reports direct empirical observations. Therefore the appropriate score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The study is empirical and descriptive. The main hand-chosen choices are the top-1% cutoff, the neighborhood radius, and the number of repetitions. The central risk is not fitted parameters but unquantified measurement noise affecting local optima detection.

free parameters (3)
  • Top-1% prominence threshold = 0.01
    Chosen by hand to define prominent regions; directly affects the cluster counts, Jaccard overlaps, and rank shift statistics in F2.
  • Neighborhood distance threshold = 1
    Defines neighbors as configurations differing in exactly one option; determines local optima counts, autocorrelation lags, and most spatial claims.
  • Number of repeated measurements = 10
    Mean of 10 runs is used as the fitness value; this choice controls noise and can influence the reported number of local optima.
assumptions (3)
  • domain assumption The mean of 10 benchmark runs is accurate enough that residual measurement noise does not materially affect local optima or global optimum identification.
    Load-bearing for F3 and F4; only confidence interval width below 10% is reported, with no significance threshold for declaring a neighbor better.
  • domain assumption Hamming and Manhattan distances with a threshold of 1 define a meaningful neighborhood structure for the configuration landscape.
    The entire landscape representation depends on this neighborhood definition, which is standard in FLA but not independently validated for these systems.
  • domain assumption The selected 15 to 20 options and 32 workloads are representative enough to support the general claims about configurable software systems.
    The paper acknowledges limited external validity from three systems; the findings are empirical and may not generalize.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking Performance Analysis for Configurable Software Systems: A Case Study from a Fitness Landscape Perspective." pith.science (2026). https://pith.science/paper/BF3FOZO3

@misc{pith2026241216888,
  author       = {Pith},
  title        = {Pith review of: Rethinking Performance Analysis for Configurable Software Systems: A Case Study from a Fitness Landscape Perspective},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BF3FOZO3}},
  note         = {Machine review of arXiv:2412.16888}
}
abstract

Modern software systems are often highly configurable to tailor varied requirements from diverse stakeholders. Understanding the mapping between configurations and the desired performance attributes plays a fundamental role in advancing the controllability and tuning of the underlying system, yet has long been a dark hole of knowledge due to its black-box nature. While there have been previous efforts in performance analysis for these systems, they analyze the configurations as isolated data points without considering their inherent spatial relationships. This renders them incapable of interrogating many important aspects of the configuration space like local optima. In this work, we advocate a novel perspective to rethink performance analysis -- modeling the configuration space as a structured ``landscape''. To support this proposition, we designed \our, an open-source, graph data mining empowered fitness landscape analysis (FLA) framework. By applying this framework to $86$M benchmarked configurations from $32$ running workloads of $3$ real-world systems, we arrived at $6$ main findings, which together constitute a holistic picture of the landscape topography, with thorough discussions about their implications on both configuration tuning and performance modeling.

Figures

Figures reproduced from arXiv: 2412.16888 by the authors.

Figure 1
Figure 1. In software configuration space (panel B), configurations are spatially related to each other, and so are their associated performance values. Traditional performance analysis (panel C) only considers the distribution of performance values as isolated data points. Our approach (panel A), instead, additionally incorporates the neighborhood relationships between configurations, which are used to construct a configurat… view at source ↗
Figure 2
Figure 2. Schematic overview of of GraphFLA. (Han et al., 2021, Velez et al., 2020, 2021, Weber et al., 2021)), our approach is complementary to them, as we focus on the black-box perspective. 3 The FLA Framework and Case Study Setup 3.1 GraphFLA As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. General fitness distributions and prominent regions. (A) Normalized fitness distributions of three workloads of LLVM. (B) Spearman’s 𝜌 of fitness between all pairs of workloads in each system. (C) Distributions of pairwise distance between top-1% configurations for a LLVM landscape and randomly sampled configurations. (D) 2D projection of the distribution of top-1% configurations in a LLVM landscape as compared to t… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Local optima, global optimum, and landscape ruggedness. (A) Number of local optima and total configurations in landscapes, error bars indicate s.t.d. across workloads. (B) Autocorrelations of landscapes of each system under different lag distances, aggregated across wo…
Figure 5
Figure 5. Figure 5: Individual and interactive fitness effects of options. (A). The distribution of fitness effects (normalized) of altering a single option of LLVM in all possible configuration backgrounds. (B) Aggregated fitness effects (i.e., feature importance) of each option across a…
Figure 6
Figure 6. Figure 6: Option interaction network for three workloads of LLVM. Each node represents a configurable option of LLVM (a look-up table is available in the Appendix). Positive interactions and negative interactions are colored differently. Non-statistical significant interactions …
Figure 7
Figure 7. Figure 7: The predictive power of performance models in fitting a landscape is highly dependent on its ruggedness. Here each point represents a workload, and the prevalence of local optima in the landscape is plotted against the 𝑅 2 score of a random forest regressor in fitting …
Figure 8
Figure 8. Figure 8: (A). Residual plots of the fitness predictions of RF and DNN models on a random workload of LLVM. Linear regression lines with 95% confidence interval (shading) are displayed. (B). Recall of the top-1, 000 configurations in the landscape selected by the two models unde…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 45 canonical work pages

  1. [5]

    ACM Trans

    HINNPerf: Hierarchical Interaction Neural Network for Performance Prediction of Configurable Systems. ACM Trans. Softw. Eng. Methodol. 32, 2 (2023), 46:1–46:30. J. Arjan G. M. de Visser and Joachim Krug

    work page 2023

  2. [8]

    Network topology of a potential energy landscape: A static scale-free network. Phy. Rev. Lett. 88, 23 (2002), 238701. Chase R. Freschlin, Sarah A. Fahlberg, Pete Heinzelman, and Philip A. Romero

    work page 2002

  3. [14]

    Algorithm runtime prediction: Methods & evaluation. Artif. Intell. 206 (2014), 79–111. Pooyan Jamshidi, Norbert Siegmund, Miguel Velez, Christian Kästner, Akshay Patel, and Yuvraj Agarwal. 2017a. Transfer learning for performance modeling of configurable systems: an exploratory analysis. InASE’17: Proc. of the 32nd IEEE/ACM International Conference on Aut...

    work page 2014

  4. [17]

    In NeurIPS’19: Proc

    Meta-Surrogate Benchmarking for Hyperparameter Optimization. In NeurIPS’19: Proc. of Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019 . 6267–6277. Sergiy S. Kolesnikov, Norbert Siegmund, Christian Kästner, Alexander Grebhahn, and Sven Apel

    work page 2019

  5. [18]

    Tradeoffs in modeling performance of highly configurable software systems. Softw. Syst. Model. 18, 3 (2019), 2265–2283. Min Li, Liangzhao Zeng, Shicong Meng, Jian Tan, Li Zhang, Ali Raza Butt, and Nicholas C. Fuller

    work page 2019

  6. [19]

    ACM Trans

    Learning from Very Little Data: On the Value of Landscape Analysis for Predicting Software Project Health. ACM Trans. Softw. Eng. Methodol. 33, 3 (2024), 58:1–58:22. Katherine M. Malan

    work page 2024

  7. [20]

    Algorithms 14, 2 (2021),

    A Survey of Advances in Landscape Analysis for Optimisation. Algorithms 14, 2 (2021),

    work page 2021

  8. [22]

    IEEE Trans

    Transfer Learning Across Variants and Versions: The Case of Linux Kernel Size. IEEE Trans. Software Eng. 48, 11 (2022), 4274–4290. Duncan C. McElfresh, Sujay Khandagale, Jonathan Valverde, Vishak Prasad C., Ganesh Ramakrishnan, Micah Goldblum, and Colin White

    work page 2022

Show all 47 references
  1. [23]

    UMAP: Uniform Manifold Approximation and Projection. J. Open Source Softw. 3, 29 (2018),

    2018

  2. [24]

    CoRR abs/2304.02396 (2023)

    AutoRL Hyperpa- rameter Landscapes. CoRR abs/2304.02396 (2023). Stefan Mühlbauer, Florian Sattler, Christian Kaltenecker, Johannes Dorn, Sven Apel, and Norbert Siegmund

    2023 arXiv

  3. [25]

    In ICSE’23: Proc

    Analysing the Impact of Workloads on Modeling the Performance of Configurable Software Systems. In ICSE’23: Proc. of the IEEE/ACM 45th International Conference on Software Engineering . IEEE Computer Society, 2085–2097. Todd Mytkowicz, Amer Diwan, Matthias Hauswirth, and Peter...

  4. [26]

    In ESEC/FSE’17: Proc

    Using bad learners to find good configurations. In ESEC/FSE’17: Proc. of the 2017 11th Joint Meeting on Foundations of Software Engineering . ACM, 257–267. Vivek Nair, Tim Menzies, Norbert Siegmund, and Sven Apel

    2017

  5. [27]

    Faster discovery of faster system configurations with spectral learning. Autom. Softw. Eng. 25, 2 (2018), 247–277. Vivek Nair, Zhe Yu, Tim Menzies, Norbert Siegmund, and Sven Apel

    2018

  6. [28]

    IEEE Trans

    Finding Faster Configurations Using FLASH. IEEE Trans. Software Eng. 46, 7 (2020), 794–811. Jeho Oh, Don S. Batory, Margaret Myers, and Norbert Siegmund

    2020

  7. [29]

    Proceedings of the National Academy of Sciences 111, 22 (2014), E2301–E2309

    Inferring fitness landscapes by regression produces biased estimates of epistasis. Proceedings of the National Academy of Sciences 111, 22 (2014), E2301–E2309. Mingdong Ou, Peng Cui, Jian Pei, Ziwei Zhang, and Wenwu Zhu

    2014

  8. [30]

    Science 382, 6673 (2023), eadh3860

    A rugged yet easily navigable fitness landscape. Science 382, 6673 (2023), eadh3860. Mateusz Pawlik and Nikolaus Augsten

    2023

  9. [32]

    IEEE Trans

    Maximum Satisfiability: Anatomy of the Fitness Landscape for a Hard Combinatorial Optimization Problem. IEEE Trans. Evol. Comput. 16, 3 (2012), 319–338. Yasha Pushak and Holger H. Hoos

    2012

  10. [33]

    ACM Trans

    AutoML Loss Landscapes. ACM Trans. Evol. Learn. Optim. 2, 3 (2022), 10:1–10:30. Cedric J. Rodriguez, Sarah L. Thomson, Tanja Alderliesten, and Peter A. N. Bosman

    2022

  11. [35]

    In EMNLP’17: Proc

    Learning to select data for transfer learning with Bayesian Optimization. In EMNLP’17: Proc. of the 2017 Conference on Empirical Methods in Natural Language Processing . Association for Computa- tional Linguistics, 372–382. Atri Sarkar, Jianmei Guo, Norbert Siegmund, Sven Apel...

    2017

  12. [36]

    CoRR abs/2008.09777 (2020)

    NAS-Bench-301 and the Case for Surrogate Benchmarks for Neural Architecture Search. CoRR abs/2008.09777 (2020). Rethinking Performance Analysis for Configurable Soft. Sys.: A Case Study from a Fitness Landscape Perspective 23 Mohammad-Hassan Tayarani-Najaran and Adam Prügel-Bennett

    2020 arXiv

  13. [37]

    IEEE Trans

    On the Landscape of Combinatorial Optimization Problems. IEEE Trans. Evol. Comput. 18, 3 (2014), 420–434. Paul Temple, Mathieu Acher, Jean-Marc Jézéquel, and Olivier Barais

    2014

  14. [38]

    IEEE Softw

    Learning Contextual-Variability Models. IEEE Softw. 34, 6 (2017), 64–70. Eeshit Dhaval Vaishnav, Carl G. de Boer, Jennifer Molinet, Moran Yassour, Lin Fan, Xian Adiconis, Dawn A. Thompson, Joshua Z. Levin, Francisco A. Cubillos, and Aviv Regev

    2017

  15. [39]

    Nature 603, 7901 (2022), 455–463

    The evolution, evolvability and engineering of gene regulatory DNA. Nature 603, 7901 (2022), 455–463. Pavel Valov, Jean-Christophe Petkovich, Jianmei Guo, Sebastian Fischmeister, and Krzysztof Czarnecki

    2022

  16. [40]

    ConfigCrusher: towards white-box performance analysis for configurable systems. Autom. Softw. Eng. 27, 3 (2020), 265–300. Miguel Velez, Pooyan Jamshidi, Norbert Siegmund, Sven Apel, and Christian Kästner

    2020

  17. [44]

    In ECML/PKDD’15: Proc

    Hyperparameter Search Space Pruning - A New Component for Sequential Model-Based Hyperparameter Optimization. In ECML/PKDD’15: Proc. of the 2015 Machine Learning and Knowledge Discovery in Databases - European Conference on Machine Learning , Vol

    2015

  18. [45]

    In SIGMETRICS’03: Proc of the 2003 International Conference on Measurements and Modeling of Computer Systems

    A recursive random search algorithm for large-scale network parameter configuration. In SIGMETRICS’03: Proc of the 2003 International Conference on Measurements and Modeling of Computer Systems. ACM, 196–205. Cemal Yilmaz, Emine Dumlu, Myra B. Cohen, and Adam A. Porter

    2003

  19. [46]

    IEEE Trans

    Reducing Masking Effects in CombinatorialInter- action Testing: A Feedback DrivenAdaptive Approach. IEEE Trans. Software Eng. 40, 1 (2014), 43–66. Yi Zhang, Jianmei Guo, Eric Blais, and Krzysztof Czarnecki

    2014

  20. [47]

    Evolutionary Multi-objective Optimization for Contextual Adversarial Example Generation. Proc. ACM Softw. Eng. 1, FSE (2024), 2285–2308

    2024

  21. [1990]

    Correlated and uncorrelated fitness landscapes and how to tell the difference. Biol. Cybern. 63, 5 (1990), 325–336. Thomas Weise

    1990

  22. [1995]

    In ICDAR’95: Proc

    Random decision forests. In ICDAR’95: Proc. of the 3rd International Conference on Document Analysis and Recognition. IEEE Computer Society, 278–282. Mingyu Huang and Ke Li. 2023a. Exploring Structural Similarity in Fitness Landscapes via Graph Data Mining: A Case Study on Num...

    2023 arXiv

  23. [2000]

    The Earth Mover’s Distance as a Metric for Image Retrieval.Int. J. Comput. Vis. 40, 2 (2000), 99–121. Sebastian Ruder and Barbara Plank

    2000

  24. [2002]

    SMOTE: Synthetic Minority Over-sampling Technique. J. Artif. Intell. Res. 16 (2002), 321–357. Tao Chen and Miqing Li

    2002

  25. [2003]

    Energy Landscape Theory, Funnels, Specificity, and Optimal Criterion of Biomolecular Binding. Phys. Rev. Lett. 90 (May 2003), 188101. Issue

    2003

  26. [2009]

    Self-Published Thomas Weise 361 (2009)

    Global optimization algorithms-theory and application. Self-Published Thomas Weise 361 (2009). Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thieme

    2009

  27. [2012]

    ACM Comput

    Search-based software engineering: Trends, techniques and applications. ACM Comput. Surv. 45, 1 (2012), 11:1–11:61. Tin Kam Ho

    2012

  28. [2013]

    A survey of techniques for characterising fitness landscapes and some possible ways forward. Inf. Sci. 241 (2013), 148–163. 22 Trovato et al. Hugo Martin, Mathieu Acher, Juliana Alves Pereira, Luc Lesoil, Jean-Marc Jézéquel, and Djamel Eddine Khelladi

    2013

  29. [2014]

    Empirical fitness landscapes and the predictability of evolution. Nat. Rev. Gen. 15, 7 (2014), 480–490. Xiaoan Ding, Yi Liu, and Depei Qian

    2014

  30. [2015]

    ACM Trans

    Efficient Computation of the Tree Edit Distance. ACM Trans. Database Syst. 40, 1 (2015), 3:1–3:40. A. Prügel-Bennett and M-H Tayarani-Najaran

    2015

  31. [2016]

    ACM Comput

    Workload Characterization: A Survey Revisited. ACM Comput. Surv. 48, 3 (2016), 48:1–48:43. Nitesh V. Chawla, Kevin W. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer

    2016

  32. [2017]

    A thousand empirical adaptive landscapes and their navigability. Nat. Ecol. Evol. 1, 2 (2017),

    2017

  33. [2018]

    Data-efficient performance learning for configurable systems. Empir. Softw. Eng. 23, 3 (2018), 1826–1867. Jichi Guo, Qing Yi, and Apan Qasem

    2018

  34. [2019]

    In Proceedings of the 41st International Conference on Software Engineering, ICSE 2019, Montreal, QC, Canada, May 25-31, 2019

    Distance-based sampling of software configuration spaces. In Proceedings of the 41st International Conference on Software Engineering, ICSE 2019, Montreal, QC, Canada, May 25-31, 2019 . IEEE / ACM, 1084–1094. Stuart A. Kauffman

    2019

  35. [2020]

    University of Chicago, Tech

    Bayesian learning for hardware and software configuration co-optimization. University of Chicago, Tech. Rep 13 (2020). Johannes Dorn, Sven Apel, and Norbert Siegmund

    2020

  36. [2021]

    ACM Trans

    Information Reuse and Stochastic Search: Managing Uncertainty in Self-* Systems. ACM Trans. Auton. Adapt. Syst. 15, 1 (2021), 3:1–3:36. Andrew Kirjner, Jason Yim, Raman Samusevich, Shahar Bracha, Tommi S. Jaakkola, Regina Barzilay, and Ila R. Fiete

    2021

  37. [2022]

    InMSR’22: Proc

    Does Configuration Encoding Matter in Learning Software Performance? An Empirical Study on Encoding Schemes. InMSR’22: Proc. of the 19th IEEE/ACM International Conference on Mining Software Repositories. ACM, 482–494. Jingzhi Gong and Tao Chen. 2024a. Deep Configuration Perfor...

    2024 arXiv

  38. [2023]

    ACM Trans

    Do Performance Aspirations Matter for Guiding Software Configuration Tuning? An Empirical Investigation under Dual Performance Objectives. ACM Trans. Softw. Eng. Methodol. 32, 3 (2023), 68:1–68:41. Jiezhu Cheng, Cuiyun Gao, and Zibin Zheng

    2023

  39. [2024]

    Nature Communications 15, 1 (2024),

    Neural network extrapolation to distant regions of the protein fitness landscape. Nature Communications 15, 1 (2024),

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.