{"id":"55266a0e-ee6b-465b-be5f-d14d6324d488","arxiv_id":"2412.16888","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Modeling configurable software performance as a spatial fitness landscape reveals highly rugged terrain, many scattered local optima, few consistently important options, and prevalent high-order interactions.","lead":"This paper treats the space of software configuration options as a fitness landscape, and analyzes about 86 million measured performance points across 32 workloads of three real-world systems. It reports six findings about how performance peaks and local optima are distributed, and releases a graph-based analysis framework plus the benchmark data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"F3's local-optima abundance is the load-bearing claim; it is computed from 10-run means with no significance threshold, and the reported '<10% of mean' CI does not bound pairwise neighbor differences, so measurement noise may create the observed ruggedness.","rationale":"The reader correctly identifies the weakest assumption: the local-optima counts in F3 are computed from 10-run means without any significance threshold, so measurement noise can create spurious local optima whenever true neighboring fitness differences are small relative to run-to-run variability. This is the single most load-bearing concern because F3 is the paper's strongest empirical claim, F4 directly builds on the set of local optima, and Section 4.7 connects local-optima abundance to performance-model accuracy. The paper's own defense, a confidence-interval width below 10% of the mean, does not address the pairwise-difference question; it is a relative-to-mean statement, not a bound on neighbor-to-neighbor errors. The NK-landscape comparison makes the risk concrete, since a k=19 NK landscape is nearly random, so the observed 2.56% to 4.24% proportions are in the range that pure noise could plausibly produce. The proposed bootstrap test would settle the question by comparing observed local-optima counts against a noise-only null model. I agree with the reader's assessment and find no reason to change the conditional verdict: the framework, dataset, and six findings are useful, but the central empirical claim needs a noise-controlled analysis before it can be taken at face value. The option-count inconsistency between the text and Table 1 is real but secondary, and the absent concrete code/data links are also secondary to the noise concern.","tokens_in":22995,"tokens_out":4049,"duration_ms":45972,"concrete_test":"Run a bootstrap noise-injection test on each of the 32 landscapes: use the 10 individual runs per configuration (or re-benchmark a random subset of 10k configurations with 30+ repetitions) to estimate each configuration's measurement-error distribution. Then generate 1,000 synthetic landscapes by adding independent noise (e.g., t-distributed with 9 degrees of freedom, scaled to the empirical residual variance) to a noise-free null model fitted to each landscape, such as an additive model with option main effects only, and count local optima in each synthetic landscape. Compare the median synthetic local-optima proportion to the reported 2.56% to 4.24% range. If the noise-only null already yields comparable proportions, F3 is not supported; if it yields near-zero proportions, the noise concern is settled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2.2 states that each configuration is evaluated with 10 independent runs and the mean is used, and Section 5 defends this by noting that the 95% confidence interval width is typically below 10% of the measured mean. But local-optimum identification (Sections 2 and 4.3) compares a configuration's mean fitness with the means of all Hamming-distance-1 neighbors, and no significance threshold or effect-size criterion is applied. A confidence interval expressed as a fraction of the mean is not the relevant quantity: what matters is the noise-to-signal ratio of pairwise neighbor differences. In landscapes with 1.7M to 5.7M configurations per workload and degree 16 to 20, a small per-comparison false-positive rate produces tens of thousands of spurious local optima, which is exactly the 10^4 to 10^5 range reported in F3. The comparison to a maximally rugged NK landscape (k=19, proportion of local optima about 4.43%) reinforces the concern, because such a landscape is nearly random; if the observed proportions of 2.56% to 4.24% arise from noise alone, F3 and the downstream findings that depend on it (F4, and the ruggedness-versus-R^2 correlation in Section 4.7) would be artifacts. The autocorrelation evidence in Figure 4B is also attenuated by measurement noise and does not by itself separate signal from error. The paper's own limitation statement in Section 5 asserts that the bias is not expected to significantly affect results, but that assertion is precisely the step that needs empirical support.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a fitness landscape perspective for performance analysis of configurable software systems, implemented in an open-source graph-based framework called GraphFLA. The authors benchmark 86M configurations from 32 workloads of LLVM, Apache, and SQLite, using exhaustive enumeration of 15-20 selected options to construct full configuration landscapes. They report six findings (F1-F6) covering fitness distribution skew, scattered and transfer-unfriendly top-1% regions, abundant local optima and ruggedness, uniform spatial distribution of local optima, sparse individual fitness effects with strong context dependence, and prevalence of higher-order interactions. They also connect landscape ruggedness to the predictive performance of random forest and DNN models, and study surrogate-guided simulated annealing.","tokens_in":23273,"tokens_out":5232,"duration_ms":42187,"significance":"The paper's central proposition is that spatial/neighborhood information reveals landscape features invisible to distribution-level performance analysis. The scale of the data collection (86M configurations, 6 months of benchmarking) and the open-sourcing of both code and data are major strengths; the GraphFLA framework itself is a reusable contribution. If the ruggedness finding survives a noise-robustness check, the paper offers a new analytical lens for configurable systems and a testable claim that landscape ruggedness predicts model performance. The descriptive findings are mostly transparently derived, though the local-optima analysis currently lacks the statistical rigor needed to support the strongest claims.","major_comments":[{"comment":"Local optima are declared by comparing the mean performance of a configuration against all Hamming-distance-1 neighbors, with no significance threshold or effect-size criterion. Section 3.2.2 states that each configuration is evaluated with 10 runs and the mean is used; Section 5 defends the protocol only via 'the width of the 95% obtained confidence interval is typically < 10% of the measured mean.' This bound is not the relevant quantity: a 10% confidence interval on each mean translates to a much larger noise-to-signal ratio for the tiny pairwise differences that determine whether a node is a local optimum. With graph degrees of 16 to 20 and millions of configurations, even a small per-comparison false-positive rate produces tens of thousands of spurious local optima, exactly the 10^4-10^5 range reported in F3. The comparison to a maximally rugged NK landscape (k=19, ~4.43% local optima) is therefore not evidence of true ruggedness; a random-noise null model could produce similar counts. The paper's own limitation statement in Section 5 ('we do not expect such bias to significantly affect our results') is an assertion, not a demonstration. Please add a noise-robustness analysis: for example, perturb each measured mean with synthetic noise calibrated to the measured run-to-run variance, recompute local optima under several noise levels, and report how many detected local optima survive; alternatively, apply a paired significance test or a minimum effect size per edge before declaring a local optimum. Unless the counts are robust to such perturbations, F3, F4, and the Section 4.7 ruggedness-R^2 correlation are not established.","section":"Sections 3.2.2, 4.3, and 5"},{"comment":"The correlation between local optima abundance and RF R^2 in Figure 7 is based on only 12 (LLVM), 10 (SQLite), and 10 (Apache) workload-level data points. The Spearman coefficients (-0.73, -0.84, -0.48) are suggestive, but with these sample sizes the 95% confidence intervals are wide, and the analysis inherits the spurious-local-optima problem from the previous comment: if measurement noise inflates the apparent local-optima count more strongly in some workloads, the observed negative correlation could be an artifact of noise rather than a genuine landscape effect. Please report confidence intervals or bootstrap/permutation test results, and, if possible, re-run the correlation using noise-corrected local-optima counts.","section":"Section 4.7"}],"minor_comments":[{"comment":"In the first paragraph, 'interal validity' should be 'internal validity'.","section":"Section 5"},{"comment":"In the opening sentence, 'configuraiton' should be 'configuration'.","section":"Conclusion"},{"comment":"Table 1 reports 10 workloads for Apache, but Section 3.2.1 says 'we created 9 workloads'; please reconcile the count.","section":"Table 1 and Section 3.2.1"},{"comment":"The manuscript references an appendix multiple times (e.g., Section 3.2.1 option selection, Figure 6 caption) but no appendix appears in the provided text; please include the appendix or remove/redirect the references.","section":"General"},{"comment":"The footnote stating that the paper is an ISSTA 2025 accepted paper and not the final camera-ready version is unnecessary in a journal submission and should be removed.","section":"Footnote, page 1"},{"comment":"The running head 'Trovato et al.' appears to be a leftover template artifact and should be replaced with the correct running title.","section":"Running head"},{"comment":"The text refers to 'SQLlite' in the description of panel F; the correct spelling is 'SQLite'.","section":"Section 4.2, panel F description"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong empirical contribution with a valuable open-source framework and an unusually large benchmark dataset. The main risk is the absence of a noise model in the local-optima analysis; the authors should be required to provide a noise-injection or significance-threshold analysis before the central 'highly rugged' claim can be accepted. The Section 4.7 correlation is also based on a small number of workloads and should be treated as exploratory unless confidence intervals or permutation tests are added. I would not reject the paper on these grounds, as the issues are fixable within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper contributes a genuinely useful artifact: GraphFLA plus an exhaustively enumerated benchmark of 86M configurations across 32 workloads of LLVM, Apache, and SQLite. That data is hard-won, and the graph-based pipeline is a sensible way to expose spatial structure in configuration spaces. Second, the headline empirical claim—F3, that these landscapes are highly rugged with 2.56–4.24% local optima—is not yet supported. Local optima are identified by comparing 10-run means with no significance threshold, and the reported \"CI width <10% of mean\" says nothing about the noise-to-signal ratio of pairwise neighbor differences. With 1.7M–5.7M configurations per workload, even a tiny per-comparison false-positive rate produces tens of thousands of spurious local optima—exactly the range reported. The paper's Section 5 asserts that measurement bias will not significantly affect results, but that is precisely the step that needs empirical support, and the NK-landscape comparison (k=19 is nearly random) does not help.\n\nWhat the paper does well beyond the dataset: the spatial perspective is worth taking seriously, and several findings are more robust than F3 because they are about aggregate patterns rather than fine local comparisons. F2 (top-1% configurations form scattered clusters with little cross-workload overlap) and F5 (per-mutation effects are highly context-dependent) stand on their own. The modular framework and the open data are the strongest reasons to engage.\n\nSoft spots, in proportion: the noise problem is load-bearing, because F3 feeds F4 (distribution of local optima) and the Section 4.7 correlation between local optima abundance and model R^2. Those results inherit the vulnerability. There are also smaller internal inconsistencies: option counts differ between the table and the text (20/18/16 vs 20/15/18), and the configuration-space sizes do not match. The artifact links in the paper are placeholders, so I could not verify code or data availability from the text alone.\n\nWho this is for: researchers in configurable software systems, performance modeling, and empirical landscape analysis. They should use the dataset and the framework, but not cite F3/F4 as established facts. The paper deserves a serious referee. I would conditionally accept with a required noise-controlled reanalysis: apply an effect-size or significance threshold to local-optimum detection, or run noise-injection sensitivity checks, and re-report F3/F4 and the R^2 correlation accordingly.","headline":"Real asset is the 86M-point dataset and GraphFLA framework; the ruggedness headline is not yet supported because local optima are identified without a noise threshold.","tokens_in":23815,"tokens_out":3559,"would_cite":true,"duration_ms":32450,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that modeling configurable software systems as fitness landscapes uncovers spatial structure—abundant local optima, scattered peaks, and high-order interactions—that distribution-only analysis misses.","keywords":["configurable software systems","fitness landscape analysis","performance modeling","configuration tuning","local optima","feature interactions","landscape ruggedness","graph mining"],"falsifier":"Re-measure each configuration with many more repetitions, or apply a paired significance test between neighboring configurations, on one or two workloads and recount local optima; if the proportion of local optima drops far below the reported 2.56% to 4.24% or loses its near-uniform spatial distribution, the ruggedness and scattering claims would collapse.","tokens_in":22750,"feed_emoji":"⛰️","tokens_out":5209,"duration_ms":44497,"temperature":0.7,"pith_summary":"The paper argues that the usual way of analyzing configurable software—treating each measured configuration as an isolated data point and looking at the distribution of performance values—throws away the spatial structure of the configuration space. It proposes to view the configuration-performance mapping as a fitness landscape, where neighboring configurations differ by one option and performance is the elevation. Using a graph-based framework on 86 million measured configurations from three real systems, it reports six findings: the landscapes are highly skewed, the best configurations form scattered clusters, local optima are abundant and nearly uniformly distributed, few options matter but their effects change sign with context, higher-order interactions are prevalent, and the accuracy of performance models drops as landscape ruggedness rises. If true, this gives both configuration tuning and performance modeling a new set of quantities to work with—ruggedness, local-optimum density, and spatial distribution—rather than summary statistics alone.","feed_headline":"86M configs show software tuning landscapes are rugged","feed_subtitle":"Local optima occupy 2.6 to 4.2 percent of configurations, scattered like a maximally rugged landscape.","key_machinery":"The carrying mechanism is the configuration landscape, defined as the triple (C, N, f): the set of configurations, a neighborhood structure connecting configurations that differ in exactly one option, and the performance function as elevation. GraphFLA encodes this as a directed attributed graph whose nodes are configurations, node attributes are measured performances, and edges point from worse to better neighbors; local optima are sink nodes. This graph representation turns landscape analysis into graph mining and scales to millions of nodes, and it is the device that lets the paper measure things like local-optimum abundance, autocorrelation along random walks, per-mutation fitness effects over all backgrounds, and pairwise and higher-order interactions.","core_discovery":"The central empirical discovery is that the configuration spaces of real-world software (LLVM, Apache, SQLite) are highly rugged fitness landscapes. Across 32 workloads, local optima make up 2.56% to 4.24% of all configurations, a proportion comparable to a maximally rugged NK landscape, and even nearest-neighbor configurations have only weakly correlated fitness values. The local optima are almost uniformly scattered, most lie far from the global optimum, and the global optimum moves to a different region when the workload changes. At the option level, only a small fraction of options have significant average effects, yet a single option's effect can flip sign depending on the other options, and interactions of order higher than two contribute substantially to performance variance. These observations, the paper argues, are invisible to distribution-based performance analysis and have direct consequences: optimizers must contend with many traps, and performance models trained on random samples lose accuracy as ruggedness increases.","pith_inferences":["One testable extension is to apply the same graph-based pipeline to other black-box tuning spaces such as hyperparameters, compiler flags, or database knobs, and compare local-optimum densities across domains; the paper's findings would gain strength if the abundance and scattering patterns replicate elsewhere.","The near-uniform scattering of local optima suggests that restart-based global search may be more robust than a single long run, since there is no single basin to escape from.","A natural stress test is to apply the framework to synthetic landscapes with known ruggedness, verifying that the graph-mining local-optimum counts are exact at the sampled resolution before trusting them on measured data.","The reported link between ruggedness and model error implies that sampling strategies for performance modeling could be adaptive: spend more evaluations in regions likely to contain local optima rather than sampling uniformly."],"forward_implications":["Tuning algorithms should expect abundant local optima; strategies like random restarting, multi-objectivization, or landscape smoothing respond to a measured property of the space rather than an assumed one.","Performance-model accuracy is not uniform across systems: a landscape's local-optima density predicts how well a random-forest or DNN surrogate will fit it.","Directly reusing top configurations across workloads is risky—top-1% sets barely overlap and ranks can shift by tens of percent—so transfer learning needs workload-aware selection.","Because only a few options carry most of the influence, search can focus on those options, but per-option effects should not be read as constant since they change sign with context.","High-order interactions are common, so models that assume only pairwise interactions will systematically miss part of the performance variance."],"supporting_citations":[{"why":"Introduces the fitness-landscape metaphor that the paper adapts to software configuration spaces.","marker":"(Wright, 1932)"},{"why":"Supplies the NK landscape model whose maximally rugged local-optima proportion is the benchmark for F3.","marker":"(Kauffman, 1993)"},{"why":"Provides the autocorrelation measure used to corroborate landscape ruggedness alongside local-optima counts.","marker":"(Weinberger, 1990)"},{"why":"Prior transfer-learning study whose assumptions about workload similarity the paper tests with fitness-distribution correlations.","marker":"(Jamshidi et al., 2017a)"},{"why":"Performance-influence models that the paper extends and contrasts with per-background mutation effects.","marker":"(Siegmund et al., 2015)"},{"why":"DeepPerf DNN used as one of the performance models whose accuracy is related to ruggedness.","marker":"(Ha and Zhang, 2019a)"},{"why":"Random-forest regressor used as the other performance model in the modeling experiments.","marker":"(Ho, 1995)"},{"why":"Workload-impact analysis that this paper places in a broader spatial-landscape context.","marker":"(Mühlbauer et al., 2023)"}],"fun_headline_variants":["86M configs: software tuning landscapes are rugged","Local optima 2.6-4.2% of configs across 32 workloads","Ruggedness rivals theoretical max in real software configs","Configurable software: global optimum not fixed across workloads"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results depend on the 10-repetition mean being an accurate enough measurement of true performance that declaring one configuration better than a neighbor is meaningful; the paper applies no statistical significance threshold, so if measurement noise is larger than the fitness gaps, some of the counted local optima and their spatial distribution would be artifacts.","fun_headline_variants_meta":{"raw":{"variants":["86M configs: software tuning landscapes are rugged","Local optima 2.6-4.2% of configs across 32 workloads","Ruggedness rivals theoretical max in real software configs","Configurable software: global optimum not fixed across workloads"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000316,"raw_usage":{"total_tokens":1781,"prompt_tokens":928,"completion_tokens":853,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":779}},"tokens_in":544,"tokens_out":853,"duration_ms":7976,"temperature":1.0,"reasoning_tokens":779,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T06:00:20.092411+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-measure each configuration with many more repetitions, or apply a paired significance test between neighboring configurations, on one or two workloads and recount local optima; if the proportion of local optima drops far below the reported 2.56% to 4.24% or loses its near-uniform spatial distribution, the ruggedness and scattering claims would collapse.","supporting_citations":[],"review_version":1}