{"id":"ffd43ccc-9c52-4a9a-997d-afe12240be0d","arxiv_id":"2607.15884","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A framework using Shapley Effects and Pareto fronts ranks hyperparameter influence per objective from a coarse grid-search lookup table, without proposing a new optimizer.","lead":"This paper applies a game-theoretic sensitivity analysis (Shapley Effects) plus Pareto front analysis to figure out which neural-network hyperparameters actually matter for multiple objectives. It demonstrates the approach on three network types and claims the resulting rankings can shrink the search space before expensive tuning.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-seed lookup table leaves Shapley rankings and Pareto fronts vulnerable to training noise; repeated-seed check needed to validate central claim.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: the entire lookup table is constructed from one training run per configuration, so training stochasticity is confounded with hyperparameter sensitivity. My analysis agrees that this is the central vulnerability. The paper itself admits the single-evaluation issue in Section II-C, but the proposed mitigation via Pareto optimality is logically flawed: Pareto dominance on single noisy evaluations selects exactly the configurations that benefited from lucky noise. Moreover, the Shapley-effect estimates are not accompanied by any uncertainty quantification, so there is no way to tell whether the reported rankings are signal or noise. This is not an internal mathematical inconsistency in the Sobol–Shapley machinery, but an empirical validity gap in the demonstration. Since the authors are transparent about the single-evaluation setup and position the work as a proof of concept, a conditional verdict is appropriate: the framework may be useful, but the central interpretability claim requires repeated-seed evidence. My concrete test would settle whether the noise level is actually problematic. I therefore do not change the reader's verdict—CONDITIONAL remains appropriate.","tokens_in":10135,"tokens_out":3494,"duration_ms":39653,"concrete_test":"Re-run a random 20% subset of configurations from each lookup table (e.g., ~60 PINN, ~80 CNN, ~60 DNN) with R=5 independent seeds to estimate per-configuration objective noise σ². Compute Shapley effects (i) from a single seed, (ii) from seed-averaged objectives, and (iii) from bootstrap resampling of the five seeds. If the spread of Shapley effects across seeds is comparable to the spread across hyperparameters, or if Kendall's τ between single-seed and seed-averaged rankings is below ~0.8, the reported sensitivity rankings are noise-dominated. Additionally, check how many points on each reported Pareto front remain non-dominated when seed-averaged objectives are used; if a large fraction disappear, the front is largely a noise artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the coarse-grained lookup table D, built from one training run per configuration, yields reliable Shapley-effect rankings and Pareto fronts that reveal which hyperparameters matter. Section II-C explicitly states that 'each configuration is evaluated only once' and asserts that Pareto optimality mitigates 'lucky' runs. This assertion is not supported: Pareto non-dominance is computed on the noisy objective vectors themselves (Eqs. 11–12), so single-run noise cannot be filtered out; on the contrary, configurations with favorable noise realizations are precisely those that appear on the frontier. For sensitivity, the MC conditional means bµ(S,g_S) in Eq. 3c average over the |I(S,g_S)| configurations matching g_S. If per-training noise has variance σ², each bµ has noise variance roughly σ²/|I|, which grows for larger coalitions. This noise enters every Shapley marginal contribution in Eq. 6, and no error bars or repeated-seed estimates are provided. The paper acknowledges noise but never quantifies it, and the reported Shapley rankings and Pareto fronts could therefore be dominated by seed variation rather than genuine hyperparameter influence. Since the framework's practical value is exactly the interpretation of which hyperparameters matter, this unquantified noise is a load-bearing gap, not a minor caveat.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a game-theoretic framework for interpretable hyperparameter sensitivity analysis. Hyperparameters are treated as players in a cooperative game; a coarse-grained grid search builds a lookup table of configurations and multi-objective outputs, Shapley effects are used for global sensitivity analysis to rank hyperparameter influence per objective, and Pareto-front analysis identifies non-dominated configurations for early-stage model evaluation. The framework is demonstrated on three neural-network case studies: a PINN for a dual spring–mass–damper system, a CNN on CIFAR-100, and a DNN on the Adult Income dataset. The paper claims the resulting sensitivity rankings and Pareto fronts provide transferable prior knowledge and can guide subsequent optimization and reduce search space.","tokens_in":10375,"tokens_out":4373,"duration_ms":46299,"significance":"If the framework's claims were fully supported, it would provide a useful, model-agnostic interpretability tool for hyperparameter importance, building on established Shapley-effect methodology for global sensitivity analysis. The proposed pipeline is simple, computationally modest after the initial grid search, and the authors provide a public codebase. The case studies cover diverse architectures and objectives. However, the central empirical claim—that the rankings and Pareto fronts reliably reflect genuine hyperparameter influence—is currently not supported because the entire analysis rests on a single evaluation per configuration with no quantification of training stochasticity. This is a load-bearing gap that must be addressed before the practical recommendations can be accepted.","major_comments":[{"comment":"The manuscript states that Pareto optimality is used to mitigate the effect of 'lucky' runs from single evaluations, but Pareto non-dominance is computed on the noisy objective vectors themselves. A configuration with a favorable noise realization is more likely to appear on the frontier, so the Pareto front does not filter noise; it is biased toward lucky runs. No repeated-seed validation is provided to show that the reported fronts are stable. This directly undermines the framework's practical claims of 'early-stage model evaluation' and the conclusions drawn from Fig. 4 (e.g., that the CNN has reached a saturated performance regime).","section":"§II-C, Eqs. (11)–(12)"},{"comment":"All Shapley effects are computed from a lookup table where each configuration is evaluated once. The conditional-mean estimator bµ(S,g_S) in Eq. (3c) inherits a noise variance of roughly σ²/|I(S,g_S)|, which is larger for larger coalitions because fewer matching configurations exist. This noise propagates through every marginal contribution in Eq. (6) and into the reported Shapley rankings in Fig. 3. The paper acknowledges training stochasticity but never quantifies σ² or provides error bars, repeated-seed estimates, or an external check (e.g., fixing non-influential hyperparameters and showing performance is preserved). Without this, the influential/non-influential classification may be an artifact of seed variation.","section":"§II-B, Eqs. (3c), (5b)–(5c), (6)"},{"comment":"The actionable output of the framework is the classification of hyperparameters as influential or non-influential via the threshold τ in Eq. (8). However, τ is chosen arbitrarily, no sensitivity analysis of the classification to τ is reported, and the classification inherits all the single-run noise described above. The conclusions that the DNN is 'comparatively less sensitive' and that fine-tuning would provide 'only limited improvement' are strong practical statements that rest entirely on unvalidated, noisy estimates. A minimal validation would be to rerun a subset of configurations with multiple seeds and show that the rankings and the conclusions in Fig. 3 are stable.","section":"§III-B (global sensitivity analysis) and §IV"}],"minor_comments":[{"comment":"The unweighted average bµ_S over g_S is used in Eq. (5c). For the full-factorial designs considered here, an unweighted average happens to equal the overall mean when conditional means are exact, but this is not stated. Clarify the relationship to avoid confusion about weighted vs unweighted estimates.","section":"§II-B, Eq. (5b)"},{"comment":"Please specify whether the random seed is fixed across all configurations or varies. If seeds vary, the 'lucky run' concern is as stated; if seeds are fixed, the noise source is different and should be described. This detail is essential for interpreting the single-run evaluations.","section":"§II-C"},{"comment":"Fig. 3 reports Shapley effects without any uncertainty indicators. Even a simple bootstrap over the lookup table entries would help assess stability. At minimum, the threshold τ used in each subplot should be stated in the caption or text.","section":"§III-B"},{"comment":"The paper repeatedly refers to 'Sobol-Shapley' sensitivity; the method is a Shapley-effect decomposition of Sobol indices. Using the standard name 'Shapley effects' consistently would align with the cited literature.","section":"General"},{"comment":"The sample lookup tables are labeled 'Run 1', 'Run 2', etc., but the ordering across tables is inconsistent (e.g., PINN Run 1 is Tanh while CNN Run 1 is ReLU). This is fine, but a note on how runs are ordered would improve readability.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal and addresses a timely problem. The methodological skeleton (Shapley effects for hyperparameter sensitivity) is sound, but the empirical demonstration does not yet support the central claim. The single-run lookup table and absence of noise quantification are not merely caveats; they affect every reported sensitivity value and Pareto-front conclusion. I would encourage the authors to add a repeated-seed or external-validation experiment, even on a subset of configurations, and to report uncertainty in the Shapley estimates. If such an experiment confirms stability, a revised manuscript could make a solid contribution; as it stands, the evidence is insufficient for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this paper does something sensible and mostly standard. It treats hyperparameters as players in a cooperative game, computes Sobol–Shapley effects from a coarse-grained grid of configurations, and uses Pareto fronts to give an early read on a model class. The math in Section II is correct — Eq. (6)–(7) are the standard Shapley Effects, and the estimation procedure is a straightforward Monte Carlo over conditional subsets. The writing is honest: the authors flatly state that coarse-grained search is a proof of concept, that the threshold τ is user-defined, and that HPO baselines are out of scope. They also ship code and data. That is real credit where it is earned.\n\nThe soft spot is exactly where the stress-test lands. Section II-C says each configuration is evaluated once and claims Pareto optimality mitigates “lucky” runs. That claim is not supported. Pareto non-dominance is computed on the noisy objective vectors themselves, so favorable noise realizations are precisely what lands a configuration on the front. The Shapley effects are built from conditional means over subsets of a single-run lookup table; the noise enters every marginal contribution, and larger coalitions have smaller subsample sizes, so the variance of those means grows just when you need them most. No error bars, no repeated seeds, no validation. The abstract promises that the framework reveals which hyperparameters matter, but the empirical rankings could be seed variation in disguise. The reader’s “conditional” verdict is the right one — the paper is not worthless, but the load-bearing evidence is thin.\n\nThere is also a mild circularity: the same lookup table that produces the sensitivity values is treated as the prior knowledge to guide later search. That would be fine if the paper independently showed that fixing the “non-influential” hyperparameters preserves performance. It doesn’t. A few repeated-seed runs and one such external check would substantially fix this.\n\nCompared to existing work, the components are known — HyperSHAP and multi-objective AutoML importance already do Shapley-based hyperparameter analysis. The novelty is the specific packaging with Pareto fronts and the three case studies, which are demonstrations, not breakthroughs. I would not overstate the contribution, but I also would not dismiss it.\n\nVerdict: send to peer review, but with reviewers who will ask for the repeated-seed experiment. If the authors add error bars and a no-harm check on fixing non-influential hyperparameters, this is a solid applied paper. As-is, it is a reasonable workshop-level contribution whose central idea is sound but whose headline claims outrun the evidence.","headline":"A competent application of Shapley Effects to hyperparameter sensitivity, with the standard math done correctly, but the single-run lookup table leaves the rankings and Pareto fronts potentially noise-driven.","tokens_in":10922,"tokens_out":1510,"would_cite":false,"duration_ms":19570,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62-08","91A12"],"pacs":[],"model":"deepseek-v4-flash","headline":"A coarse-grained lookup table, read through Shapley-effect sensitivity and Pareto-front analysis, identifies which hyperparameters drive each objective and whether fine-tuning a model class is worthwhile at all.","keywords":["hyperparameter sensitivity analysis","Shapley effects","global sensitivity analysis","Pareto front analysis","multi-objective optimization","cooperative game theory","lookup table surrogate","early-stage model evaluation"],"falsifier":"Run the coarse-grained grid multiple times with different random seeds — say five to ten repeats per configuration — and recompute the Shapley effects from each seed's lookup table. If the influential/non-influential classification flips across seeds, or if the between-seed variance of a hyperparameter's Shapley effect is comparable to its mean value, then the single-run lookup table is too noisy to support the sensitivity claims. A cheaper complementary check: fix a hyperparameter the framework labels non-influential, vary the rest, and see whether the objective distributions change materiall","tokens_in":9960,"feed_emoji":"🎛️","tokens_out":15002,"duration_ms":130857,"temperature":0.7,"pith_summary":"This paper argues that costly, fine-grained hyperparameter search is not needed to find out which hyperparameters matter. A single coarse-grained sweep — one training run per configuration, stored in a lookup table — combined with Shapley-effect sensitivity analysis and Pareto-front identification, is enough to rank hyperparameters by their influence on each objective and to reveal the attainable trade-offs among objectives. If the rankings are right, practitioners can fix or deprioritize hyperparameters that show negligible influence, shrinking the search space before any fine-tuning, and can judge up front whether a model class can reach acceptable performance at all. The authors demonstrate the approach on three architectures — a physics-informed neural network, a CNN on CIFAR-100, and a DNN on Adult income — and show that sensitivity patterns differ sharply across tasks, so hyperparameter priorities cannot be assumed to transfer.","feed_headline":"A cheap grid search reveals which hyperparameters matter","feed_subtitle":"Shapley-effect sensitivity plus Pareto fronts show when fine-tuning pays off, and when it is wasted effort.","key_machinery":"The load-bearing object is the Sobol–Shapley effect: the Shapley value of a hyperparameter when coalition values come from a first-order Sobol variance decomposition, v(S) = Var(E[Y | G_S]) / Var(Y). Conceptually, it splits each objective's output variance among hyperparameters, averaging marginal contributions over all coalitions so the scores sum to exactly 1 per objective. It is variance-based and model-agnostic — no gradients, and interactions and nonlinearities are captured — yet Shapley-based, so each player gets a unique attribution. Conditional variances are estimated by Monte Carlo sampling (N = 10^4) over conditional subsets of the lookup table, avoiding exhaustive retraining. The","core_discovery":"The paper's central claim: treating hyperparameters as players in a cooperative game turns sensitivity analysis into a problem a cheap grid can solve. A lookup table from one evaluation per coarse configuration is read through Sobol–Shapley effects, whose influence scores sum to unity per objective. A user-set threshold classifies each hyperparameter as influential or non-influential, and Pareto dominance identifies non-dominated configurations. Across three case studies the tools deliver distinct verdicts: the PINN spans roughly 0–99% accuracy with clear tuning headroom; the CIFAR-100 CNN saturates near 50% accuracy; the Adult-income DNN sits in a narrow 83–86% band. These conclusions, draw","pith_inferences":["My inference: the framework is best read as a pre-screening stage for HPO. If the single-pass grid is reliable, the efficient strategy it implies is to spend the tuning budget only in the influential subspace — the paper suggests this direction but does not measure the computational savings or quality loss relative to full-space tuning.","My inference: the sharpest test the paper does not run is a fix-and-refine experiment — freeze the non-influential hyperparameters, fine-tune only the influential ones, and compare the resulting objectives with full-space optimization. The gap between the two would quantify how much the sensitivity ranking is worth.","My inference: because every configuration is evaluated once, the threshold τ is chosen without any estimate of run-to-run noise. A natural extension is repeated-seed evaluation that separates training stochasticity from hyperparameter-driven variance; that would make the influential/non-influential cut statistically principled rather than user-chosen.","My inference: the transferability claim — that the sensitivity spectra constitute reusable prior knowledge for similar models and tasks — is asserted but untested. A direct check would compute Shapley spectra on one dataset or architecture and verify that the influential sets carry over to a neighboring task."],"forward_implications":["Hyperparameters classified as non-influential can be fixed to a single representative value or restricted to a small candidate set, shrinking the search space before any fine-grained optimization begins.","The Pareto front doubles as an early-stage model evaluation: it reveals whether a model class can reach acceptable performance for a task (as with the PINN) or has already saturated (as with the CIFAR-100 CNN), before expensive tuning is spent.","Hyperparameter influence is application- and architecture-specific: the learning rate can dominate one model's accuracy and matter little in another, so sensitivity priorities must be derived per task rather than assumed from prior experience.","Because Shapley effects sum to unity per objective, the framework yields a normalized comparison of influence across objectives, making trade-offs between accuracy and training cost explicit.","The framework is not tied to coarse grid search: any input–output mapping of configurations to objectives can be fed through the same sensitivity and Pareto analysis, making it reusable as transferable prior knowledge for later hyperparameter optimization."],"fun_headline_variants":["Game theory reveals which hyperparameters matter most","A cheap grid search plus Shapley effects ranks hyperparameters","Which hyperparameters matter? Ask the Shapley effects","Pareto fronts and Shapley effects decode hyperparameter impact","Hyperparameter sensitivity, interpreted via a cooperative game"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that a lookup table built from exactly one training run per coarse-grained configuration reflects true hyperparameter influence well enough that the variance decompositions and Pareto rankings are signal rather than training noise — the paper acknowledges this noise in its methods section and its discussion of limitations, but does not quantify it or use repeated runs — and that the chosen search space and discretization are representative of the h","fun_headline_variants_meta":{"raw":{"variants":["Game theory reveals which hyperparameters matter most","A cheap grid search plus Shapley effects ranks hyperparameters","Which hyperparameters matter? Ask the Shapley effects","Pareto fronts and Shapley effects decode hyperparameter impact","Hyperparameter sensitivity, interpreted via a cooperative game"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000197,"raw_usage":{"total_tokens":1153,"prompt_tokens":648,"completion_tokens":505,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":392,"completion_tokens_details":{"reasoning_tokens":426}},"tokens_in":392,"tokens_out":505,"duration_ms":5563,"temperature":1.0,"reasoning_tokens":426,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T22:02:44.702119+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the coarse-grained grid multiple times with different random seeds — say five to ten repeats per configuration — and recompute the Shapley effects from each seed's lookup table. If the influential/non-influential classification flips across seeds, or if the between-seed variance of a hyperparameter's Shapley effect is comparable to its mean value, then the single-run lookup table is too noisy to support the sensitivity claims. A cheaper complementary check: fix a hyperparameter the framework labels non-influential, vary the rest, and see whether the objective distributions change materiall","supporting_citations":[],"review_version":1}