{"id":"25263896-3923-4ba9-9d30-5e2a1c07a792","arxiv_id":"2507.22877","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"SHAP feature rankings in multi-view deep learning models for multi-omics data are sensitive to architecture and random weight initialization, so important-biomolecule lists should be inspected across repeated runs.","lead":"On two viral infection datasets, SHAP feature rankings from a multi-view deep learning model changed substantially with architecture choices and random weight initializations. The study shows that biomolecule importance lists from such models should be treated as unstable until verified across repeated training runs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Instability may be a small-sample, overparameterized-feed-forward artifact; the evidence does not yet support the general caution for multi-view deep learning on multi-omics.","rationale":"The reader's main concern was that the simplified architecture may not represent all multi-view deep learning models. My concern is closely related but more specific: all experiments are in a small-sample, overparameterized, deterministic-feed-forward setting, so the observed seed-to-seed and layer-size variation could be a fitting artifact rather than a general property of attribution in multi-view deep learning. This is load-bearing because the abstract's conclusion is worded as a general caution. It is not a fatal flaw: the experiments are internally consistent, the data are honestly described, and the main quantitative findings (degrading Kendall's Tau with added noise, wide rank distributions across seeds) would support a scoped claim. A larger-cohort reproducibility check would settle whether the caution generalizes. I would keep the reader's conditional verdict but add the large-data/regularization check as a condition, in addition to code/data release and clarification of the proposed diagnostic.","tokens_in":9440,"tokens_out":9199,"duration_ms":128169,"concrete_test":"Re-run the same 10-seed SHAP ranking protocol on a substantially larger multi-omics cohort (e.g., TCGA BRCA, n≈1,000) using the same simplified feed-forward architecture and training schedule, and compute pairwise weighted Kendall's Tau between the resulting SHAP feature rankings. If the median Tau across seeds remains above 0.9, the headline instability is a small-sample overparameterization artifact and the paper's caution must be explicitly scoped to that regime.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is stated broadly: SHAP rankings are sensitive to architecture and initialization in multi-view deep learning for multi-omics. But all core experiments use the authors' simplified deterministic feed-forward topology (Methods: Model Architecture) on ICL102/ICL104, with n=60 and n=120 and thousands of input features, trained by full-batch Adam with no hyperparameter tuning. This is a heavily overparameterized regime, so different random initializations can land in different but nearly equally good basins, making SHAP rankings vary across seeds. The same applies to layer-size changes: with more capacity than samples, attribution reflects arbitrary features of the fitted function rather than a stable property of the underlying multi-view modeling strategy. The one repeated experiment with the Lee et al. model in the supplement checks only a single adjacent architecture and does not control for sample size or regularization. Thus the abstract's blanket warning overreaches; the observed instability may not persist for larger cohorts, stochastic encoders, or regularized training.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates the consistency of SHAP-based feature attributions across architectures and random initializations in a simplified multi-view feed-forward network applied to two small multi-omics datasets (ICL102, n=120 and ICL104, n=60). Experiments add noise features to one view, vary hidden layer sizes, repeat training with different random seeds, and evaluate downstream random-forest classification and clustering performance using top-ranked features. The authors report that SHAP rankings degrade with added noise, vary across architectures, and vary substantially across training runs. They conclude that SHAP rankings are not reliable in this setting and recommend caution, proposing visual inspection of rank distributions across runs as an alternative diagnostic. The raw data are available from public repositories, but processed data are only available on request and no code is provided.","tokens_in":9617,"tokens_out":5023,"duration_ms":57113,"significance":"If the conclusion is limited to the tested regime, the paper provides a useful empirical caution and a concrete demonstration of attribution instability in small-sample, overparameterized multi-view networks. The design includes sensible independent checks: downstream random-forest and clustering tasks are used to assess the quality of selected features rather than relying solely on internal attribution rankings, and weighted Kendall's tau focuses on top-rank agreement. However, the contribution is currently weakened by the narrow architectural and dataset scope, the unspecified alternative method promised in the abstract, and several internal inconsistencies. The study does not yet support a general claim about SHAP stability across multi-view deep learning for multi-omics.","major_comments":[{"comment":"The abstract and conclusion state that the results suggest caution when using attribution methods on multi-view deep learning models applied to multi-omics data, but all core experiments use a single simplified feed-forward architecture (Methods: Model Architecture) on two small datasets (ICL104 with n=60 and ICL102 with n=120) with thousands of input features, full-batch Adam, and no hyperparameter tuning. The authors explicitly describe the architecture as a simplification of Lee et al. and deliberately avoid stochastic encoders and variational losses. The one supplementary repetition with the Lee et al. model does not control for sample size, regularization, or stochasticity. The observed instability may be specific to heavily overparameterized deterministic networks on small samples rather than a general property of multi-view deep learning on multi-omics. To support the stated broad conclusion, the authors should either restrict the claim to the tested architecture regime or add experiments with regularized and/or stochastic architectures and, if possible, a larger cohort.","section":"Abstract / Methods: Model Architecture / Data"},{"comment":"The abstract promises 'an alternative, simple method to assess the robustness of identification of important biomolecules,' but the only proposal in the manuscript is visual inspection of rank distributions across training runs (Discussion, Figure 3). No algorithm, threshold, or diagnostic statistic is defined, the method is not compared with existing stability measures (e.g., rank stability, Jaccard index, rank-biased overlap), and it is not validated on data with known ground-truth importance. Since the claimed contribution includes this method, it needs to be specified precisely and evaluated quantitatively.","section":"Abstract / Discussion / Conclusion"},{"comment":"Figure 3 and its caption state that ranks are computed 'out of 351 possible ranks,' but the dataset dimensions reported in the Data section (ICL102: 80 metabolites, 496 lipids, 3722 proteins; ICL104: 138, 491, 4896) and the noise-augmentation experiments (up to 4000 added features) do not produce 351 features in any described configuration. If ranks are computed on a subset (e.g., only annotated metabolites) or after filtering, that must be stated. As written, the rank denominator is unexplained and undermines the quantitative interpretation of the figure.","section":"Results: Variation in Features, Figure 3"},{"comment":"There is a direct contradiction between the Results section, which states 'We do however see a decrease in V-measure score as we reduce the proportion of top features selected,' and the Discussion, which states 'we saw no significant drop as measured by V-measure.' Because the paper uses downstream clustering performance as evidence that selected features retain discriminative quality, this inconsistency affects the interpretation of a central result and must be resolved.","section":"Results: Binary and Multi-class Viral Dataset / Discussion"}],"minor_comments":[{"comment":"The manuscript contains numerous typographical errors, including 'ePect' for 'effect', 'diPerent' for 'different', 'DiPicult' for 'difficult', and others. A careful proofreading pass is needed.","section":"Throughout"},{"comment":"Processed data are available only on request and no code is provided. For an empirical study whose claims rest on specific preprocessing, feature sets, and training procedures, releasing code and processed data would substantially improve reproducibility.","section":"Data Availability"},{"comment":"The text refers to 'Lee et. al.' without specifying whether this is reference [7] (Lee and van der Schaar, variational information bottleneck) or reference [15] (Lee et al., gene-centric convolutional encoders). The description of the architecture suggests [7], but the ambiguity should be removed.","section":"Methods: Model Architecture"},{"comment":"The paper reports AUC for a multiclass dataset but does not specify whether this is one-vs-rest, macro-average, or another aggregation. The definition should be given in the Methods section.","section":"Results: Performance of Features Subsets"},{"comment":"The weighted Kendall's tau values in Figure 2 are reported without confidence intervals or statistical tests. Given that only 10 training runs are used per condition, some measure of uncertainty would help assess whether the observed drop-offs are significant.","section":"Methods: Effect of Feature Compression"},{"comment":"The paper does not discuss the approximation error of DeepExplainer (the SHAP variant used). Since DeepExplainer is an approximation to Shapley values, its own variability could contribute to the observed instability; a brief comment or sensitivity analysis would strengthen the interpretation.","section":"Discussion"}],"recommendation":"major_revision","confidential_remarks":"The paper is an honest empirical study with a useful cautionary finding, but its central claim is broader than the evidence supports. The main fixes needed are: (1) scope the conclusion to the tested architecture and sample-size regime, or add experiments with more representative architectures and regularization; (2) define and validate the proposed alternative method; (3) resolve the Figure 3 rank denominator and the V-measure contradiction. If the authors make these changes, the paper could become a solid contribution. I would also encourage the editor to require code/data release, given the importance of reproducibility in this type of empirical work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know about this paper. First, it is an honest, clearly-written empirical study of SHAP stability in a simplified multi-view feed-forward network on two small multi-omics datasets (n=60 and n=120). The core finding—SHAP rankings vary a lot across random initializations and across layer-size changes—is credible, and the noise-injection experiment is a genuinely useful addition. Second, the paper's broad conclusion overreaches: the evidence does not yet support a general caution for multi-view deep learning on multi-omics, because the regime tested is narrow and possibly pathological.\n\nWhat is new: the specific combination of multi-view integration, SHAP, and a controlled noise-injection protocol that inflates one view's feature count while holding the other fixed. The rank-variation plots across retraining runs are a practical diagnostic. They also do a sensible thing by checking whether the selected features still work in a random-forest and clustering pipeline; that is more than most attribution papers bother with. The writing is straightforward and the limitations are acknowledged.\n\nThe soft spots are real but not fatal. The stress-test concern lands: with thousands of features and only 60–120 samples, full-batch Adam on unregularized MLPs is the textbook regime where different seeds land in different but equally good basins, so attribution instability may be an overparameterization artifact rather than a property of multi-view models per se. The one repeated experiment with the Lee et al. model is adjacent, not a generalizability check. They do not cite the existing literature on attribution instability across retraining, which weakens the claim of novelty. No code or processed data are released, so the experiments cannot be reproduced. The 'alternative method' is really a recommendation—run several times and inspect rank variation—rather than a new method, and it is not compared against simpler baselines like averaging SHAP values across runs.\n\nMy bottom line: this is a solid cautionary case study, not a broad negative result. It deserves a serious referee, but the referee should push for code and data, for a more precise scoping of the claim, and for an experiment that controls for overparameterization (e.g., stronger regularization or a larger cohort). I would not cite it in my own work until a revision addresses those points.","headline":"A careful empirical caution about SHAP instability in small-sample multi-view nets, but the broad warning outruns the evidence.","tokens_in":10134,"tokens_out":3427,"would_cite":false,"duration_ms":42423,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that SHAP-based feature rankings in multi-view deep learning models for multi-omics data are unstable across architecture choices and random weight initializations, so a single attribution run should not be trusted as a…","keywords":["SHAP","feature attribution","multi-omics","multi-view deep learning","rank stability","weighted Kendall's tau","biomarker discovery","model interpretability"],"falsifier":"Train a multi-view deep network on synthetic multi-omics data with a planted, known sparse set of driver biomolecules, rerun SHAP over many random initializations and across several architecture families, and check whether the top-ranked features consistently recover the planted drivers. If the planted drivers stay at the top across all runs and architectures, the claimed sensitivity would be specific to the paper's two datasets rather than a general property of SHAP on multi-view networks.","tokens_in":9265,"feed_emoji":"🧬","tokens_out":8701,"duration_ms":104310,"temperature":0.7,"pith_summary":"The paper sets out to test whether SHAP, a popular method for explaining which input features drive a deep model's predictions, gives stable answers when the model is a multi-view network trained on multi-omics data. It finds that the ranked list of important biomolecules changes with the network's layer sizes, with the way views are combined, and even across training runs that differ only in random weight initialization and dropout. The conclusion is a caution: a single SHAP-ranked feature list from such a model is not a reliable basis for naming driver biomolecules. The paper also offers a simple remedy: train several times and inspect the distribution of ranks rather than trusting one ranking, and it shows that the instability can be hidden because downstream classifiers still do well on the selected features.","feed_headline":"Feature rankings from SHAP shift with architecture and seeds","feed_subtitle":"On molecular data, one run's top list can flip for another random seed; repeat runs show the spread.","key_machinery":"The argument is carried by two instruments: the SHAP attribution score, averaged in absolute value over samples, which converts a trained network into a ranked list of biomolecules; and the weighted Kendall's tau, a rank-correlation measure that gives more weight to matches at the top, used to quantify how much rankings degrade when noise features are added or architecture changes. A third instrument is the per-feature distribution of ranks over repeated training runs, which reveals that supposedly top-ranked biomolecules can fall to middle ranks under a different random initialization. These diagnostics jointly show that attribution is not an intrinsic property of the biomolecule but an artifact of the particular fitted network.","core_discovery":"SHAP scores computed on feed-forward multi-view networks do not yield consistent feature rankings. Adding irrelevant noise features to one omics view degrades the weighted Kendall's tau between original and augmented rankings, and the pattern depends on the hidden layer sizes of the other view. Even for the highest-ranked biomolecules, ranks fluctuate widely across training runs started with different random weights; a feature that occasionally spikes to high importance can mostly sit near the bottom. The same instability appears in a more complex multi-view baseline architecture used as a validation check. Despite this, random-forest classifiers fit on the top 10% of SHAP-ranked features retain much of the full-feature AUC, so predictive performance alone does not reveal attribution instability.","pith_inferences":["If the observed instability scales with model complexity, then generative multi-view architectures such as variational autoencoders likely show even larger SHAP variance than this feed-forward template, since their training adds further random components such as variational sampling.","A natural testable extension is an ensemble consensus rule: average SHAP ranks across many seeds and architectures and test whether the consensus top-k is more reproducible and biologically plausible than any single run; the paper stops short of proposing this as a remedy.","The noise-inflation experiment implies a practical diagnostic for applied studies: permute or knock out one view's features and measure the rank shift of known positive controls before trusting cross-view importance comparisons.","Because both datasets are small and the simplified architecture avoids variational losses, the magnitude of the observed instability may be a lower bound for real-world applications with more complex models and larger, noisier omics panels."],"forward_implications":["A single SHAP run on one trained multi-view network is not enough to nominate biomarkers; the same data and task can produce a different top list if the weights are re-initialized.","Reports of SHAP-identified genes or metabolites should include the spread of ranks across multiple training runs, since a feature can be top-ranked in one run and below median in another.","Increasing the number of uninformative features in one omics view can push true drivers of that view down in the global ranking, so cross-view importance comparisons are sensitive to relative view size.","Downstream performance checks, such as fitting a random forest or clustering on the top-ranked features, can look fine even when the underlying rankings are unstable, so good performance is not evidence that the attribution is consistent.","Matching or dynamically adjusting layer sizes does not reliably stabilize SHAP rankings; the effect reverses depending on the base size of the other view's network, so there is no simple capacity-based fix."],"supporting_citations":[{"why":"Defines SHAP and the implementation whose feature rankings are being tested.","marker":"[11]"},{"why":"Provides the gradient-based approximation method that underlies the SHAP scores used in the experiments.","marker":"[10]"},{"why":"Supplies the multi-view architecture template that the paper simplifies, and the comparison model used to check generalizability of the instability pattern.","marker":"[7]"},{"why":"Supplies the weighted Kendall's tau rank-correlation measure used to quantify rank instability under noise and architecture changes.","marker":"[20]"},{"why":"Provides the two host-response multi-omics datasets on which all experiments are run.","marker":"[19]"},{"why":"Supplies the V-measure used to evaluate clustering quality of SHAP-selected feature subsets.","marker":"[23]"}],"fun_headline_variants":["SHAP feature rankings unstable across architectures and seeds","Attribution rankings flip with random seeds in multi-omics models","On multi-omics, SHAP top lists vary wildly by seed","Feature attribution consistency lacking in deep multi-omics models","SHAP scores unreliable for ranking biomolecules in multi-omics"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's warning about SHAP transfer depends on its simplified feed-forward multi-view network being representative of the multi-view deep learning models that researchers actually apply to multi-omics data; if those models behave differently, the measured instability may not generalize.","fun_headline_variants_meta":{"raw":{"variants":["SHAP feature rankings unstable across architectures and seeds","Attribution rankings flip with random seeds in multi-omics models","On multi-omics, SHAP top lists vary wildly by seed","Feature attribution consistency lacking in deep multi-omics models","SHAP scores unreliable for ranking biomolecules in multi-omics"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000446,"raw_usage":{"total_tokens":2233,"prompt_tokens":902,"completion_tokens":1331,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":1249}},"tokens_in":518,"tokens_out":1331,"duration_ms":11100,"temperature":1.0,"reasoning_tokens":1249,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:10:41.326071+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a multi-view deep network on synthetic multi-omics data with a planted, known sparse set of driver biomolecules, rerun SHAP over many random initializations and across several architecture families, and check whether the top-ranked features consistently recover the planted drivers. If the planted drivers stay at the top across all runs and architectures, the claimed sensitivity would be specific to the paper's two datasets rather than a general property of SHAP on multi-view networks.","supporting_citations":[{"cited_title":"Learning Important Features Through Propagating Activation DiPerences,","cited_arxiv_id":null,"evidence_quote":"Provides the gradient-based approximation method that underlies the SHAP scores used in the experiments."},{"cited_title":"A Variational Information Bottleneck Approach to Multi-Omics Data Integration,","cited_arxiv_id":null,"evidence_quote":"Supplies the multi-view architecture template that the paper simplifies, and the comparison model used to check generalizability of the instability pattern."},{"cited_title":"A Weighted Correlation Index for Rankings with Ties","cited_arxiv_id":"1404.3325","evidence_quote":"Supplies the weighted Kendall's tau rank-correlation measure used to quantify rank instability under noise and architecture changes."},{"cited_title":"A compendium of multi-omics data illuminating host responses to lethal human virus infections,","cited_arxiv_id":null,"evidence_quote":"Provides the two host-response multi-omics datasets on which all experiments are run."},{"cited_title":"V-Measure: A Conditional Entropy-Based External Cluster Evaluation Measure,","cited_arxiv_id":null,"evidence_quote":"Supplies the V-measure used to evaluate clustering quality of SHAP-selected feature subsets."}],"review_version":1}