{"id":"3bb77f9b-ba1b-4f7d-9058-f05e7eeeaf5a","arxiv_id":"2508.08724","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Hierarchical-CPI computes variable-importance groups in highly correlated medical data with explicit family-wise error rate control.","lead":"The paper introduces Hierarchical-CPI, a method for ranking groups of variables by their predictive importance while controlling the risk of false discoveries. It is aimed at medical imaging, where highly correlated data often defeat standard interpretability tools.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"FWER control claim presumes a fixed, correctly specified variable tree; the abstract does not show how tree selection or misspecification is handled.","rationale":"The reader identified the meaningfulness of the hierarchical tree as the weakest assumption. I partially agree, but sharpen the concern: the tree's role is not only about transferring importance to variables of interest; it is central to the mathematical FWER proof. If the tree is data-dependent or misspecified, the hypothesis set and the testing order change, which can break family-wise error control even when each conditional test is valid. The abstract does not specify whether the tree is fixed a priori or learned from the data, nor does it provide a proof sketch; the full text is unavailable, so the guarantee cannot be checked. The proposed null simulation is a direct falsification test: if empirical FWER is controlled under misspecified and data-adaptive trees, the concern does not land; if not, the central claim must be qualified. Since the paper remains unverifiable from the abstract alone, the appropriate verdict is unchanged from the reader's UNVERDICTED, rather than accepting the claimed FWER guarantee at face value.","tokens_in":598,"tokens_out":2725,"duration_ms":32357,"concrete_test":"Simulate data under the global null with p independent predictors and an outcome independent of all predictors. Apply Hierarchical-CPI at nominal FWER alpha = 0.05 under three settings: (i) a fixed tree matching the true independent groups; (ii) a fixed but deliberately misspecified tree, e.g., with randomly permuted variable groupings; (iii) a tree estimated or adapted from the same data. Across at least 1000 replicates, record the proportion of runs with at least one rejected group. If setting (ii) or (iii) exceeds the nominal level by more than simulation error, then the FWER guarantee as stated requires additional conditions on the tree; if all settings stay at or below alpha, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Hierarchical-CPI gives explicit FWER control while exploring subgroups along a hierarchical tree. For any FWER guarantee, the set of hypotheses tested and the order of exploration must be fixed independently of the data, or the selective inference induced by data-dependent choices must be explicitly corrected. The abstract does not state whether the tree is prespecified, estimated, or adapted during the search, and no proof sketch is available in this abstract-only record. If the tree is built from or modified using the same data, nominal FWER control can fail even if the per-node tests are valid. Additionally, the tree-based importance allocation is intended to address vanishing conditional importance under high correlation, but allocation can change the null distribution of the group statistics; without a stated correction, FWER control under arbitrary correlation among variables is not established. Because the paper's headline contribution is exactly this statistical guarantee, the missing specification of the tree's status is the load-bearing concern.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Hierarchical-CPI, a model-agnostic variable importance method for prediction models in medical settings. The method is described as framing variable importance as the discovery of jointly predictive variable groups along a hierarchical tree, with claims of computational tractability, explicit family-wise error rate (FWER) control, and a tree-based importance allocation mechanism that addresses vanishing conditional importance under high correlation. The authors report benchmarks against state-of-the-art methods and applications to two neuroimaging datasets, ADNI and TDBRAIN, with biologically plausible findings. The available record consists only of the abstract, so the assessment is based solely on the claims made there.","tokens_in":772,"tokens_out":2429,"duration_ms":28701,"significance":"If the claims are established in the full manuscript, the paper would offer a practically valuable contribution: a model-agnostic variable importance method with explicit FWER control and a workaround for a known weakness of conditional importance measures under high correlation. These properties would be particularly relevant for interpretability in medical imaging, where correlated features are common and rigorous error control is expected. The paper clearly identifies a real problem and provides a concrete solution outline. However, the significance is conditional on the availability of proofs, algorithm details, and benchmark methodology, none of which are inspectable from the abstract alone.","major_comments":[{"comment":"The central claim of explicit family-wise error rate control is not accompanied by any statement about whether the hierarchical tree is prespecified, estimated from data, or adapted during the search; a FWER guarantee requires the set of hypotheses and the order of exploration to be fixed independently of the data, or a valid selective-inference correction. If the tree is built from or modified using the same data, nominal FWER control can fail even when the per-node tests are individually valid, so this missing specification is load-bearing for the headline contribution.","section":"Abstract, FWER control claim"},{"comment":"The proposed allocation mechanism is introduced as a remedy for vanishing conditional importance under high correlation, but the abstract does not state how the allocation affects the null distribution of the group-level test statistics or whether the per-node tests and FWER bound are adjusted for arbitrary correlation among variables. Without such an adjustment, the explicit FWER control claim is not established in the high-correlation setting that the method is specifically designed to address.","section":"Abstract, tree-based importance allocation"},{"comment":"The benchmark claims against state-of-the-art methods are not verifiable from the abstract because the number of methods, their tuning, the multiple-testing procedures, and the metric used to compare selected variable sets are unspecified; these details are necessary to assess whether the reported effectiveness on ADNI and TDBRAIN is due to the method's statistical properties or to favorable implementation choices. This concern is secondary to the FWER issue but still necessary for evaluating the practical claims.","section":"Abstract, benchmark evaluation"}],"minor_comments":[{"comment":"The acronym CPI is not expanded in the abstract; it should be defined as Conditional Predictive Impact (or the intended phrase) at first occurrence.","section":"Abstract, terminology"},{"comment":"The abstract does not state whether the claimed FWER control is finite-sample or asymptotic, nor the level at which control is asserted; specifying this would help readers interpret the guarantee.","section":"Abstract, FWER specification"},{"comment":"The statement that the identified variables are 'biologically plausible' lacks a description of the comparison basis or criterion used to determine plausibility; a brief operationalization would make this claim more meaningful.","section":"Abstract, biological plausibility"}],"recommendation":"uncertain","confidential_remarks":"The review is based only on the abstract, which makes a formal verdict impossible under normal standards. Before a decision can be reached, the full manuscript must be examined, with particular attention to whether the hierarchical tree is fixed in advance or data-dependent, and to how the allocation mechanism preserves the FWER guarantee under high correlation. I recommend requesting the full manuscript and resubmitting for a complete review."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this is an abstract-only record, so I can't certify the method, but the idea is plausible and the headline claim—explicit FWER control for hierarchical conditional predictive importance—is exactly the kind of thing a referee should check. I'd want the full paper before citing it, but I wouldn't desk reject it.\n\nWhat's actually new: Hierarchical-CPI combines three pieces that haven't been put together before, as far as the abstract shows: hierarchical tree search over variable groups, conditional predictive importance, and FWER control. The tree-based allocation to counter vanishing conditional importance under high correlation is a sensible response to a known failure mode of CPI. If the full paper has the proof and the algorithm, this is a useful incremental contribution to interpretable medical ML, not a paradigm shift.\n\nSoft spots, in proportion: the biggest one is that FWER control is a load-bearing claim, and the abstract doesn't show the machinery. For the guarantee to hold, the tree and the set of tested hypotheses need to be fixed independently of the data, or the search needs a selective-inference correction. The abstract doesn't say whether the tree is prespecified, estimated, or adapted during the search. It also doesn't say how the allocation mechanism changes the null distribution of the group statistics. These aren't necessarily flaws—I can't tell from an abstract—but they are exactly the places where a proof can silently break. The reader's worry about tree misspecification is related but secondary; even a meaningful tree doesn't buy you FWER control if the tree is chosen using the outcome.\n\nWhat I can't assess: no pseudocode, no proof sketch, no error analysis, no full text, so I can't speak to the data analysis or citation pattern. The two neuroimaging applications sound reasonable, but 'biologically plausible' is a weak bar.\n\nBottom line: if the full manuscript actually contains the proof and clearly states the status of the tree, this deserves a serious referee. If the proof is absent or the tree is data-dependent without correction, the central claim collapses. Based on this record alone, I'd ask the authors for the full text before making a final call, but I'd rather see referees engage with it than see it desk rejected.","headline":"Plausible incremental method with a strong statistical claim that the abstract alone cannot support; worth reading the full paper before trusting the FWER guarantee.","tokens_in":1178,"tokens_out":2830,"would_cite":false,"duration_ms":30376,"reading_group":"maybe","serious_thinker":"unclear","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces Hierarchical-CPI, a model-agnostic variable-importance method that tests groups of variables along a tree, giving explicit family-wise error-rate control even under high correlation.","keywords":["hierarchical variable importance","conditional permutation importance","family-wise error rate","model-agnostic interpretability","correlated predictors","medical imaging","group testing","neuroimaging"],"falsifier":"Apply Hierarchical-CPI to synthetic data with a known block-correlation structure where exactly one block of variables drives the outcome and all other variables are noise, then check whether the method controls the family-wise error rate at its nominal level and recovers the predictive block; a failure in either respect would contradict the central claim.","tokens_in":466,"feed_emoji":"🧠","tokens_out":2046,"duration_ms":25002,"temperature":0.7,"pith_summary":"The paper proposes Hierarchical-CPI, a variable-importance method for machine-learning predictions that assesses importance at the level of variable groups arranged in a hierarchical tree rather than at the level of individual variables. The authors aim to solve two problems in medical imaging: conditional importance of single variables vanishes when predictors are highly correlated, and multiple-testing corrections become computationally or statistically costly. They claim that the tree-based group inference provides explicit family-wise error-rate control while remaining computationally tractable, and that the method identifies biologically plausible variables on two neuroimaging benchmarks.","feed_headline":"Group-wise tests give valid importance for correlated medical predictors","feed_subtitle":"Hierarchical-CPI controls family-wise error while staying tractable on imaging data.","key_machinery":"The central object is a hierarchical tree over the variable set, where each node represents a group of variables. At every node, the method tests whether that group, as a whole, is conditionally associated with the outcome given the remaining variables, using a model-agnostic conditional permutation scheme. The tree-based importance allocation mechanism then distributes importance down the tree in a way that avoids vanishing importance under high correlation, while the tree structure keeps the number of tests small enough to permit explicit family-wise error-rate control.","core_discovery":"Hierarchical-CPI frames variable-importance inference as discovering groups of variables that are jointly predictive of an outcome. It walks a hierarchical tree over variables, testing each node as a group with a conditional permutation test, and uses a tree-based importance allocation mechanism to credit variables even when correlation would otherwise cancel their individual signals. The paper's central claim is that this procedure controls the family-wise error rate explicitly, stays computationally feasible, and outperforms current conditional-importance methods on highly correlated imaging data, as demonstrated on dementia classification from MRI and EEG analysis of the Berger effect.","pith_inferences":["The same hierarchical-grouping idea could transfer to other high-correlation domains such as genomics or climate modeling, where individual features are noisy but blocks of features carry signal.","The choice of tree becomes a modeling decision with a direct effect on power and interpretability, so data-driven tree construction or stability checks over alternative trees would be a natural extension.","The tree-based allocation mechanism might be adapted to overlapping or continuous groupings, such as spatial neighborhoods in brain images, rather than only disjoint hierarchical partitions.","A conservative reading is that the explicit FWER control holds for the tree as tested, not necessarily for every variable discovered after allocation; the paper's claims should be read with that distinction in mind."],"forward_implications":["If the method works as claimed, it gives medical researchers a way to obtain statistically valid variable-importance rankings from black-box predictors without sacrificing power on correlated imaging features.","The group-level tests can highlight clusters of jointly predictive variables, which may correspond to interpretable biological structures rather than isolated voxels or electrodes.","Because the method is model-agnostic, it could be applied to any supervised predictor, broadening the use of conditional importance beyond linear or additive models.","The computational tractability claim suggests the procedure can scale to high-dimensional datasets where pixel- or voxel-wise permutation testing would be prohibitive."],"supporting_citations":[],"fun_headline_variants":["Hierarchical-CPI: valid variable importance for correlated imaging","Error-controlled importance for correlated medical predictors","Tree-based importance with explicit false-positive control","Hierarchical-CPI: group-wise importance that survives correlation","Valid importance for correlated imaging via hierarchical testing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's validity rests on the hierarchical tree over variables matching the true dependency or grouping structure: if the tree groups variables incorrectly, the group-level tests and the error-rate control may not say anything meaningful about the actual variables of interest.","fun_headline_variants_meta":{"raw":{"variants":["Hierarchical-CPI: valid variable importance for correlated imaging","Error-controlled importance for correlated medical predictors","Tree-based importance with explicit false-positive control","Hierarchical-CPI: group-wise importance that survives correlation","Valid importance for correlated imaging via hierarchical testing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000572,"raw_usage":{"total_tokens":2645,"prompt_tokens":831,"completion_tokens":1814,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":447,"completion_tokens_details":{"reasoning_tokens":1743}},"tokens_in":447,"tokens_out":1814,"duration_ms":11943,"temperature":1.0,"reasoning_tokens":1743,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:32:04.537665+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Apply Hierarchical-CPI to synthetic data with a known block-correlation structure where exactly one block of variables drives the outcome and all other variables are noise, then check whether the method controls the family-wise error rate at its nominal level and recovers the predictive block; a failure in either respect would contradict the central claim.","supporting_citations":[],"review_version":1}