{"id":"a4a77920-1a1b-4a0b-b152-abdc0a62dfcc","arxiv_id":"2412.02805","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"STORM uses entropy imbalance and decision-tree logic to select informative modalities for rare-event classification, and reports that temporal expert features do not help SOZ detection.","lead":"STORM is a new algorithm that chooses the most useful data types for spotting rare events like seizure-onset zones in brain scans, and it concludes one expert-suggested temporal data type adds nothing. Generalists might read it because it tackles the widespread problem that using all available data is often wasteful and no more accurate than using the right subset.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Entropy imbalance ηMd is not established as a valid proxy for modality discriminative value; a synthetic overlapping-features test could settle this.","rationale":"The reader's weakest_assumption — that low ηMd does not necessarily imply discriminative value — is exactly the load-bearing concern I identify. The paper's own Equation (5) and the sentence 'Ideally the best classifier should have an intermediate representation that has the lowest value of ηMd' are the unproven pivot. My stress test adds two specifics beyond the reader's general concern: (1) a concrete failure mode where balanced entropies coexist with class overlap, and (2) a methodological mismatch in Eq. (6) where ηR and ηMd are computed from different representation types. The proposed synthetic experiment is simple, cheap, and would definitively test the causal link between ηMd and classification performance. The paper does provide a real clinical dataset and a plausible pipeline, but it lacks the controlled validation needed to justify a general modality-selection algorithm. Since the concern is testable and the current verdict already conditions acceptance on additional evidence, I do not move the verdict; I agree that CONDITIONAL is appropriate. If the synthetic test fails, the verdict would need to move toward REJECT; if it passes, the central assumption would be substantially supported.","tokens_in":8224,"tokens_out":3200,"duration_ms":35651,"concrete_test":"Build a synthetic binary classification dataset with a rare class (e.g., 5% positives) and two candidate modalities: M_A, features drawn from well-separated class-conditional Gaussians but with unequal variances (yielding large ηMd), and M_B, features drawn from identical overlapping Gaussians for both classes (yielding ηMd ≈ 0). Run Algorithm 1 with the same classifier (e.g., SVM) on each modality separately, using the paper's configuration. If STORM selects M_B or judges it equivalent to M_A, the ηMd criterion fails. Additionally, create 100 random linear projections of a fixed dataset with known class labels, compute ηMd and rare-class F1 for each projection, and report the Spearman correlation. The central claim requires a monotone (at least non-negative) relationship between low ηMd and high rare-class F1.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on Eq. (5): the classifier with the lowest entropy imbalance ηMd is 'ideally the best' because 'representative class features were learned' (Sec. II.B). This is asserted, not proved. ηMd = max_r(θr − E(θr)) measures only the degree to which class-wise entropies of the learned representations are imbalanced, i.e., whether the local density of one class's representations differs from the average. It does not measure inter-class separation, class-specific rare-event information, or the ability to distinguish the rare class from others. A representation with completely overlapping classes can have nearly identical class entropies and hence ηMd ≈ 0, while a representation with well-separated classes but unequal within-class variances can have large ηMd. Therefore, minimizing ηMd can select a useless modality over a discriminative one. The paper provides no theoretical derivation, no controlled experiment, and no statistical evidence linking ηMd to rare-class F1; the single SOZ case study lacks error bars and baselines. Additionally, Eq. (6) compares ηR computed from raw-data representations (PSNR or VGG penultimate layer) with ηMd from classifier intermediate representations (CNN/SVM); if these representations live on different geometric scales, the 'gain' EIG may reflect representation geometry rather than modality informativeness. The central claim that STORM 'identifies the most discriminative features' is therefore unsupported unless the metric's predictive validity is demonstrated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents STORM, an entropy-based algorithm for selecting modalities in rare-event classification. The algorithm computes class-wise entropy imbalances of learned representations and selects modalities that maximize the entropy imbalance gain. The method is applied to seizure onset zone (SOZ) detection from resting-state fMRI, using a basic spatial modality and two expert-derived modalities (D1 spatial, D2 temporal). The reported results suggest that the basic modality and D1 are necessary, while D2 is unnecessary for SOZ classification. The paper also mentions a coronary artery disease case study but does not report results for it.","tokens_in":8561,"tokens_out":4311,"duration_ms":38624,"significance":"If validated, the STORM framework could offer a principled alternative to ad-hoc modality selection in biomedical AI, with the potential to reduce data collection and processing costs. The paper's strengths include its focus on a clinically relevant problem, the use of expert knowledge, and the presentation of a concrete algorithm with pseudocode. However, the central entropy-imbalance criterion is not theoretically derived or empirically validated, and the reported experiments lack error bars and statistical rigor. The contribution is therefore currently a proposal with a single case study, rather than an established method.","major_comments":[{"comment":"The assertion that the classifier with the lowest ηMd is ideally the best because 'representative class features were learned' is not justified. ηMd only measures the dispersion of class-wise entropies; it does not directly measure inter-class separation or the information content of a modality for discriminating the rare class. A task with overlapping classes can have low ηMd, while a task with well-separated classes but heterogeneous within-class densities can have high ηMd. The authors need to provide a controlled experiment (e.g., synthetic data with known discriminative structure) that demonstrates a monotonic relationship between ηMd and rare-class F1 or similar metrics.","section":"Section II.B, Eq. (5)"},{"comment":"The 0% rows for 'Modality D1 excluded' and 'Modality D1b excluded' are not valid performance measurements; they indicate a degenerate classifier output. The authors should explain what occurred (e.g., no positive predictions) and report results over cross-validation folds with standard deviations. Additionally, the identical scores across 'All modalities included,' 'Modality D2 excluded,' 'Modality D2a excluded,' and 'Modality D2b excluded' suggest that D2 features are not being used in the final ensemble; the paper must verify that the pipeline correctly consumes D2 features and that the result is not an artifact of a code path.","section":"Section III.B, Table I"},{"comment":"The entropy imbalance gain EIG compares ηR (from raw-data representations such as PSNR or VGG penultimate-layer features) with ηMd (from classifier intermediate representations). Since these representations have different dimensionalities and scales, the difference may reflect representation geometry rather than modality informativeness. The authors should either normalize the entropies or use comparable representation types when computing the gain.","section":"Section II.B, Eq. (6)"},{"comment":"The class-wise entropy computation depends on the choice of distance metric and the nearest-neighbor count K, but the paper neither states the value of K nor analyzes the sensitivity of the results to K and to the thresholds ϵm, ϵg, and dth. These free parameters should be justified and a robustness analysis should be provided.","section":"Section II.A"},{"comment":"The second case study (CAD detection) is described only as a potential application, with no experimental results. Since the abstract and introduction claim a general-purpose algorithm, the inclusion of a second case study would strengthen the empirical support; as it stands, the paper provides only a single clinical demonstration.","section":"Section III.C"}],"minor_comments":[{"comment":"The phrase 'Note than even though' should be 'Note that even though.'","section":"Section I.C"},{"comment":"The use of PSNR as a representation for raw data is unclear, as PSNR is a scalar quality metric; the authors should clarify what distance between instances is being computed.","section":"Section II.A"},{"comment":"The connection to Hunt's algorithm is mentioned but not explained; a brief description of how the decision tree is built would improve readability.","section":"Section II.B"},{"comment":"The table notes N=52 but does not specify the number of SOZ instances; the class distribution should be reported to contextualize the rare-event setting.","section":"Table I"},{"comment":"Reference [24] is incomplete ('R. U. et al.'); the full author list should be provided.","section":"References"},{"comment":"Figure 2 is not described in the text; the caption should be self-contained and the figure should be referenced in the relevant section.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The paper relies heavily on self-citations (refs [2], [6], [8], [9], [11], [12], [13], [16]), which may be appropriate given the authors' prior work on the same dataset, but the novelty of STORM relative to those works should be clarified. The editor may also wish to consider that the empirical evidence is limited to a single clinical dataset with a small sample size."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper has a plausible idea and one potentially useful empirical observation, but the load-bearing metric is asserted rather than validated, and the single evaluation is too fragile to support the claims.\n\nWhat's new: STORM names a specific way to combine class-wise entropy with a decision-tree search over modalities, and on this pediatric rs-fMRI dataset it finds that the temporal derived modality D2 adds nothing to SOZ detection. That last result is genuinely interesting if real, because it contradicts the expert assumption that both D1 and D2 matter.\n\nSoft spots: the entropy imbalance ηMd is the core of the method, but the paper never shows that lower ηMd tracks better rare-class separability. The stress-test note is right: overlapping classes can have similar entropies, so ηMd≈0, while well-separated classes with unequal variances can have high ηMd. Without a synthetic test or a theoretical argument, the selection criterion isn't grounded. Second, Table I is hard to trust. No error bars or cross-validation on a dataset of 52 patients. The 0% rows for D1-excluded and D1b-excluded configurations look like pipeline failures, not measured performance—if removing one rule feature truly collapsed F1 from 91.6 to 0, that would suggest the model is not learning anything robust. Also, EIG is computed from trained classifiers on the same data used to select them, so there's a circularity that isn't addressed. The 'another case study' on CAD is a paragraph, not a case study, and there are no comparisons to existing modality-selection methods, including the ones they cite (e.g., He et al. [4]).\n\nWho it's for: someone working on multi-modal medical ML who wants a concrete example of how not to validate a selection criterion. It could be useful as a cautionary case.\n\nRecommendation: I'd send it to peer review because the underlying question is worthwhile and the empirical claim about D2 is testable. But the reviewers should ask for a validation of ηMd (synthetic experiments, comparison with feature-selection baselines) and a much more careful evaluation with confidence intervals. As it stands, I wouldn't cite it for the method.","headline":"The entropy-imbalance selection criterion is asserted, not validated, and the single fragile evaluation makes the D2-redundancy finding interesting but unproven.","tokens_in":9033,"tokens_out":2704,"would_cite":false,"duration_ms":27776,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"STORM is an entropy-based algorithm that selects the minimal informative modality set for rare-event classification, and in seizure-onset-zone detection it shows the expert-derived temporal modality is redundant, with F1 unchanged at 91.6%.","keywords":["modality selection","rare event classification","entropy imbalance","seizure onset zone","multimodal learning","class imbalance","expert knowledge","rs-fMRI"],"falsifier":"Feed STORM a modality whose features are randomized within each class, so class labels carry no feature information; if the entropy-imbalance gain EIG for that modality stays high enough to be selected, the metric is not measuring discriminative content. A second check: on a dataset built so the rare class is visible only in the temporal modality, STORM should select that temporal modality, and if it drops it, the selection rule is wrong.","tokens_in":8048,"feed_emoji":"🧠","tokens_out":8777,"duration_ms":80945,"temperature":0.7,"pith_summary":"STORM is an entropy-based algorithm for deciding which data modalities a classifier actually needs when the target event is rare. It scores each candidate modality by the spread of class-wise entropies in its learned representation, on the theory that a modality with the smallest such spread has learned the most discriminative class structure. In a seizure-onset-zone detection study on 52 pediatric epilepsy patients, STORM identifies the raw spatial fMRI modality plus one expert-derived spatial feature set as sufficient; the expert-derived temporal modality is unnecessary. Omitting the temporal modality leaves accuracy at 84.6% and F1 at 91.6% unchanged, while omitting the spatial-derived features collapses F1 to 0%. The paper argues this replaces manual trial-and-error modality selection with a systematic, quantifiable criterion.","feed_headline":"STORM: entropy-based modality picker keeps F1 at 91.6%","feed_subtitle":"On 52 epilepsy patients, dropping the temporal modality leaves F1 unchanged, saving compute.","key_machinery":"The central object is the entropy imbalance metric $\\eta_{M_d}$, computed by (i) taking a trained classifier's intermediate representation $F_{M_d}(y_i)$ for each instance, (ii) measuring each instance's local density via K-nearest-neighbor distances, (iii) converting those densities into a class-wise entropy $\\theta_r$ using the entropy-based sampling formulas of [19], and (iv) taking the maximum deviation $\\max_{c_r}(\\theta_r - \\mathbb{E}(\\theta_r))$ across classes. STORM then uses the gain $\\mathrm{EIG}(M_d) = \\eta_R - \\eta_{M_d}$ as a splitting criterion in a decision tree built with Hunt's algorithm, with thresholds on EIG equivalence, Gini impurity, and classifier confidence controlling when to stop or cascade classifiers.","core_discovery":"The paper claims that modality selection for rare-event classification can be solved by measuring, for each candidate modality's learned representation, the spread of class-wise entropies: $\\eta_{M_d} = \\max_{c_r} (\\theta_r^{M_d} - \\mathbb{E}(\\theta_r^{M_d}))$, where $\\theta_r^{M_d}$ is the entropy of class $c_r$ under classifier $M_d$. A modality whose representation has the lowest entropy imbalance is taken to have learned the most discriminative class structure, and selection proceeds greedily via Hunt's algorithm using the entropy-imbalance gain $\\mathrm{EIG}(M_d) = \\eta_R - \\eta_{M_d}$. In the seizure-onset-zone case study, applying STORM identifies the basic spatial modality plus the expert-derived spatial modality D1 as sufficient; the expert-derived temporal modality D2 is deemed unnecessary because excluding it leaves accuracy at 84.6% and F1 at 91.6% unchanged, while excluding D1 collapses F1 to 0%.","pith_inferences":["By the same logic, the entropy-imbalance criterion should transfer to any multimodal pipeline with per-modality learned representations; a natural next experiment is the CAD case the paper sketches but does not run, comparing ECG images against extracted time-series leads.","The ablation pattern suggests feature-level selection is the next layer: since removing D1b alone collapses F1 to 0 exactly as removing all of D1 does, STORM could be run over individual expert rules rather than whole modalities, likely pruning D2a/D2b even within a selected modality.","A consequence of this design is a divergence between expert usefulness and machine usefulness: temporal features (D2) are reported as meaningful for human SOZ reading but contribute nothing to the classifier, implying that clinically meaningful modalities may still be algorithmically redundant, which matters for data collection and storage decisions."],"forward_implications":["On the 52-patient rs-fMRI dataset, the full modality set and the reduced set (basic plus D1) both reach 84.6% accuracy and 91.6% F1; dropping D2 costs nothing.","Removing the spatial-derived modality D1, or just its D1b rule, drops F1 to 0%, so at least one expert-derived spatial rule is load-bearing for SOZ detection.","Raw data alone is not enough: excluding all derived modalities drops F1 to 63.0%, and excluding the basic modality drops it to 67.0%, so basic and derived modalities contribute complementary information.","STORM gives a quantitative stopping rule for modality collection: a modality whose entropy-imbalance gain is within threshold of another's can be omitted before training a full multimodal model.","The approach is framed for rare-event biomedical tasks and is proposed for other expert-guided domains, such as coronary artery disease detection from ECG images versus extracted time-series leads."],"supporting_citations":[{"why":"Defines the SOZ detection task and the expert rules D1a/D1b/D2a/D2b that the derived modalities are built from.","marker":"[11]"},{"why":"Supplies the rs-fMRI dataset, ICA preprocessing, and the evaluation metrics used in the case study.","marker":"[12]"},{"why":"Provides the class-wise entropy and density formulas (Eqns. 1-3) on which the entropy imbalance metric is based.","marker":"[19]"},{"why":"Supplies the VGG16 penultimate-layer representation tested as an alternative raw-data representation.","marker":"[20]"},{"why":"Provides Hunt's algorithm used to build the decision tree that selects modalities by entropy-imbalance gain.","marker":"[21]"},{"why":"Provides the Gini impurity measure used as the stopping and purity threshold in STORM.","marker":"[22]"},{"why":"Supplies SMOTE, the oversampling method applied before SVM for the derived-modality classifiers.","marker":"[23]"}],"fun_headline_variants":["STORM picks spatial only, drops temporal, F1 stays 91.6%","Entropy-based STORM keeps F1 at 91.6% while dropping temporal","STORM: drop temporal, keep spatial, F1 unchanged at 91.6%","STORM modality selection: spatial alone suffices, F1 91.6%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the entropy-imbalance value $\\eta_{M_d}$ of a modality's learned representation tracks how much discriminative information that modality actually carries for the rare class; if low entropy imbalance does not predict good rare-class separability, STORM's ranking of modalities is not grounded.","fun_headline_variants_meta":{"raw":{"variants":["STORM picks spatial only, drops temporal, F1 stays 91.6%","Entropy-based STORM keeps F1 at 91.6% while dropping temporal","STORM: drop temporal, keep spatial, F1 unchanged at 91.6%","STORM modality selection: spatial alone suffices, F1 91.6%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000416,"raw_usage":{"total_tokens":2132,"prompt_tokens":919,"completion_tokens":1213,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":1130}},"tokens_in":535,"tokens_out":1213,"duration_ms":9321,"temperature":1.0,"reasoning_tokens":1130,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:04:52.827724+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Feed STORM a modality whose features are randomized within each class, so class labels carry no feature information; if the entropy-imbalance gain EIG for that modality stays high enough to be selected, the metric is not measuring discriminative content. A second check: on a dataset built so the rare class is visible only in the temporal modality, STORM should select that temporal modality, and if it drops it, the selection rule is wrong.","supporting_citations":[{"cited_title":"The Expert’s Knowledge combined with AI outperforms AI Alone in Seizure Onset Zone Localization using resting state fMRI,","cited_arxiv_id":null,"evidence_quote":"Defines the SOZ detection task and the expert rules D1a/D1b/D2a/D2b that the derived modalities are built from."},{"cited_title":"Automated seizure onset zone locator from resting-state functional MRI in drug-resistant epilepsy,","cited_arxiv_id":null,"evidence_quote":"Supplies the rs-fMRI dataset, ICA preprocessing, and the evaluation metrics used in the case study."},{"cited_title":"Entropy-based sampling approaches for multi- class imbalanced problems,","cited_arxiv_id":null,"evidence_quote":"Provides the class-wise entropy and density formulas (Eqns. 1-3) on which the entropy imbalance metric is based."},{"cited_title":"Deep facial analysis: A new phase I epilepsy evaluation using computer vision,","cited_arxiv_id":null,"evidence_quote":"Supplies the VGG16 penultimate-layer representation tested as an alternative raw-data representation."},{"cited_title":"More Interpretable Decision Trees,","cited_arxiv_id":null,"evidence_quote":"Provides Hunt's algorithm used to build the decision tree that selects modalities by entropy-imbalance gain."},{"cited_title":"A simple method for measuring inequality,","cited_arxiv_id":null,"evidence_quote":"Provides the Gini impurity measure used as the stopping and purity threshold in STORM."}],"review_version":1}