{"id":"09c38ee1-dc13-4240-be81-fa4eb6aaca58","arxiv_id":"2507.04362","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A kNN-based CMI estimator applied to classification decomposes feature importance into unique, redundant, and synergistic components, validated on synthetic and real data.","lead":"The authors extend the Hi-Fi feature importance method to classification by measuring conditional mutual information with a k-nearest-neighbor estimator. The method splits each feature's contribution into unique, redundant, and synergistic parts and is tested on synthetic and real gene-expression data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Greedy forward selection (Eqs. 4–5) cannot detect synergistic effects that appear only after conditioning on two or more variables jointly: in parity-type systems it terminates at Zmax=∅ and reports S=0 despite a true one-bit synergy, so the claimed high-order capability is unverified.","rationale":"The reader identified the greedy search as the weakest assumption; I agree but sharpen it. The reader's phrasing is 'if the search terminates early or selects spurious variables,' which suggests an empirical or estimation failure. The deeper issue is algorithmic: for genuine higher-order synergy the greedy rule cannot even take its first step, regardless of sample size or estimator quality. Thus the paper's demonstrations, which are pairwise, do not establish the advertised high-order capability. The conditional verdict remains appropriate: the method may be sound for pairwise interactions, and the authors are candid about the limitation, but the claim should be scoped accordingly or the search replaced by subset/block optimization. The parity test directly checks whether this concern lands.","tokens_in":12321,"tokens_out":8526,"duration_ms":95216,"concrete_test":"Generate N=1000 samples from X1, X2, X3, X4 iid Uniform(0,1); set B_i = 1{X_i > 0.5}, Y = B1 xor B2 xor B3, with X4 irrelevant. For source X1, ground-truth components are S = 1 bit and U = R = 0 (the same holds for X2 and X3). Run the proposed search with k=10 and Nsurr=100, using the 95th percentile significance rule as in §III, and record Zmax and S. If the algorithm returns Zmax = ∅ and S ≈ 0, the greedy search fails on a third-order synergy, directly contradicting the general claim of high-order feature effects. Repeat with a four-variable parity (Y = B1 xor B2 xor B3 xor B4) to confirm that the failure persists as the interaction order grows.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the method disentangles high-order feature effects requires the greedy search (Eqs. 4–5) to find the subsets Zmin and Zmax that attain the relevant CMI extrema. The search is strictly sequential: a variable is admitted only if the one-step increment I(Y;X|Zj−1,V) − I(Y;X|Zj−1) is statistically significant. For an interaction of order three or higher that is invisible until several variables are conditioned simultaneously, this one-step test fails by construction. Example: take X1, X2, X3 iid uniform in [0,1], define B_i = 1{X_i > 0.5}, and Y = B1 xor B2 xor B3. For source X1, I(Y;X1) = 0, I(Y;X1|X2) = 0, and I(Y;X1|X3) = 0, while I(Y;X1|X2,X3) = 1 bit. At the first Zmax step every candidate yields a zero (non-significant) increment, so the surrogate test terminates with Zmax = ∅, and the method outputs U = R = S = 0 instead of S = 1 bit. This is a structural limitation of forward selection, not finite-sample noise or a poor k choice. The synthetic validation in §III contains only pairwise synergy (Eq. 18; the X4–X5 block of Eq. 20), so it cannot expose the failure; §V explicitly leaves 'high-dimensional, highly interactive systems' to future work. The 'high-order' part of the central claim is therefore not supported by the evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes an information-theoretic extension of the Hi-Fi method for classification problems. For a source feature X and a conditioning set Z, it defines unique (U), redundant (R), and synergistic (S) contributions using conditional mutual information evaluated at subsets Zmin and Zmax that minimize and maximize I(Y;X|Z). The subsets are found by a greedy forward search with a kNN-based mixed-variable CMI estimator and surrogate-based significance tests. The method is validated on Gaussian synthetic systems whose theoretical values are obtained by Monte Carlo integration, on a non-Gaussian synthetic system with expected qualitative patterns, and on a TCGA-BRCA gene expression dataset.","tokens_in":12737,"tokens_out":6786,"duration_ms":77775,"significance":"The core idea is attractive: a CMI-based decomposition of feature importance avoids committing to a specific PID redundancy measure and yields an observation-driven interpretation of unique, redundant, and synergistic effects. The synthetic Gaussian validation is a methodological strength, since the theoretical values are computed by Monte Carlo integration independently of the kNN estimator, and the reported bias and variance are small. The mixed discrete/continuous kNN estimator is a useful practical contribution. However, the advertised high-order capability is not established: the greedy search can fail on interactions that require joint conditioning, and the experiments only exercise pairwise synergy. With appropriate qualifications and additional validation, the method could be a useful tool for low-order or sequential interaction analysis, but the current presentation overstates its high-order scope.","major_comments":[{"comment":"The greedy forward search used to construct Zmin and Zmax is not guaranteed to find the subsets that attain the extrema of I(Y;X|Z), and this undermines the central claim of capturing high-order feature effects. The search admits a variable only if the one-step increment in CMI is statistically significant. A pure third-order interaction is invisible to this test: take independent uniform X1, X2, X3 and define Y = 1{X1>0.5} xor 1{X2>0.5} xor 1{X3>0.5}. For source X1, I(Y;X1)=0 and I(Y;X1|X2)=I(Y;X1|X3)=0, while I(Y;X1|X2,X3)=1 bit. Both searches therefore terminate at the empty set, and Eqs. (3) produce U=R=S=0 instead of S=1 bit. The same failure occurs for any interaction that appears only after conditioning on two or more variables jointly. Since Eq. (3) is applied to the greedy output, S is not necessarily the maximal synergistic contribution. The synthetic experiments in Eqs. (18) and (20) involve only pairwise synergy and cannot expose this limitation. The authors should either replace the greedy search with a search that can consider joint additions (for example, exhaustive or beam search over small conditioning sets, with a validity statement), or explicitly restrict the claims to sequential and pairwise effects and add a parity-type experiment demonstrating the limitation.","section":"II.A (Eqs. 4-5); III.B (Eqs. 18, 20); V"},{"comment":"The significance test that controls termination of the search is load-bearing but is not calibrated. The paper uses only Nsurr=100 surrogate permutations and a fixed 95th percentile threshold, with no correction for the multiple candidate variables tested at each step and across iterations. With 100 surrogates, the estimated 95th percentile has substantial sampling variability, and in the real-world analysis many candidate features are tested per source. Since the selected Zmin and Zmax directly determine U, R, and S, the operating characteristics of the selection rule matter. The paper should report a calibration experiment (for example, the false-positive selection rate under independent features) or a sensitivity analysis across Nsurr and threshold values.","section":"II.B (surrogate test); III (Nsurr=100); IV.B (real data)"}],"minor_comments":[{"comment":"Equations (4) and (5) write 'j = arg min' and 'j = arg max', but the quantity being selected is the variable V_j; please correct to 'V_j = arg min' and 'V_j = arg max'.","section":"II.A, Eqs. (4)-(5)"},{"comment":"The text introduces 'two additional features X5 and X5' and later refers to X6; the second occurrence should be X6.","section":"IV.A"},{"comment":"The gene label 'SCL22A5' in Figure 6 is inconsistent with 'SLC22A5' used in the text and in the reference list.","section":"Figure 6"},{"comment":"Some references are incomplete, most visibly [2], which lists only the author and volume/page information; please supply full bibliographic details for all entries.","section":"References"},{"comment":"The term 'high-order' is never formally defined; please define the order of an interaction (for example, the size of the conditioning subset) and use it to state precisely which orders the present greedy algorithm can and cannot detect.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper contains a number of self-citations to the authors' prior work (refs. [6, 9, 12]), but these are directly relevant to the method and I do not see an impropriety. The main issue is the mismatch between the advertised high-order capability and what the greedy algorithm can actually compute; this is a substantive correctness concern rather than a presentation issue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me skip the throat-clearing. This paper takes the Hi-Fi decomposition from the authors' earlier regression work, swaps in a kNN-based CMI estimator that handles a discrete class with continuous features, and validates it on Gaussian mixtures with Monte Carlo ground truth. That validation is the real strength: the bias is small, the confidence intervals look honest, and the multiple-interactions setup (Eq. 20) does a decent job of showing that unique, redundant, and synergistic components are separated in a controlled setting. The real-world gene expression analysis is more qualitative, but it's not dressed up as more than that.\n\nThe new piece is mostly engineering: the mixed discrete/continuous kNN estimator, adapted from the companion paper [12], and the way the neighbor search is projected from the full space to avoid the usual bias. The decomposition equations themselves are algebraic identities from refs [6,9]; calling that a circularity would be wrong, since the theoretical values are computed independently via Monte Carlo. Self-citation here is legitimate.\n\nThe soft spot, and it's a real one, is the greedy search in Eqs. (4)-(5). Forward selection admits a variable only if the one-step CMI increment is significant. That means any synergistic effect that only appears after conditioning on two or more variables jointly is invisible: in a parity system with Y = B1 xor B2 xor B3, the first Zmax step sees zero increments for every candidate and terminates with Zmax = empty, reporting S = 0 instead of the true 1 bit. The synthetic validation only exercises pairwise synergy (the X4-X5 block of Eq. 20), so it can't catch this. The paper's conclusion does say 'high-dimensional, highly interactive systems' need future work, but the abstract and title claim high-order capability more strongly than the evidence supports. That's the main revision I'd ask for: either temper the high-order language or add a triple-wise synergy experiment to show the search can sometimes find it.\n\nMinor: Nsurr=100 and the 95th percentile threshold are a bit arbitrary, and the real-world analysis leans on qualitative agreement with the literature. Those are minor.\n\nWho it's for: someone who wants a practical, model-agnostic feature importance breakdown into unique/redundant/synergistic components in a classification setting, and is okay with the greedy-search caveat. It's worth a serious referee, but I'd send it back for a tightened scope and an extra synthetic test.","headline":"Useful classification extension of Hi-Fi with a solid kNN-CMI estimator, but the greedy search means true high-order synergy is likely missed; the paper is honest about this at the end.","tokens_in":13238,"tokens_out":2458,"would_cite":false,"duration_ms":27235,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Feature importance decomposes into unique, redundant, and synergistic parts using conditional mutual information.","keywords":["feature importance","conditional mutual information","k-nearest-neighbor estimation","synergy","redundancy","high-order interactions","classification","gene expression data"],"falsifier":"On a small Gaussian system with known ground truth, compute the decomposition by exhaustive enumeration of all conditioning subsets and compare it with the greedy-search result; a systematic mismatch, or a case where the surrogate test accepts an irrelevant feature often enough to change the sign of $R$ or $S$, would refute the paper's central claim. A second observable: a synthetic system in which a decisive interaction requires a three-variable conditioning set but the greedy search terminates at two variables, producing a nonzero synergy claim that the exhaustive CMI values do not support.","tokens_in":12184,"feed_emoji":"🧠","tokens_out":7457,"duration_ms":70922,"temperature":0.7,"pith_summary":"This paper aims to turn the High-order interactions for Feature importance (Hi-Fi) method into a fully information-theoretic procedure for classification. Instead of measuring the drop in predictive performance when a feature is removed, it measures the conditional mutual information between the class variable and the source feature given subsets of the other features. By finding the conditioning subsets that minimize and maximize this quantity, it splits a feature's total predictive information into unique, redundant, and synergistic components, without committing to a particular redundancy measure. The authors build a k-nearest-neighbor estimator that works with a discrete class and continuous features, and validate it against Monte Carlo ground truth in Gaussian mixture systems, on a non-Gaussian synthetic XOR-like design, and on breast-cancer gene expression data. If the method holds, it gives practitioners a model-independent way to know not just whether a feature matters, but whether it matters alone, in parallel with others, or only in combination.","feed_headline":"Feature effects decomposed into unique, redundant, synergistic parts","feed_subtitle":"Conditional mutual information with kNN estimation recovers ground truth in Gaussian systems and known markers in breast-cancer gene data.","key_machinery":"The central mechanism is the conditional-mutual-information decomposition of Eqs. (2)–(3). For a source feature $X$, class $Y$, and remaining features $\\mathcal{Z}$, the CMI $I(Y;X|\\mathcal{Z})$ is the information about $Y$ that $X$ adds beyond $\\mathcal{Z}$; minimizing it identifies redundant overlap, maximizing it identifies synergy. Two greedy searches (Eqs. (4)–(5)) grow conditioning sets $\\mathcal{Z}_{\\min}$ and $\\mathcal{Z}_{\\max}$ one feature at a time, retaining a feature only when the change in CMI passes a surrogate-based significance test at the 95th percentile. The kNN estimator of Section II.B computes the four mutual information terms in Eq. (6) by first finding neighbors in the highest-dimensional space $\\{X, \\mathcal{Z}_{j-1}, V\\}$ and projecting the resulting distances to the lower-dimensional spaces, reducing the bias that would arise from comparing neighborhoods across different dimensions. The estimator is validated against high-precision Monte Carlo evaluation (Section II.C) of the CMI under a Gaussian-mixture model for the class-conditioned features.","core_discovery":"The central claim is that the full predictive information carried by a source feature $X$ about the class variable $Y$, in the context of all other features $\\mathcal{Z}$, can be decomposed into three interpretable parts using only conditional mutual information: the maximum shared information satisfies $I(Y;X|\\mathcal{Z}_{\\max}) = S + R + U$, with $S = I(Y;X|\\mathcal{Z}_{\\max}) - I(Y;X)$, $R = I(Y;X) - I(Y;X|\\mathcal{Z}_{\\min})$, and $U = I(Y;X|\\mathcal{Z}_{\\min})$, where $\\mathcal{Z}_{\\min}$ and $\\mathcal{Z}_{\\max}$ are the subsets of the other features that, respectively, minimize and maximize the CMI $I(Y;X|\\cdot)$. The paper argues that a greedy forward search, with surrogate-based significance tests deciding when to stop, identifies these subsets, and that a kNN estimator for mixed discrete/continuous variables—computing all required mutual information terms from a single neighbor search in the highest-dimensional space—recovers the theoretical decomposition in simulated Gaussian systems and reproduces expected interaction patterns in non-Gaussian and real breast-cancer gene expression data. The intended payoff is a model-independent feature-importance analysis that separates unique, synergistic, and redundant contributions without relying on any specific partial information decomposition.","pith_inferences":["An exhaustive search over all conditioning subsets on the small simulated systems would quantify how often the greedy search's $\\mathcal{Z}_{\\min}$ and $\\mathcal{Z}_{\\max}$ coincide with the true extrema; the paper does not report this comparison, so the decomposition's fidelity hinges on that unstated agreement.","Because redundancy is defined through the particular minimizing subset found, the 'redundant' label is conditional on the search result rather than an intrinsic property of the feature set; different feature orderings or ties could redistribute $R$ and $U$.","The same greedy-decomposition logic could be transferred to regression settings by estimating CMI with a continuous target, or to high-dimensional genomics data with a stability selection step over bootstrap replicates.","If the method were embedded in a feature-selection pipeline, it would naturally produce small feature groups with complementary (synergistic) rather than merely individually predictive features, since synergy is measured directly."],"forward_implications":["Feature selection can rank features by their estimated unique plus synergistic information, prioritizing features that cannot be replaced by the rest of the input set.","The decomposition gives an observation-driven alternative to partial information decomposition, avoiding the need to choose a redundancy measure.","In the TCGA-BRCA analysis, most of the informational contribution of the top genes is redundant, so removing strongly redundant genes would preserve subtype information.","The method is scalable in practice because the greedy search avoids exhaustive enumeration of all feature subsets.","The estimator's accuracy on small Gaussian systems supports its use on mixed discrete/continuous classification data where ground-truth interactions are unknown."],"supporting_citations":[{"why":"Supplies the original Hi-Fi method and its greedy searching approach that this paper extends to an information-theoretic setting.","marker":"[6]"},{"why":"Provides the CMI-based decomposition of Granger causal effects that motivates decomposing maximum shared information into unique, redundant, and synergistic parts.","marker":"[9]"},{"why":"Supplies the k-nearest-neighbor mutual information estimator that is adapted here to mixed discrete and continuous variables.","marker":"[11]"},{"why":"Provides the mixed-variable kNN estimation approach used to compute conditional mutual information terms.","marker":"[12]"},{"why":"Identifies the bias arising from comparing neighborhoods across different dimensions and motivates the highest-dimensional neighbor search and projection scheme.","marker":"[13]"},{"why":"Introduces the Leave-One-Covariate-Out approach whose predictive-performance change is reinterpreted as conditional mutual information.","marker":"[8]"},{"why":"Provides the TCGA-BRCA dataset and the marginal-contribution feature importance baseline against which real-world findings are compared.","marker":"[18]"},{"why":"Supplies the Monte Carlo integration method used to compute theoretical CMI values for the Gaussian mixture ground truth.","marker":"[14]"},{"why":"Reports a PID-like analysis of the same breast cancer problem, used to compare the nature of synergy and redundancy findings.","marker":"[19]"},{"why":"Gives the PID-based interpretation of CMI as collecting unique and synergistic effects, motivating the feature-selection reading.","marker":"[10]"}],"fun_headline_variants":["Feature importance decomposed into unique, redundant, synergistic parts","CMI decomposition reveals unique, synergistic, redundant feature roles","Information-theoretic feature effects: unique, synergistic, redundant","High-order feature effects split into unique, synergistic, redundant","Decompose feature importance into unique, redundant, synergistic parts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The greedy search with its statistical significance tests is assumed to find the conditioning subsets that truly minimize and maximize the information shared between the class and the source feature; if the search stops too early or selects the wrong features, the split into unique, redundant, and synergistic parts is misassigned.","fun_headline_variants_meta":{"raw":{"variants":["Feature importance decomposed into unique, redundant, synergistic parts","CMI decomposition reveals unique, synergistic, redundant feature roles","Information-theoretic feature effects: unique, synergistic, redundant","High-order feature effects split into unique, synergistic, redundant","Decompose feature importance into unique, redundant, synergistic parts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000777,"raw_usage":{"total_tokens":3458,"prompt_tokens":992,"completion_tokens":2466,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":2385}},"tokens_in":608,"tokens_out":2466,"duration_ms":19287,"temperature":1.0,"reasoning_tokens":2385,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:48:59.951484+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a small Gaussian system with known ground truth, compute the decomposition by exhaustive enumeration of all conditioning subsets and compare it with the greedy-search result; a systematic mismatch, or a case where the surrogate test accepts an irrelevant feature often enough to change the sign of $R$ or $S$, would refute the paper's central claim. A second observable: a synthetic system in which a decisive interaction requires a three-variable conditioning set but the greedy search terminates at two variables, producing a nonzero synergy claim that the exhaustive CMI values do not support.","supporting_citations":[{"cited_title":"The two CMI values are estimated using a mixed vari- able approach based on the kNN estimator of the mu- tual information [11]","cited_arxiv_id":null,"evidence_quote":"Supplies the original Hi-Fi method and its greedy searching approach that this paper extends to an information-theoretic setting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the CMI-based decomposition of Granger causal effects that motivates decomposing maximum shared information into unique, redundant, and synergistic parts."},{"cited_title":"Research initia- tives for innovative technologies and pathways in the health and welfare sector","cited_arxiv_id":null,"evidence_quote":"Supplies the k-nearest-neighbor mutual information estimator that is adapted here to mixed discrete and continuous variables."},{"cited_title":"Ontivero-Ortega, L","cited_arxiv_id":null,"evidence_quote":"Provides the mixed-variable kNN estimation approach used to compute conditional mutual information terms."},{"cited_title":"Information-theoretic Quantification of High-order Feature Effects in Classification Problems","cited_arxiv_id":"2507.04362","evidence_quote":"Introduces the Leave-One-Covariate-Out approach whose predictive-performance change is reinterpreted as conditional mutual information."},{"cited_title":"Xiong, L","cited_arxiv_id":null,"evidence_quote":"Provides the TCGA-BRCA dataset and the marginal-contribution feature importance baseline against which real-world findings are compared."},{"cited_title":"Stramaglia, L","cited_arxiv_id":null,"evidence_quote":"Supplies the Monte Carlo integration method used to compute theoretical CMI values for the Gaussian mixture ground truth."},{"cited_title":"Metropolis and S","cited_arxiv_id":null,"evidence_quote":"Reports a PID-like analysis of the same breast cancer problem, used to compare the nature of synergy and redundancy findings."}],"review_version":1}