Pith. sign in

REVIEW 2 major objections 5 minor 31 references

Information-theoretic Quantification of High-order Feature Effects in Classification Problems

T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Feature importance decomposes into unique, redundant, and synergistic parts using conditional mutual information.

desk verdict Useful classification extension of Hi-Fi with a solid kNN-CMI estimator, but the greedy search means true high-order synergy is likely missed; the paper is honest about this at the end. read the letter →

arxiv 2507.04362 v1 pith:DAIT3Q3W submitted 2025-07-06 cs.LG physics.data-anstat.ML

Ivan Lazic , Chiara Barà , Marta Iovino , Sebastiano Stramaglia , Niksa Jakovljevic , Luca Faes This is my paper · ORCID
classification cs.LGphysics.data-anstat.ML
keywords featureimportanceconditionalmutualinformationk-nearest-neighborestimationsynergyredundancyhigh-orderinteractionsclassificationgeneexpressiondata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to turn the High-order interactions for Feature importance (Hi-Fi) method into a fully information-theoretic procedure for classification. Instead of measuring the drop in predictive performance when a feature is removed, it measures the conditional mutual information between the class variable and the source feature given subsets of the other features. By finding the conditioning subsets that minimize and maximize this quantity, it splits a feature's total predictive information into unique, redundant, and synergistic components, without committing to a particular redundancy measure. The authors build a k-nearest-neighbor estimator that works with a discrete class and continuous features, and validate it against Monte Carlo ground truth in Gaussian mixture systems, on a non-Gaussian synthetic XOR-like design, and on breast-cancer gene expression data. If the method holds, it gives practitioners a model-independent way to know not just whether a feature matters, but whether it matters alone, in parallel with others, or only in combination.

What carries the argument

The central mechanism is the conditional-mutual-information decomposition of Eqs. (2)–(3). For a source feature $X$, class $Y$, and remaining features $\mathcal{Z}$, the CMI $I(Y;X|\mathcal{Z})$ is the information about $Y$ that $X$ adds beyond $\mathcal{Z}$; minimizing it identifies redundant overlap, maximizing it identifies synergy. Two greedy searches (Eqs. (4)–(5)) grow conditioning sets $\mathcal{Z}_{\min}$ and $\mathcal{Z}_{\max}$ one feature at a time, retaining a feature only when the change in CMI passes a surrogate-based significance test at the 95th percentile. The kNN estimator of Section II.B computes the four mutual information terms in Eq. (6) by first finding neighbors in the highest-dimensional space $\{X, \mathcal{Z}_{j-1}, V\}$ and projecting the resulting distances to the lower-dimensional spaces, reducing the bias that would arise from comparing neighborhoods across different dimensions. The estimator is validated against high-precision Monte Carlo evaluation (Section II.C) of the CMI under a Gaussian-mixture model for the class-conditioned features.

What would settle it

On a small Gaussian system with known ground truth, compute the decomposition by exhaustive enumeration of all conditioning subsets and compare it with the greedy-search result; a systematic mismatch, or a case where the surrogate test accepts an irrelevant feature often enough to change the sign of $R$ or $S$, would refute the paper's central claim. A second observable: a synthetic system in which a decisive interaction requires a three-variable conditioning set but the greedy search terminates at two variables, producing a nonzero synergy claim that the exhaustive CMI values do not support.

Watch

Extended reading notes

Core claim

The central claim is that the full predictive information carried by a source feature $X$ about the class variable $Y$, in the context of all other features $\mathcal{Z}$, can be decomposed into three interpretable parts using only conditional mutual information: the maximum shared information satisfies $I(Y;X|\mathcal{Z}_{\max}) = S + R + U$, with $S = I(Y;X|\mathcal{Z}_{\max}) - I(Y;X)$, $R = I(Y;X) - I(Y;X|\mathcal{Z}_{\min})$, and $U = I(Y;X|\mathcal{Z}_{\min})$, where $\mathcal{Z}_{\min}$ and $\mathcal{Z}_{\max}$ are the subsets of the other features that, respectively, minimize and maximize the CMI $I(Y;X|\cdot)$. The paper argues that a greedy forward search, with surrogate-based significance tests deciding when to stop, identifies these subsets, and that a kNN estimator for mixed discrete/continuous variables—computing all required mutual information terms from a single neighbor search in the highest-dimensional space—recovers the theoretical decomposition in simulated Gaussian systems and reproduces expected interaction patterns in non-Gaussian and real breast-cancer gene expression data. The intended payoff is a model-independent feature-importance analysis that separates unique, synergistic, and redundant contributions without relying on any specific partial information decomposition.

Load-bearing premise

The greedy search with its statistical significance tests is assumed to find the conditioning subsets that truly minimize and maximize the information shared between the class and the source feature; if the search stops too early or selects the wrong features, the split into unique, redundant, and synergistic parts is misassigned.

Editorial extensions

If this is right

  • Feature selection can rank features by their estimated unique plus synergistic information, prioritizing features that cannot be replaced by the rest of the input set.
  • The decomposition gives an observation-driven alternative to partial information decomposition, avoiding the need to choose a redundancy measure.
  • In the TCGA-BRCA analysis, most of the informational contribution of the top genes is redundant, so removing strongly redundant genes would preserve subtype information.
  • The method is scalable in practice because the greedy search avoids exhaustive enumeration of all feature subsets.
  • The estimator's accuracy on small Gaussian systems supports its use on mixed discrete/continuous classification data where ground-truth interactions are unknown.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An exhaustive search over all conditioning subsets on the small simulated systems would quantify how often the greedy search's $\mathcal{Z}_{\min}$ and $\mathcal{Z}_{\max}$ coincide with the true extrema; the paper does not report this comparison, so the decomposition's fidelity hinges on that unstated agreement.
  • Because redundancy is defined through the particular minimizing subset found, the 'redundant' label is conditional on the search result rather than an intrinsic property of the feature set; different feature orderings or ties could redistribute $R$ and $U$.
  • The same greedy-decomposition logic could be transferred to regression settings by estimating CMI with a continuous target, or to high-dimensional genomics data with a stability selection step over bootstrap replicates.
  • If the method were embedded in a feature-selection pipeline, it would naturally produce small feature groups with complementary (synergistic) rather than merely individually predictive features, since synergy is measured directly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The manuscript proposes an information-theoretic extension of the Hi-Fi method for classification problems. For a source feature X and a conditioning set Z, it defines unique (U), redundant (R), and synergistic (S) contributions using conditional mutual information evaluated at subsets Zmin and Zmax that minimize and maximize I(Y;X|Z). The subsets are found by a greedy forward search with a kNN-based mixed-variable CMI estimator and surrogate-based significance tests. The method is validated on Gaussian synthetic systems whose theoretical values are obtained by Monte Carlo integration, on a non-Gaussian synthetic system with expected qualitative patterns, and on a TCGA-BRCA gene expression dataset.

Significance. The core idea is attractive: a CMI-based decomposition of feature importance avoids committing to a specific PID redundancy measure and yields an observation-driven interpretation of unique, redundant, and synergistic effects. The synthetic Gaussian validation is a methodological strength, since the theoretical values are computed by Monte Carlo integration independently of the kNN estimator, and the reported bias and variance are small. The mixed discrete/continuous kNN estimator is a useful practical contribution. However, the advertised high-order capability is not established: the greedy search can fail on interactions that require joint conditioning, and the experiments only exercise pairwise synergy. With appropriate qualifications and additional validation, the method could be a useful tool for low-order or sequential interaction analysis, but the current presentation overstates its high-order scope.

major comments (2)
  1. [II.A (Eqs. 4-5); III.B (Eqs. 18, 20); V] The greedy forward search used to construct Zmin and Zmax is not guaranteed to find the subsets that attain the extrema of I(Y;X|Z), and this undermines the central claim of capturing high-order feature effects. The search admits a variable only if the one-step increment in CMI is statistically significant. A pure third-order interaction is invisible to this test: take independent uniform X1, X2, X3 and define Y = 1{X1>0.5} xor 1{X2>0.5} xor 1{X3>0.5}. For source X1, I(Y;X1)=0 and I(Y;X1|X2)=I(Y;X1|X3)=0, while I(Y;X1|X2,X3)=1 bit. Both searches therefore terminate at the empty set, and Eqs. (3) produce U=R=S=0 instead of S=1 bit. The same failure occurs for any interaction that appears only after conditioning on two or more variables jointly. Since Eq. (3) is applied to the greedy output, S is not necessarily the maximal synergistic contribution. The synthetic experiments in Eqs. (18) and (20) involve only pairwise synergy and cannot expose this limitation. The authors should either replace the greedy search with a search that can consider joint additions (for example, exhaustive or beam search over small conditioning sets, with a validity statement), or explicitly restrict the claims to sequential and pairwise effects and add a parity-type experiment demonstrating the limitation.
  2. [II.B (surrogate test); III (Nsurr=100); IV.B (real data)] The significance test that controls termination of the search is load-bearing but is not calibrated. The paper uses only Nsurr=100 surrogate permutations and a fixed 95th percentile threshold, with no correction for the multiple candidate variables tested at each step and across iterations. With 100 surrogates, the estimated 95th percentile has substantial sampling variability, and in the real-world analysis many candidate features are tested per source. Since the selected Zmin and Zmax directly determine U, R, and S, the operating characteristics of the selection rule matter. The paper should report a calibration experiment (for example, the false-positive selection rate under independent features) or a sensitivity analysis across Nsurr and threshold values.
minor comments (5)
  1. [II.A, Eqs. (4)-(5)] Equations (4) and (5) write 'j = arg min' and 'j = arg max', but the quantity being selected is the variable V_j; please correct to 'V_j = arg min' and 'V_j = arg max'.
  2. [IV.A] The text introduces 'two additional features X5 and X5' and later refers to X6; the second occurrence should be X6.
  3. [Figure 6] The gene label 'SCL22A5' in Figure 6 is inconsistent with 'SLC22A5' used in the text and in the reference list.
  4. [References] Some references are incomplete, most visibly [2], which lists only the author and volume/page information; please supply full bibliographic details for all entries.
  5. [Throughout] The term 'high-order' is never formally defined; please define the order of an interaction (for example, the size of the conditioning subset) and use it to state precisely which orders the present greedy algorithm can and cannot detect.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the U/R/S decomposition is definitional, the validation is independent, and self-citations are not load-bearing.

full rationale

I find no circular step that reduces a prediction to an input or to a self-citation. Equations (2)-(3) define U, R, and S so that their sum equals I(Y;X|Zmax); this is a deliberate algebraic identity, not a fitted or empirically derived result. The substantive content lies in the greedy search for Zmin and Zmax (Eqs. 4-5) and in the kNN CMI estimator, and these are validated against Monte-Carlo-computed theoretical values from fully specified Gaussian generative models (Section II C and Section III), so the estimator is checked against an independent numerical evaluation of the same information quantities. The non-Gaussian and TCGA-BRCA analyses are checked against experimenter-controlled interaction patterns and external literature, not against quantities fit by the method. The paper does cite prior work by the same group for the search scheme [6], the CMI decomposition [9], and the mixed kNN estimator [12], but none of these citations replaces an empirical check: the decomposition is elementary algebra and the estimator is tested here on synthetic ground truth. The greedy forward-search limitation for higher-order interactions that appear only after joint conditioning is a correctness/scope concern, acknowledged in Section V, not a circularity. Accordingly, the circularity score is low (2), reflecting only minor self-citation that is not load-bearing.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method depends on several hyperparameters and domain assumptions. The validation uses an independent Monte Carlo estimator for ground truth, and the search is heuristic. No new physical or conceptual entities are introduced.

free parameters (3)
  • k (number of nearest neighbors) = 10
    Set to 10 following common practice in kNN MI estimation (refs [15,16]); not optimized.
  • Nsurr (number of surrogate permutations) = 100
    Used for significance tests of MI and CMI changes; chosen for computational convenience.
  • significance level for surrogate test = 0.05
    95th percentile threshold for rejecting the null hypothesis; conventional but arbitrary.
assumptions (4)
  • domain assumption The mixed kNN estimator provides unbiased estimates of the specific mutual information terms (Eqs. 7-12).
    Borrowed from the authors' companion paper (ref [12]); validated on synthetic data but no formal consistency proof.
  • domain assumption The surrogate permutation test correctly controls the false positive rate for the sequential search.
    Assumed for the significance decisions in Eqs. (4)-(5); with only 100 surrogates the threshold is noisy.
  • domain assumption The joint feature distribution conditioned on the class is a Gaussian mixture in the theoretical validation.
    Used only for computing the Monte Carlo ground truth in Section II.C, not required for the method itself.
  • domain assumption The decomposition into unique, redundant, and synergistic components via Zmin and Zmax is a meaningful characterization of feature importance.
    This is the conceptual framework borrowed from refs [6,9]; no formal proof that the greedy search recovers the intended components.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Information-theoretic Quantification of High-order Feature Effects in Classification Problems." pith.science (2026). https://pith.science/paper/DAIT3Q3W

@misc{pith2026250704362,
  author       = {Pith},
  title        = {Pith review of: Information-theoretic Quantification of High-order Feature Effects in Classification Problems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DAIT3Q3W}},
  note         = {Machine review of arXiv:2507.04362}
}
read the original abstract

Understanding the contribution of individual features in predictive models remains a central goal in interpretable machine learning, and while many model-agnostic methods exist to estimate feature importance, they often fall short in capturing high-order interactions and disentangling overlapping contributions. In this work, we present an information-theoretic extension of the High-order interactions for Feature importance (Hi-Fi) method, leveraging Conditional Mutual Information (CMI) estimated via a k-Nearest Neighbor (kNN) approach working on mixed discrete and continuous random variables. Our framework decomposes feature contributions into unique, synergistic, and redundant components, offering a richer, model-independent understanding of their predictive roles. We validate the method using synthetic datasets with known Gaussian structures, where ground truth interaction patterns are analytically derived, and further test it on non-Gaussian and real-world gene expression data from TCGA-BRCA. Results indicate that the proposed estimator accurately recovers theoretical and expected findings, providing a potential use case for developing feature selection algorithms or model development based on interaction analysis.

Figures

Figures reproduced from arXiv: 2507.04362 by the authors.

Figure 1
Figure 1. Decomposition into unique, redundant and synergi [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Decomposition into unique, redundant and syn [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Selection of the features to be included in the cond [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Order and percentage of occurrence of the features [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Decomposition into unique, redundant and synergi [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: TCGA-BRCA feature analysis results. Barplots rep [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 26 canonical work pages

  1. [12]

    Ontivero-Ortega, L

    M. Ontivero-Ortega, L. Faes, J. M. Cortes, D. Mari- nazzo, and S. Stramaglia, Physical Review E 111, L033301 (2025)

    work page 2025

  2. [1]

    Arsenault, S

    P.-D. Arsenault, S. Wang, and J.-M. Patenaude, ACM Computing Surveys 57, 1–37 (2025)

    work page 2025

  3. [2]

    and basic information-theoretic approaches [3], to more sophisticated techniques such as SHAP, which re- lies on the weighted average contribution of features across possible subsets [4]. Despite their success, these methods often fail to iso- late true marginal contributions of individual features, particularly in the presence of high-order interactions,...

    work page

  4. [3]

    I(Y ;X|Z) = I(Y ;X, Z) −I(Y ; Z)

    exploiting the CMI between the class and the source feature conditioned to the remaining features, i.e. I(Y ;X|Z) = I(Y ;X, Z) −I(Y ; Z). (1) Intuitively, the CMI (1) captures the information pro- vided to the target Y by the source X above and be- yond the information brought by the other sources in Z. This measure was investigated in [10] in terms of PI...

    work page

  5. [4]

    with the selected Zmin and Zmax. B. Estimation approach To perform the searching algorithm and find Zmin and Zmax, at the generic step j we need to compute the two CMI terms to be compared for quantifying the variation in the predictive improvement brought by the candidate variable V , i.e., I(Y ;X|Zj−1,V ) = I(Y ;X, Zj−1,V ) −I(Y ; Zj−1,V ), I(Y ;X|Zj−1) ...

    work page

  6. [5]

    König, E

    G. König, E. Günther, and U. von Luxburg, arXiv preprint arXiv:2410.23772 (2024) , arXiv:2410.23772 [cs.LG]

    arXiv 2024

  7. [6]

    The two CMI values are estimated using a mixed vari- able approach based on the kNN estimator of the mu- tual information [11]

    from a dataset of N observations of the class and feature variables, and then describe the procedure fol- lowed to assess the statistical significance of the CMI variation obtained adding the selected Vj to Zj−1 in- side the conditioning set. The two CMI values are estimated using a mixed vari- able approach based on the kNN estimator of the mu- tual infor...

    work page

  8. [7]

    P. L. Williams and R. D. Beer, arXiv preprint arXiv:1004.2515 (2010) , 11 arXiv:1004.2515 [cs.IT]

    arXiv 2010

Show all 31 references
  1. [8]

    to resolve the prediction error reduction in a regres- sion problem into unique, synergistic, and redundant contributions of the features. The crucial novelty of this approach is that it allows to quantify the High-order ef- fects in Feature importance (Hi-Fi method), capturin...

    2025 arXiv

  2. [9]

    J. R. Vergara and P. A. Estévez, Neural Computing and Applications 24, 175–186 (2013)

    2013

  3. [10]

    Lundberg and S.-I

    S. Lundberg and S.-I. Lee, arXiv preprint arXiv:1705.07874 (2017) , arXiv:1705.07874 [cs.AI]

    2017 arXiv

  4. [11]

    Research initia- tives for innovative technologies and pathways in the health and welfare sector

    is recomputed. This procedure is repeated Nsurr times, producing a distribution of surrogate values. The significance of the original measure is then evaluated by comparing it to the 95th percentile of the surrogate dis- tribution. If the observed value exceeds this threshold, ...

    2022

  5. [13]

    J. Lei, M. G’Sell, A. Rinaldo, R. J. Tibshirani, and L. W. and, Journal of the American Statistical Association 113, 1094 (2018) , https://doi.org/10.1080/01621459.2017.1307116

    2018

  6. [14]

    Stramaglia, L

    S. Stramaglia, L. Faes, J. M. Cortes, and D. Marinazzo, Physical Review Research 6, L032007 (2024)

    2024

  7. [15]

    Wollstadt, S

    P. Wollstadt, S. Schmitt, and M. Wibral, arXiv preprint arXiv:2105.04187 (2023) , arXiv:2105.04187 [cs.IT]

    2023 arXiv

  8. [16]

    Kraskov, H

    A. Kraskov, H. Stögbauer, and P. Grassberger, Phys- ical Review E—Statistical, Nonlinear, and Soft Matter Physics 69, 066138 (2004)

    2004

  9. [17]

    C. Barà, Y. Antonacci, M. Iovino, I. Lazic, and L. Faes, arXiv preprint arXiv:2409.13506 (2024)

    2024 arXiv

  10. [18]

    Xiong, L

    W. Xiong, L. Faes, and P. C. Ivanov, Phys. Rev. E 95, 062114 (2017)

    2017

  11. [19]

    Metropolis and S

    N. Metropolis and S. Ulam, Journal of the American Statistical Association 44, 335 (1949)

    1949

  12. [20]

    C. Barà, R. Pernice, C. A. Catania, M. Hilal, A. Porta, A. Humeau-Heurtier, and L. Faes, Biocybernetics and Biomedical Engineering 44, 380 (2024)

    2024

  13. [21]

    Xiong, L

    W. Xiong, L. Faes, and P. C. Ivanov, Physical Review E 95 (2017), 10.1103/physreve.95.062114

    2017 doi

  14. [22]

    Tomczak, P

    K. Tomczak, P. Czerwińska, and M. Wiznerowicz, Wspolczesna Onkologia 1A, 68–77 (2015)

    2015

  15. [23]

    Catav, B

    A. Catav, B. Fu, J. Ernst, S. Sankarara- man, and R. Gilad-Bachrach, arXiv preprint arXiv:2010.07910 (2020) , arXiv:2010.07910 [cs.LG]

    2020 arXiv

  16. [24]

    Westphal, S

    C. Westphal, S. Hailes, and M. Mu- solesi, arXiv preprint arXiv:2405.19212 (2024) , arXiv:2405.19212 [cs.LG]

    2024

  17. [25]

    Janssen, V

    J. Janssen, V. Guan, and E. Robeva, arXiv preprint arXiv:2204.09938 (2024) , arXiv:2204.09938 [stat.ML]

    2024 arXiv

  18. [26]

    Covert, S

    I. Covert, S. Lundberg, and S.-I. Lee, arXiv preprint arXiv:2004.00668 (2020) , arXiv:2004.00668 [cs.LG]

    2020 arXiv

  19. [27]

    Catalina-Rodriguez, V

    O. Catalina-Rodriguez, V. K. Kolukula, Y. Tomita, A. Preet, F. Palmieri, A. Wellstein, S. Byers, A. J. Giac- cia, E. Glasgow, C. Albanese, and M. L. A vantaggiati, Oncotarget 3, 1220–1235 (2012)

    2012

  20. [28]

    Katnik, A

    E. Katnik, A. Gomulkiewicz, A. Piotrowska, J. Grzegrzolka, A. Rusak, A. Kmiecik, K. Ratajczak-Wielgomas, and P. Dziegiel, Current Issues in Molecular Biology 45, 2681–2698 (2023)

    2023

  21. [29]

    Orrantia-Borunda, P

    E. Orrantia-Borunda, P. Anchondo-Nuñez, L. E. Acuña-Aguilar, F. O. Gómez-Valles, and C. A. Ramírez-Valdespino, in Breast Cancer (Exon Publica- tions, 2022) pp. 31–42

    2022

  22. [30]

    C. Wang, I. P. Uray, A. Mazum- dar, J. A. Mayer, and P. H. Brown, Breast Cancer Research and Treatment 134, 101–115 (2012)

    2012

  23. [31]

    K. Xu, J. Usary, P. C. Kousis, A. Prat, D.-Y. Wang, J. R. Adams, W. Wang, A. J. Loch, T. Deng, W. Zhao, R. D. Cardiff, K. Yoon, N. Gaiano, V. Ling, J. Beyene, E. Zacksenhaus, T. Gridley, W. L. Leong, C. J. Guidos, C. M. Perou, and S. E. Egan, Cancer Cell 21, 626–641 (2012)

    2012

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.