REVIEW 4 major objections 4 minor 22 references
Assessing high-order effects in feature importance via predictability decomposition
T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read This paper claims that a single Leave One Covariate Out importance score can be decomposed into unique, redundant, and synergistic components by comparing the smallest and largest LOCO values over feature subsets.
desk verdict Useful heuristic for decomposing LOCO, but the synergy/redundancy labels ride on an unproven analogy and need external benchmarking. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying identity is Eq. (2), $L_{z_{\max}}(X \to Y) = S + R + U$, with $U = L_{z_{\min}}$, $R = L_{\emptyset} - L_{z_{\min}}$, and $S = L_{z_{\max}} - L_{\emptyset}$, where $L_z$ is LOCO over subset $z$ and $L_{\emptyset}$ is the pairwise explained variance. The identity converts the definitions of redundancy as information retrievable from multiple sources and synergy as information available only from the whole into a search over feature subsets. The subsets are found exhaustively for small $n$ and, for larger $n$, by a greedy one-variable-at-a-time search with a permutation-based stopping rule.
What would settle it
For a small Gaussian system where a full information-theoretic partial information decomposition can be computed exactly, compare its redundancy and synergy values with Hi-Fi's $R$ and $S$; a qualitative disagreement across correlation strengths would show that the min/max LOCO labels are not tracking the quantities they are named after.
Extended reading notes
Core claim
On the paper's own terms, LOCO computed with the best subset $z_{\max}$ decomposes exactly as $L_{z_{\max}} = S + R + U$. Here $U$ is the LOCO value at the subset that minimizes it, the unique two-body contribution; $R$ is the gap between the pairwise explained variance and that minimum, the redundancy shared with other features; and $S$ is the gap from pairwise variance to the maximizing LOCO, the synergy supplied by partners. The authors therefore claim that the maximal predictive power of $X$ to $Y$ is a sum of a pure two-body effect and two cooperative effects, and that the optimizing subsets name which other features participate.
Load-bearing premise
The load-bearing premise is that the smallest and largest LOCO values over feature subsets can be read as redundancy and synergy; the paper states this mapping rather than proves it, and the variance-based measure deliberately gives up formal information-theoretic guarantees.
Editorial extensions
If this is right
- Features that are redundant with others look less important under standard LOCO; Hi-Fi restores them by assigning the lost predictive power to $R$.
- Features that matter only in combination, such as suppressors, are missed by pairwise explained variance; Hi-Fi assigns their effect to $S$ and names the partners.
- In the particle discrimination example, $\beta$ and $p$ are synergistic because together they encode particle mass, while the detector responses are redundant with $\beta$ and $p$ because of Bethe-Bloch energy loss.
- On the datasets analyzed, the greedy search reaches the same decomposition as exhaustive search, so the method can scale beyond the toy case.
- The components depend on the chosen regression hypothesis space, so robustness can be checked by repeating the analysis with several models.
Reading between the lines
- Beyond the paper's demonstrations, a direct check of the nomenclature would compute a true partial information decomposition on a Gaussian system and compare its $R$ and $S$ with Hi-Fi's components; the paper does not supply this validation.
- The greedy search's stopping rule has only been compared with exhaustive search on small-$n$ datasets; in high-dimensional feature sets the global min or max could be unreachable, so the labels may depend on the search path.
- Because the decomposition is built on prediction-error variance, the same recipe could be applied to other losses, making Hi-Fi a general way to expose cooperation in classification or survival models, not just regression.
- The synergy term effectively detects suppressor variables, so Hi-Fi could double as a screening tool for suppression in regression diagnostics.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Hi-Fi, an adaptive version of Leave One Covariate Out (LOCO) in which, for a given driver variable X, one searches for the subset of covariates that minimizes and the subset that maximizes the LOCO value. Defining U = L_{zmin}, R = L_empty - L_{zmin}, and S = L_{zmax} - L_empty leads to the identity L_{zmax} = S + R + U (Eq. 2). The authors interpret U, R, and S as unique, redundant, and synergistic predictive contributions, respectively, and illustrate the method on a constructed Gaussian toy model, a wine-quality dataset, and a GEANT-simulated proton/pion discrimination problem. A greedy search with permutation-based stopping is used for scalability, and code is provided.
Significance. If the min/max LOCO decomposition can be rigorously connected to information-theoretic redundancy and synergy, Hi-Fi would be a simple and computationally efficient extension of one of the most widely used feature-importance measures. The algebraic identity is correct, the code is open, and the particle-physics interpretation involving the Bethe-Bloch formula is physically plausible. However, the central conceptual contribution currently rests on an analogy that is asserted rather than derived, and the empirical claims lack any form of uncertainty quantification. The paper is therefore potentially useful but needs substantial additional support before the labels 'unique', 'redundant', and 'synergistic' can be taken as factual descriptions of feature behavior.
major comments (4)
- [Eq. (2) and surrounding definitions] The decomposition in Eq. (2) is an algebraic identity: for any subsets zmin and zmax, L_{zmax} = L_{zmin} + (L_empty - L_{zmin}) + (L_{zmax} - L_empty). The substantive claim is that the subset minimizing Lz 'captures the amount of redundancy R' and the subset maximizing Lz 'leads to the amount of synergy S,' but this mapping is asserted rather than derived. The paper's own concession that replacing entropy with variance 'sacrifices the formal rigor of information theory' (citing [11]) means that the labels unique, redundant, and synergistic are not automatically the PID quantities. I request either a formal derivation showing under which conditions min/max LOCO equals the corresponding PID components, or a benchmark against a recognized PID measure on small synthetic systems, or a clear reframing of the method as a heuristic whose names are only analogical. As written, the interpretations in Figs. 1, 3, and 5 are not supported as factual statements about information-theoretic redundancy and synergy.
- [Toy example and Figs. 1-2] The toy example does not independently validate the mapping. Because U, R, and S are defined directly from Lzmin and Lzmax, a correctly implemented computation will always recover the labels used to construct the example. The example shows internal consistency, but it does not show that min/max LOCO tracks redundancy and synergy in the information-theoretic sense. A benchmark in which the true PID redundancy and synergy are known, such as a comparison against a recognized PID measure on the same Gaussian or categorical system, would be needed.
- [Greedy search and final paragraph] The manuscript states that exhaustive evaluation gave the same results as the greedy search on the analyzed datasets, but it does not report the exhaustive-search procedure, the stopping thresholds, or any comparison. Because the greedy stopping rule involves a permutation-test significance threshold and multiple-comparison correction, the reader cannot assess whether the reported multiplets are stable or artifact-driven. Please provide the implementation details and either the exhaustive results or a sensitivity analysis across thresholds.
- [Empirical results and Figs. 3-6] None of the empirical claims in Figs. 3-6 are accompanied by uncertainty quantification. The predictions are evaluated on the training data (per footnote 14), and the differences between, for example, the synergistic role assigned to residual sugar and the beta-p synergy are presented as point estimates only. The absence of confidence intervals or cross-validated errors makes it impossible to judge whether the reported differences are meaningful. This is load-bearing for the paper's concrete claims about wine quality and particle discrimination.
minor comments (4)
- [Fig. 3 caption] The caption of Fig. 3 refers to the 'particle discrimination problem,' but the figure shows the Wine Quality dataset; the caption should be corrected.
- [Text near reference [19]] The text says 'UCL Machine Learning Repository'; the correct name is the UCI Machine Learning Repository.
- [Reproducibility details] The polynomial kernel degree is stated, but regularization or kernel scaling parameters, if any, are not specified, which makes exact reproduction difficult despite the provided code.
- [Notation] The notation for the full set Z and the subset z is visually similar; please distinguish them clearly, for example by using bold or calligraphic symbols, to avoid ambiguity in Eqs. (1) and (2).
Circularity Check
The decomposition in Eq. (2) is an algebraic identity: U, R, and S are defined so that they sum to Lzmax, and the redundancy/synergy labels are imported by analogy from the authors' own Ref. [13] rather than derived from partial information decomposition.
-
self definitional
[Methodology, page 5, Eq. (2) and preceding paragraph]
"it is intuitive that searching for zmin minimizing Lz(X → Y ) captures the amount of redundancy R ... we may define R = L∅(X → Y ) − Lzmin(X → Y ). The unique predictive power U ... will be given by U = Lzmin(X → Y ). On the other hand, searching for zmax maximizing Lz(X → Y ), leads to the amount of synergy S ... S = Lzmax(X → Y ) − L∅(X → Y ). It follows that: Lzmax(X → Y ) = S + R + U, (2)"
U, R, and S are defined as Lzmin, L∅−Lzmin, and Lzmax−L∅, so S+R+U = Lzmax is an algebraic identity that holds for any chosen subsets zmin and zmax. The decomposition therefore has no independent content; the three terms are constructed to sum to Lzmax. The semantic claims that U is a 'pure two-body effect', R is 'redundancy', and S is 'synergy' are asserted via 'it is intuitive' and 'we may define', not derived from partial information decomposition. All subsequent interpretations in Figs. 1–6 rest on this asserted labeling rather than on a demonstrated equivalence.
-
self citation load bearing
[Introduction and methodology transition, pages 4–5]
"As discussed in [13], LZ(X → Y ) underestimates the importance of X when X exhibits redundancy with some variables in Z. Conversely, considering only the driver X ... overlooks synergies between X and Z. ... Adapting this framework by replacing conditional mutual information with LOCO enables a decomposition of LOCO into unique, redundant, and synergistic components"
The paper's central interpretive claim—that the subset minimizing Lz corresponds to redundancy and the subset maximizing Lz corresponds to synergy—is justified by reference to [13], which is authored by the same four authors (Stramaglia, Faes, Cortes, Marinazzo). The paper itself concedes that substituting variance for entropy 'sacrifices the formal rigor of information theory', so the min/max LOCO quantities are not shown to equal any established PID redundancy or synergy measure. Without an independent benchmark or derivation, the labels are an imported ansatz from the authors' prior work rather than a result established here.
full rationale
The paper is transparent about its definitions, but the core decomposition is definitional: Eq. (2) is a tautology given the definitions of U, R, and S. The main novel content is the semantic interpretation of the min/max LOCO terms as unique, redundant, and synergistic contributions, and that interpretation is inherited from the authors' own Ref. [13] by analogy. The paper's own admission that replacing entropy with variance 'sacrifices the formal rigor of information theory' reinforces that the mapping is an assumed analogy, not a proven equivalence. The toy example is a sanity check of the definitions rather than an independent validation, since it is constructed so that the heuristic yields the intended labels. The real-data applications do provide independent content in the form of computed LOCO values and greedy subset searches, and the physical interpretations (e.g., beta and p synergy in particle discrimination) are plausible; this independent anchoring prevents the paper from being fully circular. However, because the central decomposition and its labels reduce to definitions plus a load-bearing self-citation, a moderate circularity score is appropriate.
Assumptions & free parameters
free parameters (3)
- Permutation-test significance threshold for greedy stopping
- Number of surrogates for permutation tests
- Polynomial kernel degree =
2
assumptions (4)
- standard math LOCO is non-negative for the considered regression hypothesis spaces (linear models, inhomogeneous polynomial and Gaussian kernels).
- domain assumption In-sample mean squared error approximates generalization error when sample size is large.
- ad hoc to paper For the analyzed datasets, greedy subset search yields the same result as exhaustive search.
- ad hoc to paper The min/max LOCO search captures information-theoretic redundancy and synergy.
invented entities (1)
-
Hi-Fi decomposition components (U, R, S)
Cite this review
Pith. "Pith review of Assessing high-order effects in feature importance via predictability decomposition." pith.science (2026). https://pith.science/paper/CI2DBBF3
@misc{pith2026241209964,
author = {Pith},
title = {Pith review of: Assessing high-order effects in feature importance via predictability decomposition},
year = {2026},
howpublished = {\url{https://pith.science/paper/CI2DBBF3}},
note = {Machine review of arXiv:2412.09964}
}
read the original abstract
Leveraging the large body of work devoted in recent years to describe redundancy and synergy in multivariate interactions among random variables, we propose a novel approach to quantify cooperative effects in feature importance, one of the most used techniques for explainable artificial intelligence. In particular, we propose an adaptive version of a well-known metric of feature importance, named Leave One Covariate Out (LOCO), to disentangle high-order effects involving a given input feature in regression problems. LOCO is the reduction of the prediction error when the feature under consideration is added to the set of all the features used for regression. Instead of calculating the LOCO using all the features at hand, as in its standard version, our method searches for the multiplet of features that maximize LOCO and for the one that minimize it. This provides a decomposition of the LOCO as the sum of a two-body component and higher-order components (redundant and synergistic), also highlighting the features that contribute to building these high-order effects alongside the driving feature. We report the application to proton/pion discrimination from simulated detector measures by GEANT.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[13]
S. Stramaglia, L. Faes, J.M. Cortes, and D. Marinazzo, Phys. Rev. Res. 6, L032007 (2024)
work page 2024
-
[22]
G. K¨ onig, E. G¨ unther and U. von Luxburg, Disentangling Interactions and Dependencies in Feature Attribution, preprint arXiv:2410.23772 (2024). 13
arXiv 2024
- [11]
-
[1]
S. Ali, T. Abuhmed, S. El-Sappagh, K. Muhammad, J.M. Alonso-Moral, R. Confalonieri, R. Guidotti, J. Del Ser, N. D ´ ıaz-Rodr ´ ıguez, and F. Herrera,Information Fusion 99, 1566 (2023)
work page 2023
-
[2]
G. Casalicchio, C. Molnar, and B. Bischl, Lecture Notes in Computer Science 1105, 655 (2019)
work page 2019
-
[3]
J. Lei, M. G’sell, A. Rinaldo, R.J. Tibshirani, and L. Wasserman, J. Am. Stat. Ass. 113, 1094 (2018)
work page 2018
- [4]
-
[5]
F. Battiston, E. Amico, A. Barrat, G. Bianconi, G. Ferraz, Arruda, B. Franceschiello, I. Iacopini, S. K´ efi, V. Latora, Y. Moreno, M. Murray, T. Peixoto, F. Vaccarino, and G. Petri, Nat. Phys. 17, 1093 (2021)
work page 2021
Show all 22 references
-
[6]
P. L. Williams and R. D. Beer, arXiv preprint arXiv:1004.2515 (2010)
2010 arXiv
-
[7]
J. T. Lizier, N. Bertschinger, J. Jost, and M. Wibral, Entropy 20, 307 (2018)
2018
-
[8]
Angelini, M
L. Angelini, M. de Tommaso, D. Marinazzo, L. Nitti, M. Pellicoro, and S. Stramaglia, Phys. Rev. E 81, 037201 (2010). 12
2010
-
[9]
L. Faes, D. Marinazzo, S. Stramaglia, F. Jurysta, A. Porta, Nollo G., Philos Trans A Math Phys Eng Sci. 374, 2067 (2016)
2016
-
[10]
Stramaglia, L
S. Stramaglia, L. Angelini, G. Wu, J.M. Cortes, L. Faes, D. Marinazzo, IEEE Trans. Biomed. Eng. 63, 2518 (2016)
2016
-
[12]
Schreiber, Phys
T. Schreiber, Phys. Rev. Lett. 85, 461 (2000)
2000
-
[14]
In general it is preferable to estimate the mean squared error by cross validation, however when the number of samples is sufficiently high, like in the examples reported in this paper, evaluation on the training data may be used as well
-
[15]
Ancona and S
In particular, for all learning methods for regression whose risk minimizer remains unchanged when new variables, statistically independent of input and target variables, are added to the set of input variables; this property holds for linear models and for hypothesis spaces i...
2006
-
[16]
Conger, Educational and Psychological Measurement 34, 35 (1974)
A.J. Conger, Educational and Psychological Measurement 34, 35 (1974)
1974
-
[17]
Maasen and A.B
G.H. Maasen and A.B. Bakker, Sociological Methods & Research 30, 241 (2001)
2001
-
[18]
Shawe-Taylor and N
J. Shawe-Taylor and N. Cristianini, Kernel Methods For Pattern Analysis . Cambridge Uni- versity Press (2004)
2004
-
[19]
Cortez, A
P. Cortez, A. Cerdeira, F. Almeida, T. Matos, and J. Reis, Decision support systems 47, 4 (2009)
2009
-
[20]
https://geant4.web.cern.ch
-
[21]
https://www.kaggle.com/datasets/naharrison/particle-identification-from-detector-responses
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.