REVIEW 5 major objections 6 minor 12 references
Information Subtraction: Learning Representations for Conditional Entropy
T0 review · 5 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read This paper claims that a generator trained to maximize $I(Y;X,Z)$ while minimizing $I(X;Z)$ learns a representation of the conditional entropy $H(Y|X)$ in continuous variables, and that iterating the procedure decomposes multivariate…
desk verdict The paper's main empirical evidence is impossible—Table 2 reports I(Z;X) exceeding H(X) for a 3-level categorical variable—so the central claim collapses, despite an interesting conceptual framing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is a three-network architecture with one generator and two discriminators. The generator $N_A$ maps the target $Y$ to the representation $Z$. Discriminator $N_C$ estimates $I(Y;X,Z)$ through the Donsker–Varadhan representation, and discriminator $N_D$ estimates $I(X;Z)$; both are implemented as MINE/SMILE neural estimators, which make the information terms differentiable with respect to $Z$ and therefore trainable by backpropagation. Training first reconstructs $Y$ from $Z$ to initialize useful features and then updates the generator along $\nabla(\hat{I}(Y;X,Z)-\lambda \hat{I}(X;Z))$. Because the method only needs neural estimates of two mutual informations, it makes no distributional assumption on $X$, $Y$, or $Z$ and naturally handles continuous conditioning variables. The iterative extension reuses the same objective with different target/condition pairs, so the same machinery produces $Z_1,\ldots,Z_7$ for all sectors of a three-variable Venn diagram.
What would settle it
Compute the exact mutual informations $I(X;Z)$ and $I(Y;X,Z)$ on a small synthetic system where $H(Y|X)$ is known analytically, after training with the proposed objective; the claim stands only if $I(X;Z)$ is near zero and $I(Y;X,Z)$ approaches $H(Y)$. A compact check already lies in the paper's own synthetic fair-learning table, where the trained $Z$ has $I(Z;X)=2.98$ against $I(Z;Y)=0.19$, and in Appendix B where sector representations retain substantial conditional-variable information; reproducing those experiments with tighter estimators would settle whether the subtraction is real.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that conditional information can be generated rather than sampled: instead of selecting training samples that share the conditioning value, as conditional contrastive methods do, one can write down the objective $\max_{P_{Z|Y}} I(Y;X,Z) - \lambda I(X;Z)$ and train a generator against two mutual-information discriminators to realize it. The paper claims the resulting $Z$ contains the information of $H(Y|X)$, and it reports evidence in a Lotka–Volterra ecosystem: with $I(S;G)=0.89$, $H(G)=3.28$, and $H(G|S)=2.39$, the learned representation achieves $I(Z;G)=2.06$, $I(Z;S)=0.28$, and $I(Z;G|S)=1.44$ bits. Iterating the procedure on three variables is claimed to decompose the joint information into seven mutually exclusive sectors—conditional entropies, conditional mutual informations, and the trivariate interaction—so that arbitrary information terms can be assembled by adding sectors. In the fair-learning experiment, subtracting the protected attribute $C$ from features $X$ yields $Z$ with near-zero mutual information to $C$, and prediction from $Z$ raises balanced accuracy from 0.628 to 0.810 while cutting the group gap roughly in half. In domain generalization, adding the subtracted representation $Z$ to original features $X$ raises test accuracy from 0.566 to 0.598 on an unseen region.
Load-bearing premise
The load-bearing premise is that training on the surrogate objective $I(Y;X,Z)-\lambda I(X;Z)$, using neural-network estimates of the two mutual informations and gradient descent, actually drives the true information $I(X;Z)$ to zero while preserving the true $I(Y;X,Z)$; if the estimator gradients diverge from the true information quantities, the learned code can be heavily contaminated by $X$ even when the training loss looks correct.
Editorial extensions
If this is right
- For any pair of continuous variables, this objective gives a trainable code that purports to isolate the information in $Y$ not carried by $X$, eliminating the need for conditional sampling or clustering.
- Iterating the subtraction decomposes a multivariate system's joint information into mutually exclusive sectors, so representations of conditional entropy, conditional mutual information, and higher-order interactions become obtainable from one architecture.
- Subtracting protected attributes from features before classification can produce substantially fairer predictors: the Adult experiment reports balanced accuracy rising from 0.628 to 0.810 and group-gap metrics dropping by more than half.
- In domain generalization, adding the subtracted 'universal' representation to the original domain-specific features can improve accuracy on an unseen domain, here from 0.566 to 0.598 on the held-out Cover Type region.
- If the representation truly equals $H(Y|X)$, the same code can serve as a semantic decomposition: in the predator-prey system it encodes peaks of grass population that sheep population alone cannot predict.
Reading between the lines
- Because the training objective is a weighted sum of two separately estimated information terms, the method's guarantee is only as good as the mutual-information estimators; an obvious next test is swapping SMILE for tighter estimators or for exact MI in small synthetic systems and checking whether the reported leakage shrinks.
- The iterative sector decomposition assumes that subtracting a conditioned code from an already-subtracted code yields an independent sector; enforcing orthogonality or statistical independence between the generated $Z_i$ seems a natural extension the paper leaves open.
- A testable application beyond the paper: use Information Subtraction on temporal data to isolate the part of one time series not explained by another, which would connect the method to Granger-causality and unobserved-confounding analyses.
- The fair-learning result suggests treating protected/discrete variables and continuous nuisance variables in one unified subtraction scheme; extending the current discrete protected-attribute experiments to continuous sensitive variables (e.g., age) would test the method's central promise.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Information Subtraction, a generative framework intended to learn representations Z of conditional entropy H(Y|X) (and, by iteration, other information-theoretic sectors of a Venn diagram) by simultaneously maximizing a neural estimate of I(Y;X,Z) and minimizing a neural estimate of I(X;Z). The method is demonstrated on a Lotka-Volterra synthetic system, a synthetic fair-learning setup, the Adult income dataset, and the Cover Type dataset, with claims that the learned representations carry semantic features of conditional entropy, improve fairness, and aid domain generalization. The central claim is that the framework can preserve desired information while eliminating undesired information, and can be iterated to represent arbitrary information components between continuous variables.
Significance. If valid, the paper would address a real gap: extending conditional-entropy representations beyond discrete conditioning variables and providing a compositional way to isolate information sectors. The authors release code, and the downstream fairness evaluation in Table 4 uses metrics that are independent of the neural mutual-information estimators, which is a strength. However, the core empirical validation is not sound: Table 2 reports impossible mutual-information values, Appendix B explicitly concedes that the framework cannot disentangle information sectors, and the evaluation of the conditional-entropy claim uses the same estimator family that appears in the training objective. These issues are load-bearing, so the paper's main contributions are not currently established.
major comments (5)
- [Section 5.3, Table 2, Eq. (13)] The synthetic fair-learning experiment has a three-level categorical variable X, so H(X) <= log2(3) approximately 1.585 bits. Table 2 reports I(Z;X) = 2.98 bits, which exceeds this information-theoretic upper bound. Moreover, Algorithm 1 generates Z as a function of Y only, so X -> Y -> Z is a Markov chain; the data-processing inequality then requires I(Z;X) <= I(Y;X), which Table 2 reports as 0.25 bits. The reported 2.98 bits therefore cannot be a true mutual information. This means the SMILE estimates used in this experiment are not valid measures of mutual information, and the claim that Z contains H(Y|X) while eliminating X is an estimator artifact rather than evidence. Because the same estimator family enters the training objective in Eq. (10), the optimized representation need not satisfy the claimed information properties.
- [Appendix B, Table B.2] Appendix B explicitly concedes that "the framework cannot effectively disentangle and represent the information within each individual sector." The numbers in Table B.2 support this: for the first sector, I(Z1;G|S,W) = 0.08 while the leakage I(Z1;S,W) = 1.14, and I(Z7;G|Z1,Z4,Z5) = 0.00. This contradicts the Section 5.2 claim that iteratively applying Information Subtraction can represent arbitrary sectors such as conditional mutual information and trivariate mutual information. This is not a minor implementation detail; the multivariate decomposition is one of the paper's stated contributions and is central to the general framework.
- [Section 4.3, Eq. (10); Tables 1-3] The success metrics used to verify the representation are estimated with the same family of neural mutual-information estimators (MINE/SMILE) as the training objective. For example, Table 2 reports I(Z;Y|X) = 2.76 as evidence that Z represents H(Y|X), but this is computed by the same type of estimator that the generator is trained against in Eq. (10). Without an independent ground-truth estimate for the synthetic Gaussian-mixture model in Eq. (13), or a non-neural estimator, the high reported conditional mutual information could reflect fitting the estimator rather than actual information content. The downstream fairness metrics in Table 4 are more independent, but they do not by themselves establish that Z is a representation of conditional entropy.
- [Section 2, Eq. (6); Appendix G] No theoretical guarantee is provided that a finite value of the trade-off parameter lambda in the Lagrangian relaxation Eq. (6) yields a representation with I(X;Z) = 0 or with I(Z;Y|X) = H(Y|X). Appendix G shows that I(X;Z) varies by orders of magnitude as lambda changes, but the paper gives no criterion for selecting lambda to satisfy the constraint, and the algorithm in Section 4.3 does not actually implement lambda (Algorithm 1 line 8 uses lambda = 1 implicitly). The claimed equivalence between the optimized objective and the target conditional-entropy sector is therefore not established.
- [Appendix D, Algorithm D.1] Algorithm D.1, which underpins the real-data fair-learning and domain-generalization results, is internally inconsistent. The inner loop samples (x,c), but line 12 sets z <- NA(y) with y undefined, and line 13 uses NC(x,y,z) even though the task is to debias X with respect to C; the outer loop correctly uses z <- NA(x). As written, the algorithm cannot be executed, so the Adult and Cover Type results are not reproducible from the pseudocode.
minor comments (6)
- [Table 3] Table 3 reproduces the column headers of Table 1 ("I(S;G)", "H(G)", "H(G|S)", etc.) even though the Adult experiment concerns X, C, and Z; the quantity reported in each column is therefore undefined.
- [Algorithm 1, Eq. (10)] Algorithm 1 line 8 minimizes I(X;Z) - I(Y;X,Z), which corresponds to lambda = 1, while Eq. (10) and Appendix G describe a tunable lambda; the sensitivity analysis is not connected to the algorithm as stated.
- [Section 5.1, text near Eq. (12)] The sentence beginning "The generated representation Z provides an additional 1.44 bits..." ends with "contain minimal information about G," but the preceding discussion and Table 1 concern information about S; this appears to be a typo for S.
- [Figure 4] The three panels of Figure 4 all share the caption label "(c)"; distinct letters are needed for the dynamics of S and G, Z and G, and Z and S.
- [Appendix A, Eq. (A.1)] Equation (A.1) writes the discrete increment as (1/Delta t) times the derivative, which is not the standard Euler update; with Delta t = 800 this would produce a very small increment, so this is presumably a typo for Delta t times the derivative.
- [Tables 4 and 5] The real-data results are reported without error bars or repeated-seed variation, and the fairness metrics are not accompanied by confidence intervals or significance tests.
Circularity Check
Central 'conditional entropy representation' claim is validated by the same SMILE estimator used as the training objective; Table 2's impossible I(Z;X)=2.98 for three-level X confirms the result is an estimator artifact rather than independent evidence.
-
fitted input called prediction
[Section 4.3, Eq. (10) and Algorithm 1; Section 5.3, Table 2 (also Tables 1 and 3)]
"max θ1∈Θ1,θ2∈Θ2,PZ|Y Iθ1(Y ;X,Z) − λIθ2(X;Z) (10) ... Table 2 demonstrates that the generated representation Z contains the information about H(Y |X) in this example."
The generator Z is trained by gradient descent on SMILE estimates ND(x,z) and NC(x,y,z), i.e. exactly the I(X;Z) and I(Y;X,Z) terms in Eq. (10). The paper then reports I(Z;Y|X)=2.76 and I(Z;X)=2.98 as evidence that Z represents H(Y|X) while eliminating X. These are the same quantities being optimized, not an independent ground-truth test; in this synthetic case X has only three levels, so H(X)≤log2(3)=1.585 bits and I(Z;X)=2.98 is impossible for true mutual information (the Markov chain X→Y→Z also forces I(X;Z)≤I(X;Y)=0.25). The claimed 'representation of conditional entropy' is therefore a fitted value of the estimator family used as the loss, not an external validation, and the reported elimination of sensitive information is an estimator artifact.
full rationale
The core conditional-information evaluation is entangled with the training objective. The architecture optimizes SMILE estimates of I(Y;X,Z) and I(X;Z) (Eq. 10, Algorithm 1 lines 8/13/14), and then Tables 1-3 'demonstrate' success using the same family of neural MI estimators. Since Iθ is only a lower bound, maximizing it can inflate the estimate without increasing true MI, so the experiment is partially circular: the reported I(Z;Y|X) and I(Z;X) are the very quantities being optimized. This is confirmed by Table 2, where the reported I(Z;X)=2.98 bits exceeds the entropy of three-level X (H(X)≤1.585 bits) and violates the data-processing inequality bound I(X;Z)≤I(X;Y)=0.25 bits. The paper's own Appendix B concedes that Z1, Z4, Z5, and Z7 'still retain some information about the conditional variables that should be eliminated' and that 'the framework cannot effectively disentangle' information sectors, undermining the 'arbitrary information components' claim. Some independent support exists: the downstream fair-learning (Table 4) and domain-generalization (Table 5) results use accuracy and fairness metrics rather than MI estimators, but they do not rescue the paper's central claim about representing conditional entropy. The self-citation to Leong et al. 2023 for Eq. (4) is a trivial information-theoretic identity and is not load-bearing. Overall, one or more central predictions reduce by construction to the fitted estimator; score 6.
Assumptions & free parameters
free parameters (1)
- lambda (trade-off hyperparameter) =
not specified (sensitivity analysis in Appendix G over roughly 1e-2 to 1e2)
assumptions (5)
- standard math Donsker-Varadhan representation gives a tight lower bound for mutual information (Eq. 7-9)
- standard math sup_Z I(Y;Z|X) = H(Y|X) (Eq. 1)
- domain assumption X provides only part of the information in Y (H(Y)>I(Y;X), H(Y|X)>0) and a deterministic map Z=f(Y) can represent the residual
- ad hoc to paper The Lagrangian relaxation in Eq. 6 is a valid trade-off for the constrained problem in Eq. 5
- ad hoc to paper The MINE/SMILE estimates are accurate enough that optimizing them yields representations with true conditional-entropy semantics
Cite this review
Pith. "Pith review of Information Subtraction: Learning Representations for Conditional Entropy." pith.science (2026). https://pith.science/paper/2OHVZ5EH
@misc{pith2026250102012,
author = {Pith},
title = {Pith review of: Information Subtraction: Learning Representations for Conditional Entropy},
year = {2026},
howpublished = {\url{https://pith.science/paper/2OHVZ5EH}},
note = {Machine review of arXiv:2501.02012}
}
read the original abstract
The representations of conditional entropy and conditional mutual information are significant in explaining the unique effects among variables. While previous studies based on conditional contrastive sampling have effectively removed information regarding discrete sensitive variables, they have not yet extended their scope to continuous cases. This paper introduces Information Subtraction, a framework designed to generate representations that preserve desired information while eliminating the undesired. We implement a generative-based architecture that outputs these representations by simultaneously maximizing an information term and minimizing another. With its flexibility in disentangling information, we can iteratively apply Information Subtraction to represent arbitrary information components between continuous variables, thereby explaining the various relationships that exist between them. Our results highlight the representations' ability to provide semantic features of conditional entropy. By subtracting sensitive and domain-specific information, our framework demonstrates effective performance in fair learning and domain generalization. The code for this paper is available at https://github.com/jh-liang/Information-Subtraction
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[4]
Learning deep representations by mutual information estimation and maximization
“Learning deep representations by mutual information estimation and maximization”. arXiv preprint arXiv:1808.06670. Hyvarinen, A. and H. Morioka. 2017, 20–22 Apr. “Nonlinear ICA of temporally dependent stationary sources”. In 20th International Conference on Artificial Intelligence and Statistics , edited by A. Singh and J. Zhu, V olume 54 of Proceedings ...
arXiv 2017
-
[6]
Towards nonlinear disentanglement in natural data with temporal sparse coding
“Towards nonlinear disentanglement in natural data with temporal sparse coding”. arXiv preprint arXiv:2007.10930. Leong, K.-H., Y . Xiu, B. Chen, and W. K. Chan
arXiv 2007
-
[10]
Learning temporally causal latent processes from general temporal data
“Learning temporally causal latent processes from general temporal data”. arXiv preprint arXiv:2110.05428. Zhang, K., Q. Wen, C. Zhang, R. Cai, M. Jin, Y . Liu, , , , et al
-
[12]
Learning bias-invariant representation by cross- sample mutual information minimization
“Learning bias-invariant representation by cross- sample mutual information minimization”. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 15002–15012. 14 A APPENDIX: IMPLEMENTA TION DETAILS FOR PART 5.1 REPRESENTA TION RELA TIONSHIPS The dynamics of the Lotka–V olterra model for the four species is sampled with discrete deriva...
work page 2022
-
[2013]
Auto-encoding variational bayes
“Auto-encoding variational bayes”. arXiv preprint arXiv:1312.6114. Klindt, D., L. Schott, Y . Sharma, I. Ustyuzhaninov, W. Brendel, M. Bethgeet al
-
[2015]
The variational fair autoencoder
“The variational fair autoencoder”. arXiv preprint arXiv:1511.00830. Ma, M. Q., Y .-H. H. Tsai, P. P. Liang, H. Zhao, K. Zhang, R. Salakhutdinovet al
-
[2018]
Mine: Mutual information neural estimation
“Mine: Mutual information neural estimation”. arXiv preprint arXiv:1801.04062. Blackard, J
-
[2020]
Representation Learning for Sequence Data with Deep Autoencoding Predictive Components
“Representation learning for sequence data with deep autoencoding predictive components”. arXiv preprint arXiv:2010.03135. Becker, B. and R. Kohavi
work page Pith review arXiv 2010
Show all 12 references
-
[2021]
Conditional Con- trastive Learning for Improving Fairness in Self-Supervised Learning
“Conditional Con- trastive Learning for Improving Fairness in Self-Supervised Learning”.arXiv preprint arXiv:2106.02866. Meng, R., T. Luo, and K. Bouchard
-
[2022]
Compressed predictive information coding
“Compressed predictive information coding”. arXiv preprint arXiv:2203.02051. Moyer, D., S. Gao, R. Brekelmans, A. Galstyan and G. Ver Steeg
-
[2023]
Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects
“Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects”. arXiv preprint arXiv:2306.10125. Zhao, H., C. Dan, B. Aragam, T. S. Jaakkola, G. J. Gordon and P. Ravikumar
-
[2024]
Adult. UCI machine learning repository (1996)
“Adult. UCI machine learning repository (1996)”. DOI: https://doi. org/10.24432/C5XW20, Accessed
1996 doi
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.