Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Adversarial Dependence Minimization

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ADM drives learned embedding dimensions to mutual independence by training small predictors to reconstruct each dimension from the others and an encoder to defeat them.

desk verdict Novel adversarial algorithm for dependence reduction, but the mutual-independence guarantee doesn't hold as stated; deserves a serious look with a corrected claim. read the letter →

arxiv 2502.03227 v2 pith:SHOWOAES submitted 2025-02-05 cs.LG cs.CV

Pierre-François De Plaen , Tinne Tuytelaars , Marc Proesmans , Luc Van Gool This is my paper · ORCID
classification cs.LGcs.CV
keywords adversarialdependenceminimizationstatisticalindependencerepresentationlearningnonlineardecorrelationdimensionalcollapseself-superviseddistancecorrelationindependentcomponentanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Standard decorrelation methods remove linear covariance, but zero correlation does not make dimensions statistically independent: a dimension can be a deterministic nonlinear function of another while having zero Pearson correlation. This paper introduces ADMin, a differentiable adversarial algorithm that targets full statistical dependence between embedding dimensions rather than just linear dependence. It trains small predictor networks to reconstruct each standardized dimension from all the others, while an encoder is trained to defeat them by making dimensions impossible to reconstruct. The paper claims that at the global optimum of this game the predictors can do no better than predicting the mean, the average reconstruction error equals the unit variance, and the dimensions are mutually independent. If that holds, ADMin provides a plug-in regularizer for representation learning across PCA-style analysis, classification, and self-supervised learning.

What carries the argument

The load-bearing mechanism is the adversarial game between the encoder $f_\theta$ and per-dimension dependency predictors $h_{\phi_i}$: each predictor takes the other $d-1$ dimensions as input and tries to reconstruct dimension $i$, while the encoder tries to maximize the squared reconstruction error. Standardization of each dimension to zero mean and unit variance, estimated from the current mini-batch as in batch normalization, makes the game bounded and fixes the mean predictor's expected error at variance one, so the encoder's objective is equivalent to minimizing $1 - \mathbb{E}\lVert z - \hat{z}\rVert_2^2$. The paper argues that this turns the hard constraint of statistical independence into a differentiable min-max loss that can be added to any task loss with a Lagrangian weight $\lambda$, and notes that all predictors can be executed together as one grouped-convolution network for GPU efficiency.

What would settle it

Train ADMin on synthetic data where one dimension is a deliberately hard-to-approximate nonlinear function of another, such as a high-frequency sine, using the paper's default two-layer MLP predictors with 32 hidden units, and monitor distance correlation between the dimensions throughout training. If the reconstruction error converges to 1 while the squared distance correlation stays well above zero, the claimed equilibrium does not correspond to statistical independence.

Watch

Extended reading notes

Core claim

The central claim is that a min-max reconstruction game over standardized representations converges to statistically independent embedding dimensions. The objective is $\min_{\phi}\max_{\theta} \mathbb{E}_{z \sim P(X;\theta)}\lVert z - \hat{z}\rVert_2^2$, where $\hat{z}_i = h_{\phi_i}(z_{-i})$ is the prediction of dimension $i$ from the other $d-1$ dimensions and $z$ is standardized per dimension to zero mean and unit variance. Proposition 4.1 states that the game converges, under standardized representations, to a solution where dependence among the dimensions is minimal, the dependency predictors predict the mean (zero) vector, and the average reconstruction error equals the unit variance. The reasoning is that predicting the mean is the best any predictor can do for independent standardized dimensions and yields expected squared error exactly one, so forcing the reconstruction error to its maximum pushes the encoder to render dimensions mutually independent. The paper treats this objective as a Lagrangian relaxation of an independence constraint and demonstrates it in three settings: nonlinear decorrelation (PICA), supervised classification with a margin-based variant, and self-supervised learning.

Load-bearing premise

The load-bearing premise is that the small dependency-predictor networks have enough capacity and are trained well enough to detect every dependence that exists among the representation dimensions; if a dependence is too complex for them, the encoder can reach unit reconstruction error while the dimensions remain dependent.

Editorial extensions

If this is right

  • PCA can be extended to nonlinear decorrelation: replacing the covariance constraint with ADMin yields PICA, which in Example 3 recovers both true latent factors $[5v_1, v_2]$ instead of the redundant PCA pair $[x_1, x_2]$.
  • In supervised classification, adding the margin-formulated ADMin loss to cross-entropy on Clevr-4 dropped the mean squared distance correlation of penultimate features from 0.409 to 0.067 and raised kNN accuracy on the unlabeled texture and color taxonomies from 25.0 and 16.4 percent to 83.7 and 100 percent.
  • In self-supervised learning, invariance plus ADMin prevents dimensional collapse: on ImageNet the average squared distance correlation reached 0.015, and a linear classifier on the frozen ResNet-50 backbone reached 63.2 percent top-1 accuracy.
  • Because the adversarial loss is a soft constraint, any task objective can be augmented with it, giving a tunable trade-off between downstream performance and statistical independence of the representation dimensions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The equilibrium is defined relative to the capacity of the dependency predictors: a fair extrapolation is that ADMin enforces independence as judged by the predictor family, so a capacity-ladder test should show residual distance correlation shrinking only as predictor capacity grows.
  • The same regularizer could be applied to VAE latents or to feature extractors used for out-of-distribution detection, where dependence among dimensions is suspected to degrade calibration; this is an extension the paper does not test.
  • The margin sweep on Clevr-4 suggests a practical tuning rule for the relaxed formulation: choose the largest margin at which downstream accuracy is preserved, since accuracy rose with margin up to 0.4 and fell for larger margins.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces ADMin, an adversarial procedure in which small dependency predictors are trained to reconstruct each standardized embedding dimension from the remaining dimensions while the encoder is trained to maximize the resulting reconstruction error. The central theoretical claim (Proposition 4.1, Section 4.1) is that this game converges to a representation whose dimensions are minimally dependent, with the dependency predictors reduced to predicting the zero vector and the reconstruction error equal to the unit variance. The authors position the method as a nonlinear, mutual-dependence generalization of covariance-based decorrelation and illustrate it on nonlinear PCA (PICA), classifier regularization, and self-supervised learning, with experiments on Clevr-4 and ImageNet.

Significance. If the central claim were correct, the method would provide a scalable tool for enforcing statistical independence beyond pairwise linear decorrelation, which would be a valuable contribution to representation learning and self-supervised learning. The paper is clearly written, the experimental effort is substantial (including Clevr-4 and ImageNet linear evaluation), and the authors are candid about the ImageNet performance gap. However, the main theoretical claim is not established and is in fact false for the stated objective: the objective enforces zero conditional mean, not mutual independence. The reported convergence evidence is also partly self-referential, since an MSE of 1 is exactly the encoder's training target. With the claim weakened to conditional-mean independence, the paper could still have empirical value, but as submitted the headline contribution is unsupported.

major comments (4)
  1. [§4.1, Proposition 4.1 and Eq. (3)] Proposition 4.1 is false as stated. With MSE predictors, the minimax objective of Eq. (3) attains value 1 as soon as each coordinate is conditionally mean-independent of the others, i.e., E[z_i | z_-i] = 0; it cannot detect dependence that appears only in conditional variances or higher moments. Concrete counterexample: take z1 ~ N(0,1) and u ~ N(0,1) independent, and set z2 = z1 * u. Then z1 and z2 are already standardized (zero mean, unit variance), E[z2 | z1] = 0 and E[z1 | z2] = 0, so the optimal dependency predictors predict the zero vector and the reconstruction error is exactly 1, the maximum possible under standardization. Yet z1 and z2 are not independent: their distance correlation and mutual information are positive. Thus the global optimum of Eq. (3) can contain dependence even with exact conditional-mean predictors and unlimited capacity. This invalidates the abstract's assertion that mutual independence is achieved at the global optimum and the corresponding claims in Sections 4 and 5.
  2. [§4.1, "intuitive justification"] No convergence proof is supplied. The paragraph after Proposition 4.1 only shows that predicting the mean yields an upper bound of one on the reconstruction error and that detectable dependence could lower the loss; it does not prove that the alternating optimization reaches this bound, nor that every point at which the bound is attained has minimal dependence. The proposition also uses the word "converges" without a precise equilibrium notion (fixed point of minimax dynamics, Nash equilibrium, or otherwise). The empirical verification in §6.1 cannot substitute for the missing argument because it is necessarily relative to the chosen predictor architectures.
  3. [§5, Eq. (5) and Appendix G] Equation (5) is presented as a Lagrangian relaxation of the mutual-independence constraint (4), but the equivalence holds only if the dependency predictors can represent and train to the true conditional means. The footnote in Section 5 acknowledges the capacity assumption, and the default predictors are two-layer MLPs with hidden size 32 (Appendix G). The empirical distance-correlation values in Section 6.1 and Table 1 therefore measure dependence detectable by that limited family, not dependence per se. Moreover, no confidence intervals, repeated-seed statistics, or significance tests are reported for the distance-correlation estimates, so the difference between, e.g., 5.7e-4 and 2.9e-3 in Section 6.1 is not assessed statistically.
  4. [§6.1, Figure 3] The convergence evidence is in part self-referential. The left panel of Figure 3 reports that the reconstruction MSE reaches 1, but 1 is exactly the value the encoder is trained to maximize for standardized representations; reaching it mainly confirms that the optimization found its target, not that the representation is independent. The informative quantity is the external distance correlation, but it is reported only as a point estimate without error bars, without a baseline comparison, and without stating how many seeds were averaged. The claim that the algorithm "systematically converges" to minimal dependence is therefore not supported by the experiments as reported.
minor comments (5)
  1. [§4.2 and §6.2] The margin objective in Eq. (8) is written with the squared L2 reconstruction loss, but Section 6.2 and Appendix G.2 state that the classification experiments used an l1 margin loss; please define the l1 variant explicitly and specify which loss is used in every experiment.
  2. [Table 1] The heading "mean R2" is used for squared distance correlation, which is easily confused with the coefficient of determination; the table caption should define this notation.
  3. [Figure 3] The caption should state the dataset split (training or validation) and clarify which curves correspond to which dependency-predictor architecture; currently the reader must infer this from the main text.
  4. [Appendix B.1, Table 3] The ablation comparing the standardized and margin formulations also changes the reconstruction loss (MSE versus l1), so the comparison conflates the standardization choice with the loss choice; disentangling these factors would make the ablation more informative.
  5. [Eq. (5)] The text says lambda is a Lagrange multiplier, but for a penalty formulation it should be restricted to positive values; negative lambda would encourage dependence rather than discourage it.

Circularity Check

2 steps flagged · score 6.0 of 10

The central 'minimal dependence' conclusion is tied by definition to the adversarial equilibrium, and the headline convergence metric is the encoder's own objective; external distance-correlation estimates keep the circularity partial.

  1. self definitional [Section 5 (footnote 1), used to justify Section 4.1 Proposition 4.1]
    "Assuming that dependency predictors have enough capacity and that the dependence is minimal when dependency predictors are fooled."

    The relaxation of the independence constraint in Eq. 5 is justified by this footnote, which equates 'dependence is minimal' with the adversarial equilibrium in which the dependency predictors are fooled. Because the game in Eq. 3 is constructed precisely to fool the predictors into predicting the mean, the proposition's conclusion that the game converges to minimal dependence follows from this inserted definition rather than from the stated independence condition in Eq. 1. The supplied justification only shows that a mean-predicting network attains MSE equal to the unit variance; it does not derive the equivalence between that equilibrium and mutual independence, so the load-bearing step is circular.

  2. other [Section 4.1 and Section 6.1]
    "This loss is equal to zero when the dependency predictors can do no better than predicting the mean since the average expected error is then equal to one. ... the reconstruction error converges to a value of one for both dependency networks."

    The convergence metric reported in Section 6.1, MSE equal to one, is the exact quantity optimized by the encoder: Eq. 3 is min_phi max_theta E||z - zhat||^2, and Section 4.1 rewrites the encoder objective as minimizing 1 - E||z - zhat||^2. Observing that the reconstruction error converges to one is therefore a check that the optimization reached its own target, not independent evidence of the proposition's mutual-independence claim. The paper's distance-correlation estimates are external to the objective and provide separate support, which makes this partial rather than total circularity.

full rationale

The paper does not rely on self-citations or imported uniqueness theorems, and its experimental comparisons against external benchmarks and distance-correlation estimates provide independent evidence that the method reduces at least some nonlinear dependence. However, the theoretical derivation chain contains a definitional collapse: the claim that the Eq. 3 game converges to minimal dependence is supported by a footnote that defines minimal dependence as the state where dependency predictors are fooled, which is the very state the game is designed to produce. The proposition's formal content reduces to the encoder's objective when the reconstruction error reaches the standardized variance, and that same quantity is then used as the principal convergence metric. The gap is not merely a missing proof; the identification of 'predictors can do no better than the mean' with 'mutually independent dimensions' is assumed rather than derived, and is in fact false as stated since zero conditional mean does not imply independence. Because external distance-correlation measurements and downstream task evaluations are not defined in terms of the adversarial loss, the circularity is partial, not complete.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the capacity of the dependency predictors, the representational power of the encoder, and an unproved convergence statement for the adversarial game. The only hand-chosen parameters that materially affect the results are the task weight lambda and the margin alpha, plus the predictor hidden dimension; none of these are fitted to data in the sense of a derived constant, but they are tuned per experiment.

free parameters (3)
  • lambda (task weight) = 5 in Clevr-4 classification and SSL experiments
    Multiplies the adversarial loss in Equation 5; chosen by hand and affects the trade-off between task loss and dependence minimization.
  • alpha (margin) = 0.4 for margin-loss experiments
    Defines the reconstruction-error threshold up to which dependence is tolerated in the margin formulation (Section 4.2); tuned using the validation set (Appendix B, Figure 5).
  • dependency predictor hidden dimension = 32
    Capacity of the small MLPs that detect dependence; the theoretical claim assumes enough capacity, but the experiments use hidden size 32, which limits detectable dependence.
assumptions (4)
  • standard math MLPs are universal approximators, so dependency predictors can in theory approximate any relation among dimensions.
    Invoked in Section 4 to justify the ability of predictors to capture arbitrary dependencies.
  • ad hoc to paper The adversarial game reaches its global optimum under standardized representations.
    Proposition 4.1 asserts convergence to a solution where predictors predict the mean; no rigorous proof is provided.
  • domain assumption The encoder can represent a distribution over representations with mutually independent dimensions.
    Needed for the global optimum to be attainable; not stated or justified in the paper.
  • standard math Zero distance correlation is equivalent to statistical independence.
    Used in Section 6 to measure residual dependence; follows from Szekely et al. (2007).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Adversarial Dependence Minimization." pith.science (2026). https://pith.science/paper/SHOWOAES

@misc{pith2026250203227,
  author       = {Pith},
  title        = {Pith review of: Adversarial Dependence Minimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SHOWOAES}},
  note         = {Machine review of arXiv:2502.03227}
}
read the original abstract

Minimally redundant representations are typically learned by minimizing feature covariance. However, covariance-based methods fail to eliminate all dependencies/redundancies, as linearly uncorrelated variables can still exhibit nonlinear relationships. To address this, we introduce ADM, a differentiable algorithm that minimizes statistical dependence between feature dimensions through an adversarial game: auxiliary networks identify dependencies, while the encoder removes them. We prove that mutual independence is achieved at the global optimum, empirically verify convergence, and study three potential applications: extending PCA to nonlinear decorrelation, improving generalization in image classification, and preventing dimensional collapse in self-supervised learning. By promoting statistically independent representations, ADM paves the way for learning more robust, compressed, and generalizable representations across diverse applications.

Figures

Figures reproduced from arXiv: 2502.03227 by the authors.

Figure 1
Figure 1. Illustration of the joint and marginal distributions for different types of dependencies between random variables [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the adversarial dependence minimization architecture. The dependency predictors minimize the reconstruction error by learning how dimensions relate, while the encoder maximizes the error by reducing dependencies. tive as a two-player game where the encoder and depen￾dency predictors are respectively trained to maximize and minimize the expected reconstruction error: min ϕ max θ Ez∼P (X ;θ)∥z − zˆ∥ 2 … view at source ↗
Figure 3
Figure 3. Convergence analysis on TinyImageNet for lin [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Learned representations z. The colors indicate the value of the original latent factors v1 (left) and v2 (right). 13 [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Influence of the loss margin on the correlation and accuracy for classification models trained on the Clevr [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Illustration of Example 4: the random variables x1, x2 and x3 are all mutually dependent despite being all pairwise independent. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Nearest neighbors visualization for SSL models trained on the Clevr-4 dataset (left) and ImageNet dataset (right). [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Illustration of the different applications. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Self-Supervised Learning with a Multi-Task Latent Space Objective

    cs.CV 2026-02 conditional novelty 6.0 of 10

    Assigning a dedicated predictor to each view type stabilizes multi-crop Siamese SSL and, combined with asymmetric cutout views, yields consistent ImageNet gains over BYOL, SimSiam, and MoCo v3.

Reference graph

Works this paper leans on

71 extracted references · 29 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Back, A. D. and Weigend, A. S. A first application of independent component analysis to extracting structure from stock returns. International journal of neural systems, 8 0 (04): 0 473--484, 1997

    work page 1997

  3. [3]

    and Hornik, K

    Baldi, P. and Hornik, K. Neural networks and principal component analysis: Learning from examples without local minima. Neural networks, 2 0 (1): 0 53--58, 1989

    work page 1989

  4. [4]

    Vicreg: Variance-invariance-covariance regularization for self-supervised learning

    Bardes, A., Ponce, J., and LeCun, Y. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. In International Conference on Learning Representations, 2021

    work page 2021

  5. [5]

    I., Baratin, A., Rajeshwar, S., Ozair, S., Bengio, Y., Courville, A., and Hjelm, D

    Belghazi, M. I., Baratin, A., Rajeshwar, S., Ozair, S., Bengio, Y., Courville, A., and Hjelm, D. Mutual information neural estimation. In International conference on machine learning, pp.\ 531--540. PMLR, 2018

    work page 2018

  6. [6]

    Bell, A. J. and Sejnowski, T. J. An information-maximization approach to blind separation and blind deconvolution. Neural computation, 7 0 (6): 0 1129--1159, 1995

    work page 1995

  7. [7]

    Representation learning: A review and new perspectives

    Bengio, Y., Courville, A., and Vincent, P. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35 0 (8): 0 1798--1828, 2013

    2013

  8. [8]

    Guillotine regularization: Improving deep networks generalization by removing their head

    Bordes, F., Balestriero, R., Garrido, Q., Bardes, A., and Vincent, P. Guillotine regularization: Improving deep networks generalization by removing their head. arXiv preprint arXiv:2206.13378, 13, 2022

    arXiv 2022

Show all 71 references
  1. [9]

    and Bengio, Y

    Brakel, P. and Bengio, Y. Learning independent features with adversarial nets for non-linear ica. arXiv preprint arXiv:1710.05050, 2017

    2017 arXiv

  2. [10]

    Function classes for identifiable nonlinear independent component analysis

    Buchholz, S., Besserve, M., and Sch \"o lkopf, B. Function classes for identifiable nonlinear independent component analysis. Advances in Neural Information Processing Systems, 35: 0 16946--16961, 2022

    2022

  3. [11]

    P., Higgins, I., Pal, A., Matthey, L., Watters, N., Desjardins, G., and Lerchner, A

    Burgess, C. P., Higgins, I., Pal, A., Matthey, L., Watters, N., Desjardins, G., and Lerchner, A. Understanding disentangling in -vae. arXiv preprint arXiv:1804.03599, 2018

    2018 arXiv

  4. [12]

    Emerging properties in self-supervised vision transformers

    Caron, M., Touvron, H., Misra, I., J \'e gou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 9650--9660, 2021

    2021

  5. [13]

    T., Li, X., Grosse, R

    Chen, R. T., Li, X., Grosse, R. B., and Duvenaud, D. K. Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems, 31, 2018

    2018

  6. [14]

    A simple framework for contrastive learning of visual representations

    Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp.\ 1597--1607. PMLR, 2020

    2020

  7. [15]

    Infogan: Interpretable representation learning by information maximizing generative adversarial nets

    Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., and Abbeel, P. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016

    2016

  8. [16]

    Support-vector networks

    Cortes, C. Support-vector networks. Machine Learning, 1995

    1995

  9. [17]

    Approximation by superpositions of a sigmoidal function

    Cybenko, G. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 2 0 (4): 0 303--314, 1989

    1989

  10. [18]

    Analyse des liaisons de probabilit \'e

    Darmois, G. Analyse des liaisons de probabilit \'e . In Proc. Int. Stat. Conferences 1947, pp.\ 231, 1951

    1947

  11. [19]

    Imagenet: A large-scale hierarchical image database

    Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.\ 248--255. IEEE, 2009

    2009

  12. [20]

    Driscoll, M. F. On pairwise and mutual independence: characterizations of rectangular distributions. Journal of the American Statistical Association, 73 0 (362): 0 432--433, 1978

    1978

  13. [21]

    Whitening for self-supervised representation learning

    Ermolov, A., Siarohin, A., Sangineto, E., and Sebe, N. Whitening for self-supervised representation learning. In International conference on machine learning, pp.\ 3015--3024. PMLR, 2021

    2021

  14. [22]

    Generative adversarial nets

    Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. Advances in neural information processing systems, 27, 2014

    2014

  15. [23]

    Accurate, large minibatch sgd: training imagenet in 1 hour

    Goyal, P. Accurate, large minibatch sgd: training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017

    2017 arXiv

  16. [24]

    Bootstrap your own latent-a new approach to self-supervised learning

    Grill, J.-B., Strub, F., Altch \'e , F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33: ...

    2020

  17. [25]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 770--778, 2016

    2016

  18. [26]

    Momentum contrast for unsupervised visual representation learning

    He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 9729--9738, 2020

    2020

  19. [27]

    and Gimpel, K

    Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016

    2016 arXiv

  20. [28]

    beta-vae: Learning basic visual concepts with a constrained variational framework

    Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. beta-vae: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations, 2017

    2017

  21. [29]

    Hinton, G. E. and Salakhutdinov, R. R. Reducing the dimensionality of data with neural networks. science, 313 0 (5786): 0 504--507, 2006

    2006

  22. [30]

    E., Osindero, S., and Teh, Y.-W

    Hinton, G. E., Osindero, S., and Teh, Y.-W. A fast learning algorithm for deep belief nets. Neural computation, 18 0 (7): 0 1527--1554, 2006

    2006

  23. [31]

    D., Fedorov, A., Lavoie-Marchildon, S., Grewal, K., Bachman, P., Trischler, A., and Bengio, Y

    Hjelm, R. D., Fedorov, A., Lavoie-Marchildon, S., Grewal, K., Bachman, P., Trischler, A., and Bengio, Y. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2019

    2019

  24. [32]

    Multilayer feedforward networks are universal approximators

    Hornik, K., Stinchcombe, M., and White, H. Multilayer feedforward networks are universal approximators. Neural networks, 2 0 (5): 0 359--366, 1989

    1989

  25. [33]

    Analysis of a complex of statistical variables into principal components

    Hotelling, H. Analysis of a complex of statistical variables into principal components. Journal of educational psychology, 24 0 (6): 0 417, 1933

    1933

  26. [34]

    On feature decorrelation in self-supervised learning

    Hua, T., Wang, W., Xue, Z., Ren, S., Wang, Y., and Zhao, H. On feature decorrelation in self-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 9598--9608, 2021

    2021

  27. [35]

    Decorrelated batch normalization

    Huang, L., Yang, D., Lang, B., and Deng, J. Decorrelated batch normalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.\ 791--800, 2018

    2018

  28. [36]

    and Morioka, H

    Hyvarinen, A. and Morioka, H. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances in neural information processing systems, 29, 2016

    2016

  29. [37]

    and Oja, E

    Hyv \"a rinen, A. and Oja, E. Independent component analysis: algorithms and applications. Neural networks, 13 0 (4-5): 0 411--430, 2000

    2000

  30. [38]

    and Szegedy, C

    Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp.\ 448--456. pmlr, 2015

    2015

  31. [39]

    Understanding dimensional collapse in contrastive self-supervised learning

    Jing, L., Vincent, P., LeCun, Y., and Tian, Y. Understanding dimensional collapse in contrastive self-supervised learning. In International Conference on Learning Representations, 2022

    2022

  32. [40]

    Clevr: A diagnostic dataset for compositional language and elementary visual reasoning

    Johnson, J., Hariharan, B., Van Der Maaten, L., Fei-Fei, L., Lawrence Zitnick, C., and Girshick, R. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 29...

    2017

  33. [41]

    and Herault, J

    Jutten, C. and Herault, J. Blind separation of sources, part i: An adaptive algorithm based on neuromimetic architecture. Signal processing, 24 0 (1): 0 1--10, 1991

    1991

  34. [42]

    and Karhunen, J

    Jutten, C. and Karhunen, J. Advances in blind source separation (bss) and independent component analysis (ica) for nonlinear mixtures. International journal of neural systems, 14: 0 267--92, 11 2004

    2004

  35. [43]

    Variational autoencoders and nonlinear ica: A unifying framework

    Khemakhem, I., Kingma, D., Monti, R., and Hyvarinen, A. Variational autoencoders and nonlinear ica: A unifying framework. In International conference on artificial intelligence and statistics, pp.\ 2207--2217. PMLR, 2020

    2020

  36. [44]

    and Mnih, A

    Kim, H. and Mnih, A. Disentangling by factorising. In International conference on machine learning, pp.\ 2649--2658. PMLR, 2018

    2018

  37. [45]

    Kingma, D. P. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013

    2013 arXiv

  38. [46]

    Kramer, M. A. Nonlinear principal component analysis using autoassociative neural networks. AIChE journal, 37 0 (2): 0 233--243, 1991

    1991

  39. [47]

    Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012

    2012

  40. [48]

    and Yang, X

    Le, Y. and Yang, X. Tiny imagenet visual recognition challenge. CS 231N, 7 0 (7): 0 3, 2015

    2015

  41. [49]

    Y., Pinkus, A., and Schocken, S

    Leshno, M., Lin, V. Y., Pinkus, A., and Schocken, S. Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural networks, 6 0 (6): 0 861--867, 1993

    1993

  42. [50]

    Learning disentangled representation with pairwise independence

    Li, Z., Tang, Y., Li, W., and He, Y. Learning disentangled representation with pairwise independence. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp.\ 4245--4252, 2019

    2019

  43. [51]

    An application of the principle of maximum information preservation to linear systems

    Linsker, R. An application of the principle of maximum information preservation to linear systems. Advances in neural information processing systems, 1, 1988

    1988

  44. [52]

    Challenging common assumptions in the unsupervised learning of disentangled representations

    Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Sch \"o lkopf, B., and Bachem, O. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pp.\ 4114--4124. PMLR, 2019

    2019

  45. [53]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016

    2016 arXiv

  46. [54]

    Variance covariance regularization enforces pairwise independence in self-supervised representations

    Mialon, G., Balestriero, R., and LeCun, Y. Variance covariance regularization enforces pairwise independence in self-supervised representations. arXiv preprint arXiv:2209.14905, 2022

    2022 arXiv

  47. [55]

    Pearson, K. Liii. on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin philosophical magazine and journal of science, 2 0 (11): 0 559--572, 1901

    1901

  48. [56]

    Scikit-learn: Machine learning in P ython

    Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E. Scikit-learn: Machine learning in P ython. Journal of ...

    2011

  49. [57]

    and Torrecilla, J

    Ramos-Carreño, C. and Torrecilla, J. L. dcor: Distance correlation and energy statistics in Python . SoftwareX, 22, 2 2023. doi:10.1016/j.softx.2023.101326. URL https://www.sciencedirect.com/science/article/pii/S2352711023000225

    2023

  50. [58]

    E., Hinton, G

    Rumelhart, D. E., Hinton, G. E., and Williams, R. J. Learning internal representations by error propagation, parallel distributed processing, explorations in the microstructure of cognition, ed. de rumelhart and j. mcclelland. vol. 1. 1986. Biometrika, 71 0 (599-607): 0 6, 1986 a

    1986

  51. [59]

    E., Hinton, G

    Rumelhart, D. E., Hinton, G. E., and Williams, R. J. Learning representations by back-propagating errors. nature, 323 0 (6088): 0 533--536, 1986 b

    1986

  52. [60]

    J., Rizzo, M

    Sz \'e kely, G. J., Rizzo, M. L., and Bakirov, N. K. Measuring and testing dependence by correlation of distances. The Annals of Statistics, 35 0 (6): 0 2769, 2007

    2007

  53. [61]

    and Jutten, C

    Taleb, A. and Jutten, C. Source separation in post-nonlinear mixtures. IEEE Transactions on signal Processing, 47 0 (10): 0 2807--2820, 1999

    1999

  54. [62]

    H., Pascanu, R., Blundell, C., and Mitrovic, J

    Tomasev, N., Bica, I., McWilliams, B., Buesing, L. H., Pascanu, R., Blundell, C., and Mitrovic, J. Pushing the limits of self-supervised resnets: Can we outperform supervised learning without labels on imagenet? In First Workshop on Pre-training: Perspectives, Pitfalls, and Pa...

    2022

  55. [63]

    Large margin methods for structured and interdependent output variables

    Tsochantaridis, I., Joachims, T., Hofmann, T., Altun, Y., and Singer, Y. Large margin methods for structured and interdependent output variables. Journal of machine learning research, 6 0 (9), 2005

    2005

  56. [64]

    No representation rules them all in category discovery

    Vaze, S., Vedaldi, A., and Zisserman, A. No representation rules them all in category discovery. Advances in Neural Information Processing Systems, 36, 2024

    2024

  57. [65]

    X., and Lin, D

    Wu, Z., Xiong, Y., Yu, S. X., and Lin, D. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 3733--3742, 2018

    2018

  58. [66]

    Independent principal component analysis for biologically meaningful dimension reduction of large biological data sets

    Yao, F., Coquery, J., and L \^e Cao, K.-A. Independent principal component analysis for biologically meaningful dimension reduction of large biological data sets. BMC bioinformatics, 13: 0 1--15, 2012

    2012

  59. [67]

    Large batch training of convolutional networks

    You, Y., Gitman, I., and Ginsburg, B. Large batch training of convolutional networks. arXiv preprint arXiv:1708.03888, 2017

    2017 arXiv

  60. [68]

    Barlow twins: Self-supervised learning via redundancy reduction

    Zbontar, J., Jing, L., Misra, I., LeCun, Y., and Deny, S. Barlow twins: Self-supervised learning via redundancy reduction. In International conference on machine learning, pp.\ 12310--12320. PMLR, 2021

    2021

  61. [69]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  62. [70]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  63. [71]

    " F 0 @P!1 234 `p

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.