Pith. sign in

REVIEW 3 major objections 6 minor 12 references

Beyond Pairwise Correlations: Higher-Order Redundancies in Self-Supervised Representation Learning

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that SSLPM, a self-supervised method that minimizes redundancy by making masked features hard to predict, is competitive with state-of-the-art SSL methods, and that the best-performing methods exhibit low embedding…

desk verdict A solid SSL method paper with a useful redundancy hierarchy and a fixable proof error, wrapped in an over-strong interpretive story. read the letter →

arxiv 2412.01926 v2 pith:RO7HR7SF submitted 2024-12-02 cs.LG

David Zollikofer , Béni Egressy , Frederik Benzing , Matthias Otth , Roger Wattenhofer This is my paper · ORCID
classification cs.LG
keywords self-supervisedlearningredundancyreductionpredictabilityminimizationhigher-orderdependencieslinearnonlinearmaskedpredictionrepresentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper defines embedding-space redundancy beyond pairwise correlations: average absolute covariance (AAC), linear redundancy (LR), and nonlinear redundancy (NLR), and proves the hierarchy $AAC \le \sqrt{LR} \le \sqrt{NLR}$. On top of this, it introduces SSLPM, a self-supervised method in which a predictor tries to reconstruct randomly masked representation features and the encoder tries to make that prediction hard. SSLPM-RR, the variant with a closed-form ridge-regression predictor, reaches accuracy competitive with Barlow Twins and other leading SSL methods on CIFAR-10, CIFAR-100, and ImageNet-100. The paper's broader claim is that the best-performing SSL methods have low embedding redundancy even when they do not explicitly penalize redundancy, suggesting that high-quality representations reduce redundancy implicitly.

What carries the argument

The central objects are three redundancy measures computed on standardized embedding features: AAC is the mean absolute pairwise covariance; LR is $1$ minus the minimal squared error of a linear (ridge) regression predicting each feature from the others; NLR is $1$ minus the minimal squared error of an MLP predicting each feature from the others. The paper proves these form a hierarchy $AAC \le \sqrt{LR} \le \sqrt{NLR}$, with zero AAC equivalent to zero LR but not to zero NLR, illustrated by a Gaussian counterexample. SSLPM combines Barlow Twins' invariance loss with a masked-prediction loss: a Bernoulli mask hides roughly half the features of both augmented representations, a predictor reconstructs the hidden ones from the visible ones, and the encoder is updated to make reconstruction hard; SSLPM-RR solves the predictor in closed form with ridge regression.

What would settle it

Take the same set of trained SSL models and re-estimate LR and NLR using all 512 embedding features rather than a 20% subsample, with per-model capacity-tuned nonlinear estimators; if BYOL, VICReg, and NNCLR no longer have lower redundancy than SSLPM-RR, or if the redundancy-accuracy correlations reverse, the implicit-redundancy conclusion collapses.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that redundancy reduction can be lifted from pairwise covariance to general predictability: a representation has high redundancy when one or more of its features can be reconstructed from the rest, and SSLPM removes that redundancy by forcing an encoder to defeat a predictor trained on randomly masked features. The theoretical contribution is the inequality chain $AAC \le \sqrt{LR} \le \sqrt{NLR}$ for standardized features, with proofs that linear and nonlinear predictability capture multi-feature and nonlinear interactions that pairwise covariance can miss, while corollaries show zero AAC is equivalent to zero LR but not to zero NLR. Empirically, SSLPM-RR attains 92.70 Top-1 on CIFAR-10, 70.46 on CIFAR-100, and 79.17 on ImageNet-100, on par with Barlow Twins and above several contrastive baselines; the authors find no evidence that removing higher-order redundancy helps downstream performance, and they find that methods outperforming SSLPM-RR show lower measured redundancy, which they interpret as implicit redundancy reduction.

Load-bearing premise

The implicit-redundancy finding assumes that the fixed linear and MLP estimators used on a 20% feature subsample measure redundancy fairly across models with different activation distributions; if that estimator favors some models, the redundancy ranking, and with it the central interpretation, could be an artifact.

Editorial extensions

If this is right

  • SSLPM-RR is competitive with state-of-the-art SSL methods on CIFAR-10, CIFAR-100, and ImageNet-100, despite replacing pairwise decorrelation with a closed-form ridge-regression predictability loss.
  • Reducing nonlinear, higher-order redundancy via multilayer predictors does not improve downstream accuracy and can hurt it, so the practical value of redundancy reduction currently lies mainly in the linear regime.
  • The best-performing methods in the study, including BYOL, Barlow Twins, VICReg, and NNCLR, all have lower measured embedding redundancy than SSLPM-RR, which the paper reads as evidence for implicit redundancy reduction in high-quality SSL.
  • Deepening the projector reduces both linear and nonlinear embedding redundancy, making projector architecture a lever for implicit decorrelation.
  • Across SSL methods generally, redundancy and accuracy are only weakly correlated; the strong negative correlation appears specifically in methods with explicit redundancy reduction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: If implicit redundancy reduction is what separates strong SSL embeddings from weak ones, then the redundancy term in SSLPM-RR may be replaceable by architectural choices such as a deeper projector; a direct test would be whether a lambda=0 SSLPM with a deep projector closes most of the gap to SSLPM-RR.
  • Inference: The closed-form ridge predictor makes the redundancy-reduction step cheap and stable, suggesting that other SSL objectives with quadratic prediction losses could use the same trick rather than training a learned predictor.
  • Inference: The paper's measures are computed on the final embedding layer of frozen encoders; measuring redundancy at intermediate layers or with different estimator capacities could change the method ranking, so the implicit-redundancy claim has a measurement-dependence worth probing.
  • Inference: The authors' observation that too much redundancy reduction collapses performance suggests some residual redundancy is useful; connecting this to information-bottleneck trade-offs is a natural next step.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper formalizes embedding-space redundancy via three measures: average absolute covariance (AAC), linear redundancy (LR), and nonlinear redundancy (NLR). It proves basic inequalities among them and proposes SSLPM, a self-supervised method that reduces redundancy through a competitive game between an encoder and a predictor trained on randomly masked representation features. Two variants are presented: SSLPM-RR (ridge-regression predictor) and SSLPM-SGD (MLP predictor). Experiments on CIFAR-10, CIFAR-100, and ImageNet-100 with ResNet-18 show that SSLPM-RR is competitive with Barlow Twins and other SSL baselines, that within methods with explicit redundancy reduction lower LR is associated with higher downstream accuracy, and that across methods the correlation is weak. Nevertheless, the authors claim that the best-performing methods exhibit lower redundancy and therefore perform implicit redundancy reduction.

Significance. The paper addresses a timely question and provides a useful taxonomy of redundancy measures. The theoretical inequalities are simple but clearly presented, and the SSLPM method is a sensible modern instantiation of predictability minimization. The negative result that removing higher-order redundancies does not improve downstream performance is interesting and potentially important. However, the current version contains a flawed proof of a separating corollary and an over-reaching interpretive claim about implicit redundancy reduction, partly contradicted by the authors' own data and by the fact that two of the cited methods (Barlow Twins and VICReg) perform explicit redundancy reduction. With corrections and robustness checks, the paper could be a valuable contribution.

major comments (3)
  1. [Corollary 3.5 proof (page 21)] The counterexample claims that 'given any two variables, a nonlinear predictor is able to perfectly reconstruct the third variable.' This is false for Y from (X,Z): the equations determine only Y^2, not the sign of Y, so the minimum MSE predictor is E[Y|X,Z]=0 with MSE 1, giving predictability 0. The same holds for X from (Y,Z). Therefore NLR([X,Y,Z]) is 1/3, not 1, and the proof does not establish the corollary. The corollary may still be true (e.g., let Z = (X^2-1)/sqrt(2) with X ~ N(0,1); then AAC = LR = 0 but NLR = 1/2), but the proof must be corrected.
  2. [Section 4.5] The sentence 'Given that these methods do not perform active redundancy reduction' is inaccurate: Barlow Twins and VICReg both impose explicit covariance/redundancy penalties, as described in Sections 1 and 2. Including them in the 'implicit' argument conflates explicit and implicit mechanisms. The abstract's more cautious wording ('even methods without explicit redundancy reduction mechanisms') should be aligned with the body; as written, Section 4.5 overstates the evidence for implicit redundancy reduction.
  3. [Appendix A / Section 4.5] The cross-method redundancy ranking assumes that a single estimator (ridge with cross-validated penalty for LR; a fixed two-hidden-layer MLP with fixed training steps for NLR on a 20% feature subsample) is equally fair to all embedding distributions. Since LR and NLR are defined as 1 minus the achieved normalized MSE, a method whose features are harder to fit with the chosen estimator will appear less redundant without necessarily being so. The paper's own Figure 7 shows no significant across-method correlation on CIFAR-10 (LR r=-0.12, p=0.78; NLR r=-0.11, p=0.80), so the subset claim in Section 4.5 is not backed by a significance test and could be an artifact of estimator capacity. Please add robustness analyses (vary estimator capacity, report redundancy standard errors, or use a distribution-free measure) before drawing the implicit-redundancy conclusion.
minor comments (6)
  1. [Related Work] "SiamSiam" is a typo; the method is SimSiam (Chen & He).
  2. [Corollary 3.5 proof] The line "although LR and NLR are zero" should read "although AAC and LR are zero."
  3. [Section 3.3] The computation of AAC for the constructed example uses notation "(n-1 2)" and "(n 2)" that appears to be a typesetting error; please clarify the combinatorial expressions.
  4. [Figure 3] The y-axis label "Final Epochs Predictability Loss" conflicts with the definition of predictability as 1 - MSE; the plotted quantity appears to be the prediction error, so rename or clarify.
  5. [Section 4.5] The text mentions "P values of 0.79 and 0.93" but Figure 7 reports p=0.78 (LR) and p=0.80 (NLR) for CIFAR-10; the text and figures should be consistent.
  6. [Table 1] Most baselines are single-seed runs; adding at least three seeds for the main baselines would strengthen the competitive claim, though this may be infeasible.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the redundancy hierarchy is proved from definitions, SSLPM is evaluated against external baselines, and the implicit-redundancy claim is a qualified post-hoc observation.

full rationale

The derivation of the redundancy hierarchy (Lemma 3.1, Theorem 3.2, Corollaries 3.3-3.5) is self-contained: the inequalities AAC <= sqrt(LR) <= sqrt(NLR) follow directly from the definitions of AAC, LR, and NLR, with the only substantive input being that linear predictors form a subset of the MLP predictors considered in NLR. This is a mathematical fact, not an assumption equivalent to the result. SSLPM's central claim of competitiveness is benchmarked on CIFAR-10, CIFAR-100, and ImageNet-100 against baselines whose hyperparameters are taken from the external solo-learn repository, so it is not forced by construction. The only mildly self-referential element is that SSLPM-RR optimizes ridge-regression predictability during training and is later measured with LR, also a ridge-regression predictability; its low LR is therefore partly by construction. However, the paper does not present this as a prediction, and the broader conclusion about implicit redundancy reduction rests on comparisons with BYOL, VICReg, NNCLR, and Barlow Twins rather than on SSLPM-RR's own training objective. The paper also qualifies the cross-method claim by reporting weak correlations (e.g., CIFAR-10 LR r=-0.12, p=0.78 in Figure 7). The statement that methods like Barlow Twins do not perform active redundancy reduction is imprecise, but that is a correctness/interpretation issue, not circularity. No load-bearing self-citation, imported uniqueness theorem, or ansatz smuggled via citation was found.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central empirical claims rest on hyperparameters (lambda, ridge penalty, masking fraction, predictor architecture) tuned largely on CIFAR-10, and on domain assumptions about the fairness and adequacy of the redundancy estimators. The theoretical claims rest on standard probability assumptions plus one ad hoc assumption in the Corollary 3.5 counterexample proof that is not valid as written.

free parameters (6)
  • lambda (SSLPM loss weight) = grid-searched on CIFAR-10; optimal near 0.25, values up to 2.0 tested
    Controls the trade-off between invariance and predictability loss; Figure 3 shows sharp performance drop for lambda > 0.25.
  • ridge penalty mu (SSLPM-RR) = chosen to avoid singular regression; exact value not stated, batch size 256
    Required for numerical stability of the closed-form ridge solution; Figure 9 shows too-small mu causes collapse.
  • masking fraction = 0.5 default (Bernoulli(1/2))
    In expectation half of features masked; Figure 10 shows 5% masking hurts, 95% is tolerable; 0.5 is the default.
  • predictor architecture (SSLPM-SGD) = MLP 512-128-64-1 for 1-layer; 3-layer and 4-layer variants also tested
    Choice affects training stability and redundancy removal; Section 4.3.
  • LR ridge penalty mu (measurement) = chosen via cross-validation
    Used for numerical stability when estimating LR; Appendix A.
  • NLR MLP architecture and training steps = 2 hidden layers, dims 128 and 64; fixed training steps
    Affects NLR estimates; Appendix A.
assumptions (4)
  • domain assumption Embedding features are centered and standardized to unit variance before measuring redundancy.
    Used throughout Section 3.2 and Appendix A; makes AAC/LR/NLR unitless and comparable.
  • domain assumption A fixed MLP trained for a fixed number of steps on 80% of the data gives an unbiased estimate of nonlinear predictability on the remaining 20%.
    Underlies all NLR measurements; if training is insufficient, NLR is underestimated for some models.
  • domain assumption The 20% random subsample of features is representative of the full embedding space.
    Only about 102 of 512 features are measured; Appendix A.
  • ad hoc to paper The counterexample in Corollary 3.5 uses a predictor powerful enough to reconstruct the target variable from the other two.
    The proof claims perfect reconstruction from any two variables, which fails because the sign of the Gaussian is not identified; this is an unsupported assumption in the proof.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Pairwise Correlations: Higher-Order Redundancies in Self-Supervised Representation Learning." pith.science (2026). https://pith.science/paper/RO7HR7SF

@misc{pith2026241201926,
  author       = {Pith},
  title        = {Pith review of: Beyond Pairwise Correlations: Higher-Order Redundancies in Self-Supervised Representation Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RO7HR7SF}},
  note         = {Machine review of arXiv:2412.01926}
}
read the original abstract

Several self-supervised learning (SSL) approaches have shown that redundancy reduction in the feature embedding space is an effective tool for representation learning. However, these methods consider a narrow notion of redundancy, focusing on pairwise correlations between features. To address this limitation, we formalize the notion of embedding space redundancy and introduce redundancy measures that capture more complex, higher-order dependencies. We mathematically analyze the relationships between these metrics, and empirically measure these redundancies in the embedding spaces of common SSL methods. Based on our findings, we propose Self Supervised Learning with Predictability Minimization (SSLPM) as a method for reducing redundancy in the embedding space. SSLPM combines an encoder network with a predictor engaging in a competitive game of reducing and exploiting dependencies respectively. We demonstrate that SSLPM is competitive with state-of-the-art methods and find that the best performing SSL methods exhibit low embedding space redundancy, suggesting that even methods without explicit redundancy reduction mechanisms perform redundancy reduction implicitly.

Figures

Figures reproduced from arXiv: 2412.01926 by the authors.

Figure 1
Figure 1. Siamese training set up with encoder in self supervised learning. Embeddings are used in downstream [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Schematic representation of the SSLPM model with the two actors, the encoder-projector network [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Impact of λ in Equation 11 on CIFAR-10. The observations from [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Relationship between Top-1 accuracy and three different redundancy measures on CIFAR-10 [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Ablation on the number of layers of the predictor in SSLPM-SGD compared with SSLPM-RR. All [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Ablation on the number of layers of the projector for CIFAR-10. Redundancy measures have the [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: AAC, LR, and NLR embedding space redundancies plotted against Top-1 accuracy for different [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Training dynamics for different λ: (1) accuracy. (2) Linvariance (3) Lpred. (4) LSSLPM. C.3 Effect of Ridge Penalty and Batch Size in SSLPM-RR When the batch size smaller than half the size of the representation dimension on which we calculate the loss, standard linear…
Figure 9
Figure 9. Figure 9: Impact of ridge penalty on Top-1 CIFAR-10 accuracy. [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: (left) Accuracy over Training. (right) Lpred over Training From [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: AAC, LR, and NLR plotted against each other for test set embeddings of different models on [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Difference in redundancy between test and train sets for CIFAR-10 embeddings. [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Relationship between Top-1 accuracy and different redundancy measures redundancies on CIFAR [PITH_FULL_IMAGE:figures/full_fig_p023_13.png]
Figure 14
Figure 14. Figure 14: Redundancy measures vs. Top-1 accuracy of SSL methods on CIFAR-100, and ImageNet-100. [PITH_FULL_IMAGE:figures/full_fig_p023_14.png]
Figure 15
Figure 15. Figure 15: Ablation on the number of layers of the projector for CIFAR-100 and ImageNet-100. Redundancy [PITH_FULL_IMAGE:figures/full_fig_p024_15.png]
Figure 16
Figure 16. Figure 16: Difference in redundancy between test and train sets for CIFAR-100 and ImageNet-100 embeddings. [PITH_FULL_IMAGE:figures/full_fig_p024_16.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 5 canonical work pages

  1. [3]

    Hence, we conclude that the exact choice of the objective for the ridge regression is likely not of crucial importance for our method

    They feature a nearly identical final predictability loss as well as accuracies on CIFAR-10. Hence, we conclude that the exact choice of the objective for the ridge regression is likely not of crucial importance for our method. Table 3: Change of regression objective CIFAR-10 Models Top 1 Top 5 Last Epoch Avg Lpred SSLPM-RR with Wdefault 92.81 99.87 0.65 ...

    work page 2022

  2. [5]

    Understanding and improving the role of projection head in self-supervised learning.arXiv preprint arXiv:2212.11491 ,

    Kartik Gupta, Thalaiyasingam Ajanthan, Anton van den Hengel, and Stephen Gould. Understanding and improving the role of projection head in self-supervised learning.arXiv preprint arXiv:2212.11491 ,

    arXiv

  3. [10]

    Large batch training of convolutional networks.arXiv preprint arXiv:1708.03888,

    Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks.arXiv preprint arXiv:1708.03888,

    arXiv

  4. [12]

    and is a 100 category subset of the ImageNet dataset Deng et al. (2009). F Additional Evidence for Analysis & Results 22 0.05 0.10 0.15 AAC 40 50 60 70T op1 Accuracy CIFAR-100 - AAC SSLPM-RR: r=-0.89, p<0.01 Barlow T wins: r=-0.90, p<0.01 0.6 0.7 0.8 0.9 LR 40 50 60 70T op1 Accuracy CIFAR-100 - LR SSLPM-RR: r=-0.90, p<0.01 Barlow T wins: r=-0.98, p<0.01 0...

    work page 2009

  5. [2009]

    Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805 ,

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805 ,

    arXiv

  6. [2018]

    Generative adversarial networks are special cases of artificial curiosity (1990) and also closely related to predictability minimization (1991).Neural Networks, 127:58–66,

    Jürgen Schmidhuber. Generative adversarial networks are special cases of artificial curiosity (1990) and also closely related to predictability minimization (1991).Neural Networks, 127:58–66,

    work page 1990

  7. [2019]

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals

    URL https://openreview.net/forum?id=Bkg6RiCqY7. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 ,

    arXiv

  8. [2020]

    Xinlei Chen and Kaiming He

    URLhttps://proceedings.mlr.press/v119/chen20j.html. Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. in 2021 ieee. InCVF conference on computer vision and pattern recognition (CVPR) , pp. 15745–15753,

    work page 2021

Show all 12 references
  1. [2021]

    Computer vision self-supervised learning methods on time series.arXiv preprint arXiv:2109.00783,

    Daesoo Lee and Erlend Aune. Computer vision self-supervised learning methods on time series.arXiv preprint arXiv:2109.00783,

    arXiv

  2. [2022]

    Imagenet: A large-scale hierarchical image database

    12 Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pp. 248–255. Ieee,

    2009

  3. [2023]

    Vicreg: Variance-invariance-covariance regularization for self-supervised learning

    Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105.04906 ,

    arXiv

  4. [2024]

    doi: 10.3390/e26030252

    ISSN 1099-4300. doi: 10.3390/e26030252. URL https://www.mdpi.com/1099-4300/26/3/252. 13 Yao-Hung Hubert Tsai, Shaojie Bai, Louis-Philippe Morency, and Ruslan Salakhutdinov. A note on connecting barlow twins with negative-sample-free contrastive learning.arXiv preprint arXiv:21...

    arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.