pith. sign in

arxiv: 2606.04280 · v1 · pith:X77USN35new · submitted 2026-06-02 · 💻 cs.LG · cs.AI· cs.IR

The Loss Is Not Enough: Sampling Conditions and Inductive Bias in Contrastive Representation Learning

Justinas Zaliaduonis , Patrick Putzky , Till Richter , Sergios Gatidis This is my paper

Pith reviewed 2026-06-28 10:26 UTC · model grok-4.3

classification 💻 cs.LG cs.AIcs.IR
keywords contrastive learningrepresentation learninglatent geometrysampling conditionsinductive biasInfoNCEvon Mises-Fisher
0
0 comments X

The pith

Global contrastive loss minimizers recover latent geometry up to orthogonal transformation only when positive-pair sampling satisfies the diversity condition.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper develops a measure-theoretic framework that isolates a diversity condition on the support of positive-pair sampling as a necessary requirement for isometric latent recovery in contrastive learning. Under the standard full-support von Mises-Fisher distribution this condition holds, so global minimizers of the contrastive loss recover the latent geometry up to orthogonal transformation. When the conditional distributions are restricted, non-orthogonal maps can achieve strictly lower asymptotic loss. A support-corrected variant of InfoNCE is introduced that makes orthogonal recovery possible without forcing it to be unique. Experiments on synthetic data confirm the identifiability statements while CIFAR-10 runs show that limited sampling diversity increases the influence of encoder architecture.

Core claim

The central claim is that the diversity condition, a support requirement on positive-pair sampling, is necessary for isometric latent recovery. The standard full-support von Mises-Fisher setting satisfies the diversity condition and therefore global contrastive loss minimizers recover latent geometry up to orthogonal transformation. Restricted conditionals allow non-orthogonal maps to attain strictly lower asymptotic contrastive loss. The support-corrected InfoNCE variant makes orthogonal latent space recovery achievable but does not uniquely select it.

What carries the argument

The diversity condition, a support requirement on positive-pair sampling that is necessary for isometric latent recovery.

If this is right

  • Full-support von Mises-Fisher sampling implies that global loss minimizers recover latent geometry up to orthogonal transformation.
  • Restricted sampling conditionals permit non-orthogonal maps to achieve strictly lower loss.
  • The support-corrected InfoNCE makes orthogonal recovery achievable without selecting it uniquely.
  • When sampling diversity is limited, architectural inductive bias becomes more decisive, consistent with the CIFAR-10 observations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Many practical datasets may implicitly restrict positive-pair support, shifting reliance onto encoder architecture.
  • The same diversity lens could be applied to other self-supervised objectives to derive analogous recovery conditions.
  • Increasing sampling diversity in training pipelines should measurably reduce sensitivity to architectural choices.

Load-bearing premise

The diversity condition on the support of positive-pair sampling is necessary for isometric latent recovery.

What would settle it

An explicit construction or numerical check showing that, even under full-support von Mises-Fisher positive-pair sampling, some non-orthogonal map attains lower or equal asymptotic contrastive loss than every orthogonal map.

Figures

Figures reproduced from arXiv: 2606.04280 by Justinas Zaliaduonis, Patrick Putzky, Sergios Gatidis, Till Richter.

Figure 1
Figure 1. Figure 1: Overview of contrastive learning and the role of sampling diversity and inductive bias. The generative process g maps latent variables to observations, and the encoder f learns to recover the latent structure. Here f1 denotes a low inductive bias encoder (e.g., MLP) and f2 a high inductive bias encoder (e.g., a model of the inverse process). Orange dot indicates the anchor point; green dots are co-occurrin… view at source ↗
Figure 2
Figure 2. Figure 2: Generative processes mapping the unit sphere to observation space. Colors encode input coordinates (RGB = xyz), illustrating how each transformation warps the latent space: (a) identity preserves the sphere, (b) linear maps to an ellipsoid, (c) spiral twists points around the vertical axis, (d) patches applies piecewise rotations creating discontinuities, and (e) invertible MLP produces smooth nonlinear de… view at source ↗
Figure 3
Figure 3. Figure 3: Linear probe accuracy on CIFAR-10 by architecture and augmentation regime. Individual runs shown as points; bars indicate mean ±1 std. The “All” regime best approximates the diversity condition and yields highest accuracy across all architec￾tures. 6. Conclusion We have presented a theoretical framework for understand￾ing when contrastive learning recovers meaningful latent 8 [PITH_FULL_IMAGE:figures/full… view at source ↗
Figure 4
Figure 4. Figure 4: CIFAR-10 augmentation regimes. (a) Original image. (b) Crop Only: random resized crop altering spatial extent. (c) All without crop: color jitter, horizontal flip, rotation, and blur. (d) All augmentations combined. Cropping changes visible spatial extent and local statistics, while color and blur transformations mainly alter appearance in this example. representations. Our central contribution is the dive… view at source ↗
Figure 5
Figure 5. Figure 5: provides visual summaries comparing MLP and inverse encoder performance across conditions. (a) (b) (c) [PITH_FULL_IMAGE:figures/full_fig_p026_5.png] view at source ↗
read the original abstract

Contrastive learning has become a leading paradigm for self-supervised representation learning, yet the conditions under which it recovers meaningful latent geometry remain incompletely understood. We develop a measure-theoretic framework formalizing the diversity condition, a support requirement on positive-pair sampling that is necessary for isometric latent recovery. We show that the standard full-support von Mises-Fisher setting implies the satisfaction of the diversity condition and as a consequence global contrastive loss minimizers recover latent geometry up to orthogonal transformation, while restricted conditionals can make non-orthogonal maps attain strictly lower asymptotic contrastive loss. We introduce a support-corrected Information Noise Contrastive Estimation (InfoNCE) variant as a theoretical fix: this correction makes orthogonal latent space recovery achievable but does not uniquely select it. Experiments on synthetic benchmarks validate the identifiability predictions, and CIFAR-10 experiments are consistent with the qualitative prediction that architectural inductive bias becomes more important when sampling diversity is limited. Together, our results clarify how sampling mechanisms and encoder inductive bias interact in contrastive representation learning.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 2 minor

Summary. The manuscript develops a measure-theoretic framework formalizing the diversity condition on positive-pair sampling as necessary for isometric latent recovery in contrastive learning. It shows that the standard full-support von Mises-Fisher distribution satisfies the diversity condition, so that global minimizers of the contrastive loss recover latent geometry up to orthogonal transformation, while restricted-support conditionals allow non-orthogonal maps to achieve strictly lower asymptotic loss. A support-corrected InfoNCE variant is introduced, and the predictions are validated on synthetic benchmarks with CIFAR-10 experiments illustrating the increased role of architectural inductive bias under limited sampling diversity.

Significance. If the central claims hold, the work provides a valuable clarification of the interplay between sampling support and encoder inductive bias in contrastive representation learning. The formalization of the diversity condition and the explicit counterexamples under restricted support offer a useful theoretical tool for understanding identifiability, with the proposed correction highlighting that loss modification alone does not guarantee unique orthogonal recovery.

minor comments (2)
  1. The abstract states that the support-corrected InfoNCE 'makes orthogonal latent space recovery achievable but does not uniquely select it'; the precise functional form of the correction and the proof that it fails to enforce uniqueness should be stated explicitly with reference to the relevant theorem or proposition.
  2. [Experiments] In the CIFAR-10 experiments, the claim that results are 'consistent with the qualitative prediction' would be strengthened by reporting quantitative metrics (e.g., alignment or downstream accuracy deltas) rather than qualitative description alone.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for the careful reading, positive assessment of the work, and recommendation for minor revision. No specific major comments were provided in the report.

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The derivation establishes a measure-theoretic diversity condition as necessary for isometric latent recovery, proves that full-support von Mises-Fisher positive-pair sampling satisfies it, and shows that global contrastive-loss minimizers then recover geometry up to orthogonal transformation. Restricted-support counterexamples demonstrate necessity. All steps rest on standard definitions of the InfoNCE loss, von Mises-Fisher distributions, and measure-theoretic support conditions; no parameter is fitted to data and then renamed a prediction, no load-bearing claim reduces to a self-citation, and no ansatz is smuggled via prior work by the same authors. The framework is therefore self-contained against external mathematical benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 1 invented entities

The framework rests on measure-theoretic assumptions about sampling distributions and the definition of the diversity condition; no free parameters or invented entities are explicitly fitted or postulated in the abstract.

axioms (2)
  • domain assumption Positive-pair sampling distributions must satisfy a support requirement (diversity condition) for isometric recovery to be possible.
    Invoked as necessary condition in the framework development.
  • domain assumption Standard von Mises-Fisher distribution with full support satisfies the diversity condition.
    Used to link common sampling to recovery guarantees.
invented entities (1)
  • diversity condition no independent evidence
    purpose: Formal support requirement on positive-pair sampling necessary for isometric latent recovery.
    New concept introduced in the measure-theoretic framework.

pith-pipeline@v0.9.1-grok · 5720 in / 1389 out tokens · 24977 ms · 2026-06-28T10:26:29.536556+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 3 canonical work pages · 1 internal anchor

  1. [1]

    Proceedings of the 37th International Conference on Machine Learning , pages=

    A Simple Framework for Contrastive Learning of Visual Representations , author=. Proceedings of the 37th International Conference on Machine Learning , pages=. 2020 , volume=

    2020

  2. [2]

    International Conference on Learning Representations , year=

    Equivariant Contrastive Learning , author=. International Conference on Learning Representations , year=

  3. [3]

    2025 , eprint=

    CLICv2: Image Complexity Representation via Content Invariance Contrastive Learning , author=. 2025 , eprint=

    2025

  4. [4]

    2022 , eprint=

    Contrastive Unsupervised Learning of World Model with Invariant Causal Features , author=. 2022 , eprint=

    2022

  5. [5]

    Advances in Neural Information Processing Systems , volume=

    Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style , author=. Advances in Neural Information Processing Systems , volume=

  6. [6]

    Technologies , volume=

    A Survey on Contrastive Self-Supervised Learning , author=. Technologies , volume=. 2021 , publisher=

    2021

  7. [7]

    Proceedings of the 38th International Conference on Machine Learning , pages=

    Learning Transferable Visual Models From Natural Language Supervision , author=. Proceedings of the 38th International Conference on Machine Learning , pages=. 2021 , volume=

    2021

  8. [8]

    Bardes, Adrien and Ponce, Jean and LeCun, Yann , booktitle=

  9. [9]

    Journal of Machine Learning Research , volume=

    The Power of Contrast for Feature Learning: A Theoretical Analysis , author=. Journal of Machine Learning Research , volume=. 2023 , url=

    2023

  10. [10]

    Proceedings of the 38th International Conference on Machine Learning , pages=

    Toward Understanding the Feature Learning Process of Self-Supervised Contrastive Learning , author=. Proceedings of the 38th International Conference on Machine Learning , pages=. 2021 , volume=

    2021

  11. [11]

    Proceedings of the 40th International Conference on Machine Learning , pages=

    Probabilistic Contrastive Learning Recovers the Correct Aleatoric Uncertainty of Ambiguous Inputs , author=. Proceedings of the 40th International Conference on Machine Learning , pages=. 2023 , volume=

    2023

  12. [12]

    2025 , eprint=

    Contrastive Diffusion Alignment: Learning Structured Latents for Controllable Generation , author=. 2025 , eprint=

    2025

  13. [13]

    Proceedings of the 38th International Conference on Machine Learning , pages=

    Contrastive Learning Inverts the Data Generating Process , author=. Proceedings of the 38th International Conference on Machine Learning , pages=. 2021 , volume=

    2021

  14. [14]

    2009 , institution=

    Learning Multiple Layers of Features from Tiny Images , author=. 2009 , institution=

    2009

  15. [15]

    Nonlinear

    Hyv. Nonlinear. Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics , pages=. 2019 , volume=

    2019

  16. [16]

    Proceedings of the 37th International Conference on Machine Learning , pages=

    Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere , author=. Proceedings of the 37th International Conference on Machine Learning , pages=. 2020 , volume=

    2020

  17. [17]

    2019 , eprint=

    Representation Learning with Contrastive Predictive Coding , author=. 2019 , eprint=

    2019

  18. [18]

    Nature Machine Intelligence , volume=

    Delineating the effective use of self-supervised learning in single-cell genomics , author=. Nature Machine Intelligence , volume=. 2025 , publisher=

    2025

  19. [19]

    Advances in Neural Information Processing Systems , volume=

    Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning , author=. Advances in Neural Information Processing Systems , volume=

  20. [20]

    Linking Neural Collapse and

    Haas, Jarrod and Yolland, William and Rabus, Bernhard , journal=. Linking Neural Collapse and. 2023 , issn=

    2023

  21. [21]

    and Yuan, Liangzhe and Zhou, Hao and Yan, Shen and Sun, Jennifer J

    Zhao, Long and Gundavarapu, Nitesh B. and Yuan, Liangzhe and Zhou, Hao and Yan, Shen and Sun, Jennifer J. and Friedman, Luke and Qian, Rui and Weyand, Tobias and Zhao, Yue and Hornung, Rachel and Schroff, Florian and Yang, Ming-Hsuan and Ross, David A. and Wang, Huisheng and Adam, Hartwig and Sirotenko, Mikhail and Liu, Ting and Gong, Boqing , booktitle=....

    2024

  22. [22]

    2023 , eprint=

    A Survey on Self-Supervised Representation Learning , author=. 2023 , eprint=

    2023

  23. [23]

    Signal Processing , volume=

    Independent component analysis, a new concept? , author=. Signal Processing , volume=

  24. [24]

    Neural Networks , volume=

    Independent component analysis: algorithms and applications , author=. Neural Networks , volume=

  25. [25]

    IEEE Transactions on Pattern Analysis and Machine Intelligence , volume=

    Representation learning: A review and new perspectives , author=. IEEE Transactions on Pattern Analysis and Machine Intelligence , volume=

  26. [26]

    International Conference on Machine Learning , pages=

    Challenging common assumptions in the unsupervised learning of disentangled representations , author=. International Conference on Machine Learning , pages=

  27. [27]

    Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear

    Hyv. Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear. Advances in Neural Information Processing Systems , year=

  28. [28]

    and Ba, Jimmy , booktitle=

    Kingma, Diederik P. and Ba, Jimmy , booktitle=. 2015 , url=

    2015

  29. [29]

    Improved Regularization of Convolutional Neural Networks with Cutout

    Improved Regularization of Convolutional Neural Networks with Cutout , author=. arXiv preprint arXiv:1708.04552 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  30. [30]

    Advances in Neural Information Processing Systems , year=

    What makes for good views for contrastive learning? , author=. Advances in Neural Information Processing Systems , year=

  31. [31]

    International Conference on Learning Representations , year=

    Representation learning via invariant causal mechanisms , author=. International Conference on Learning Representations , year=

  32. [32]

    International Conference on Learning Representations , year=

    A Theoretical Study of Inductive Biases in Contrastive Learning , author=. International Conference on Learning Representations , year=

  33. [33]

    Tschannen, Michael and Mustafa, Basil and Houlsby, Neil , booktitle=

  34. [34]

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=

    Deep Residual Learning for Image Recognition , author=. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=

  35. [35]

    and Brendel, Wieland , booktitle=

    Rusak, Evgenia and Reizinger, Patrik and Juhos, Attila and Bringmann, Oliver and Zimmermann, Roland S. and Brendel, Wieland , booktitle=. 2025 , volume=

    2025

  36. [36]

    Exploring Simple, High Quality Out-of-Distribution Detection with

    Haas, Jarrod and Yolland, William and Rabus, Bernhard , journal=. Exploring Simple, High Quality Out-of-Distribution Detection with. 2024 , issn=

    2024

  37. [37]

    Proceedings of the IEEE , volume=

    Gradient-Based Learning Applied to Document Recognition , author=. Proceedings of the IEEE , volume=. 1998 , doi=

    1998

  38. [38]

    International Conference on Learning Representations , year=

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale , author=. International Conference on Learning Representations , year=

  39. [39]

    The Nature of Statistical Learning Theory , year =

    Vladimir Vapnik , editor =. The Nature of Statistical Learning Theory , year =

  40. [40]

    A Probabilistic Generalization of the

    Zaliaduonis, Justinas and Gatidis, Sergios , year=. A Probabilistic Generalization of the. 2601.03900 , archivePrefix=

    work page arXiv

  41. [41]

    Wood, Andrew T. A. , journal=. Simulation of the von. 1994 , doi=

    1994

  42. [42]

    2025 , doi =

    Bahrami, Mojtaba and Tejada-Lapuerta, Alejandro and Becker, S. 2025 , doi =. https://www.biorxiv.org/content/early/2025/10/15/2025.10.14.682419.full.pdf , journal =

    2025

  43. [43]

    and Soljacic, Marin , booktitle=

    Cy, Ali and Chemparathy, Anugrah and Han, Michael and Dangovski, Rumen and Lu, Peter Y. and Soljacic, Marin , booktitle=. Studying Phase Transitions in Contrastive Learning. 2023 , note=

    2023

  44. [44]

    Contrastive learning for robust representations of neutrino data , volume=

    Wilkinson, Alex and Radev, Radi and Alonso-Monsalve, Saúl , year=. Contrastive learning for robust representations of neutrino data , volume=. Physical Review D , publisher=. doi:10.1103/physrevd.111.092011 , number=

    work page doi:10.1103/physrevd.111.092011

  45. [45]

    AAAI 2022 Fall Symposium: The Role of AI in Responding to Climate Challenges , url=

    Contrastive Learning for Climate Model Bias Correction and Super-Resolution , author=. AAAI 2022 Fall Symposium: The Role of AI in Responding to Climate Challenges , url=

    2022

  46. [46]

    Information , VOLUME =

    Liu, Zihe and Hu, Keyong and Zhang, Jingxuan and Ren, Xingchen and Wang, Xi , TITLE =. Information , VOLUME =. 2026 , NUMBER =

    2026