Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Position: An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Research

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This position paper argues that identifiability theory today cannot explain why self-supervised learning works, and that an empirically grounded extension, Singular Identifiability Theory, is the path to closing that gap and accelerating…

desk verdict A well-organized position paper that names the gaps between identifiability theory and SSL practice, but whose central acceleration claim is a tractability bet — as the paper's own §4.8 shows that more identifiable latents can slightly hurt the current headline metric. read the letter →

arxiv 2504.13101 v3 pith:XFMGHLRH submitted 2025-04-17 cs.LG cs.AIstat.ML

Patrik Reizinger , Randall Balestriero , David Klindt , Wieland Brendel This is my paper · ORCID
classification cs.LGcs.AIstat.ML
keywords self-supervisedlearningidentifiabilitytheorySingulardatageneratingprocessPlatonicRepresentationHypothesistheory-practicegapaugmentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Self-supervised learning now powers most AI systems, but the field lacks a theory that explains why its representations work. This position paper argues that identifiability theory, which asks when the hidden factors behind observed data can be recovered, has produced real insights for SSL yet cannot account for what practitioners actually do: finite data, limited training time, specific architectures, and aggressive augmentations. The authors propose expanding identifiability theory into Singular Identifiability Theory (SITh), an empirically grounded framework built around realistic data generating processes. If SITh is developed, it would supply principled answers to when and why self-supervised representations converge, and would give concrete guidance for designing and evaluating SSL methods. The paper also argues that the Platonic Representation Hypothesis, the observation that different SSL methods converge to similar representations, can emerge from identifiability theory once the underlying data generating process is made explicit.

What carries the argument

The central object is the data generating process (DGP), the virtual renderer that maps latent factors to observations, together with the identifiability guarantees that hold relative to it. The proposal is to rework the DGP from a mathematical convenience into an empirically grounded description of actual SSL pipelines, including augmentations, finite data, and training dynamics. The paper's Table 1 is a load-bearing roadmap: it maps each known theory-practice gap, such as augmentations, finite data, finite time, inductive biases, dimensional collapse, the projector, compositionality, the contrastive/non-contrastive dichotomy, and evaluation, to whether theory or practice currently understands it and to concrete research questions that SITh would need to answer.

What would settle it

One concrete test: take a widely used SSL method such as SimCLR with heavy random crops, and attempt to derive a finite-sample, finite-time identifiability guarantee for a realistic data generating process that predicts which latent factors survive. If no such guarantee can be derived, or if the assumptions needed are as unrealistic as the isotropic conditionals the paper criticizes, the claim that SITh will close the theory-practice gap loses its force.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that the gap between SSL practice and theory is the main barrier to faster progress, and that the right response is a new theory, Singular Identifiability Theory (SITh), rather than more empirical scaling or more idealized theory. Current identifiability theory assumes infinite data, converged models, and unrealistic augmentation models such as isotropic conditional distributions on the hypersphere, so it cannot explain dimensional collapse, the projector phenomenon, loss saturation, or out-of-distribution behavior. SITh would keep the core construct of a data generating process but ground it in empirical observation, covering training dynamics, finite samples, batch size, data diversity, architecture, initialization, stop-gradient tricks, and principled evaluation. The paper synthesizes existing identifiability results to show that the Platonic Representation Hypothesis can emerge in SSL: methods that effectively minimize cross-entropy against the same underlying data generating process should converge to linearly related representations, which is also why the contrastive/non-contrastive split in SSL looks sterile.

Load-bearing premise

The paper's central bet is that identifiability results, which today hold only for infinite data and fully trained models, can be extended to cover finite samples, learning dynamics, and architectural choices while still explaining what practitioners see.

Editorial extensions

If this is right

  • If SITh is developed, researchers would know under which data and training conditions different SSL methods converge to the same representation, making the Platonic Representation Hypothesis a testable theorem rather than an observation.
  • Design choices such as augmentation strength, batch size, initialization, and stop-gradient usage would come with principled recommendations instead of being tuned as heuristics.
  • Evaluation would shift from a single ImageNet classification number toward benchmarks that measure which latent factors a representation actually captures, including out-of-distribution and compositional generalization.
  • The contrastive/non-contrastive split in SSL would be replaced by a common analysis based on the data generating process, since both paradigms are already viewed as different means to minimize cross-entropy or estimate entropy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors do not say this, but the DGP-centric view implies that the Platonic ideal is not a single universal representation: it would be one ideal per data generating process, so models trained on different data distributions could legitimately converge to different representations.
  • A concrete testable extension would be to measure whether replacing the isotropic augmentation model with anisotropy fitted to real crop statistics predicts which latent dimensions collapse in actual SimCLR or VICReg training.
  • Another extension the paper leaves implicit is that SITh would invert the current research workflow, letting a designer declare the target data generating process first and then derive the SSL loss that matches it, rather than reverse-engineering theory from already successful methods.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This position paper argues that current identifiability theory (IT), while successful in explaining some aspects of self-supervised learning (SSL), cannot account for SSL's empirical success, and proposes 'Singular Identifiability Theory' (SITh), an extension of IT that is grounded in empirical observations of data generating processes, training dynamics, finite samples, batch size, and inductive biases. The authors provide a practitioner-oriented introduction to IT, use SimCLR as a case study, review the state of the field, and compile a table of theory-practice gaps and research questions in Table 1. They claim that SITh will accelerate SSL research by designing realistic DGPs, formalizing when and why SSL representations converge, and providing principled recommendations for designing and evaluating SSL methods. The paper explicitly frames SITh as an umbrella term and a blueprint, not a worked-out theory.

Significance. If the thesis is correct, SITh could provide a unifying framework that connects SSL theory to the practical phenomena that dominate the field: augmentations, finite samples, convergence dynamics, architectural choices, and evaluation. The paper's strengths are its extensive and current reference list, its honest and detailed articulation of open gaps in Table 1, its accessible crash course on identifiability in Section 3, and its explicit acknowledgement that several of its core assumptions remain unresolved. It does not present a derivation or an experiment, so its value lies in the quality of the proposed research agenda and the evidence marshalled for it. The main risk, acknowledged by the authors, is that the link between identifiability certificates and the outcomes SSL research actually cares about remains unformalized; this risk is load-bearing for the central claim and is not resolved by the paper.

major comments (3)
  1. [§4.8, §4.9] The central acceleration claim lacks a defined target. The paper never states a formal or even a precise informal relationship between identifiability (recovering more ground-truth latent factors) and the outcomes that SSL research optimizes (downstream accuracy, transfer, sample efficiency, or a well-defined notion of universality). The paper itself notes in §4.8 that 'SSL methods capturing more latents sometimes have (slightly) lower downstream classification accuracy (Rusak et al., 2024)' and concedes in §4.9 that 'identifiability guarantees are not valuable per se.' Because SITh's design recommendations would be justified precisely by such a link, the proposal needs either a theorem or a strong empirical law mapping identifiability to a performance measure, or an explicit redefinition of the field's target (e.g., latent-recovery-based universality) with a corresponding evaluation protocol. As written, the acceleration claim is a tractability bet rather than an empirically grounded argument, so this gap needs to be addressed.
  2. [§4.1, §4.9] The core proposal of an 'empirically grounded' DGP is under-specified. Section 4.9 says that when a theorist constructs a DGP 'the focus should not only be on identifiability but also on the match with reality,' and §4.1 criticizes vMF conditionals as unrealistic, but the paper offers no method for deciding whether a DGP matches reality, how to fit a DGP from data, or how to validate it beyond subjective plausibility. Since the distinction between SITh and existing IT rests on this empirical grounding, the position needs at least a concrete falsifiable criterion (such as predictive checks, identifiability of parameters on real data, or a benchmark comparison) for DGP adequacy; without it, 'empirically grounded' reduces to an aspiration.
  3. [§4.3, Table 1] The intended extension from asymptotic identifiability to the non-asymptotic phenomena that motivate SITh is asserted rather than demonstrated. Table 1 lists finite samples, finite time, loss saturation, and inductive biases as open problems, and §4.3 correctly observes that IT 'cannot distinguish between the convergence speed of models.' However, no concrete mathematical bridge is proposed: there is no worked example, no finite-sample identifiability statement, and no precise sense in which the singular-learning-theory analogy transfers. The research agenda would be strengthened by one worked-out instance (for example, a linear network with a finite-sample identifiability bound, or a minimal architecture in which the SITh extension changes a theoretical prediction) to make the tractability bet concrete and falsifiable.
minor comments (4)
  1. [§4.2] The sentence 'Identifiability results assume infinite data, batch size, and converged, i.e., IT is an asymptotic theory' appears to be missing a noun; it should read something like 'converged models.'
  2. [§3.2] The text 'made the implict assumptions' contains a typo: 'implict' should be 'implicit.'
  3. [§4.7] The sentence 'both method families contrast some properties' is unclear; 'contrast' should likely be 'share' or 'contrast with respect to', and the intended meaning should be clarified.
  4. [Appendix A] The distinction between 'absolute' and 'relative' identifiability is introduced only in an appendix, but is used implicitly in the main text (e.g., in §3.1 and §4.7); a brief forward reference in the main text would help readers.

Circularity Check

0 steps flagged · score 2.0 of 10

No formal circularity: SITh is an explicitly labeled research blueprint, and the authors' self-citations are supporting evidence rather than load-bearing inputs.

full rationale

This is a position paper, not a derivation: it explicitly states that "SITh, in its current form, is an umbrella term for a future theory and a blueprint" (Section 4). There are no parameters fitted to data, no quantitative predictions that are then read back as confirmations, and no definitional equivalence between an input and a claimed output. The central proposal—that an empirically grounded extension of identifiability theory would accelerate SSL research—is a tractability bet, not a theorem; the paper lists open questions in Table 1 instead of deriving consequences. The supporting examples of IT's value do rely in part on the authors' own prior work (e.g., Zimmermann et al. 2021; Rusak et al. 2024; Reizinger et al. 2024a), but these are cited as published technical results with stated assumptions, and the motivation is also backed by independent empirical studies (Morningstar et al. 2024; Ciernik et al. 2024; Simon et al. 2023). Section 4.9 explicitly concedes the uncertainty: "Identifiability guarantees are not valuable per se, only to the extent they can describe empirical phenomena, or—and only time will tell this—they aid the development of such theories that explain such phenomena." Section 4.8 even notes that "SSL methods capturing more latents sometimes have (slightly) lower downstream classification accuracy (Rusak et al., 2024)", which weakens the assumed value chain but is a correctness risk, not a circularity. Accordingly, no step reduces to its own input by construction; the only mild issue is the heavy presence of self-citations in the motivation, which is why the score is 2 rather than 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 1 invented entities

This is a position paper, so the ledger contains no fitted parameters or data-derived constants. The load-bearing premises are the framing assumptions of the research agenda: the DGP view of SSL, the priority of identifiability as a theoretical target, the feasibility of empirically grounded DGPs, and the claim that scaling alone is insufficient. These are plausible but unproven bets.

assumptions (4)
  • domain assumption Self-supervised representations can be modeled as arising from an underlying data generating process (DGP) with latent factors.
    This is the core modeling choice inherited from ICA; the paper relies on it throughout Section 3 and Section 4 to discuss what SSL learns and when representations converge.
  • domain assumption Identifiability of the DGP is the right theoretical target for explaining SSL success.
    The paper equates explaining SSL with proving identifiability, stated in Section 3 and used as the criterion for evaluating theory in Section 4. This assumption is not questioned inside the paper.
  • ad hoc to paper Empirically grounded DGPs, matched to real augmentation distributions, are achievable and more valuable than idealized DGPs.
    The central call of SITh (Section 4) presupposes that DGP design can be grounded in empirical observations without losing tractability, which is asserted but not demonstrated with an example.
  • domain assumption Scaling of data and compute will not suffice to advance SSL, so a theoretical framework is needed.
    Defended in Section 5 against the scaling view, but the supporting evidence is largely anecdotal (personal observations at a NeurIPS workshop and the existence of scaling laws) rather than a systematic counterfactual.
invented entities (1)
  • Singular Identifiability Theory (SITh)
    purpose: Proposed umbrella framework to extend identifiability theory across the entire SSL pipeline, covering finite samples, training dynamics, and inductive biases.
    The paper defines SITh only as a blueprint and future theory (Section 4), with no formal statements, axioms, or falsifiable predictions that could be tested now. It is a name for a research direction, not an entity with independent evidence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Position: An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Research." pith.science (2026). https://pith.science/paper/XFMGHLRH

@misc{pith2026250413101,
  author       = {Pith},
  title        = {Pith review of: Position: An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Research},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XFMGHLRH}},
  note         = {Machine review of arXiv:2504.13101}
}
read the original abstract

Self-Supervised Learning (SSL) powers many current AI systems. As research interest and investment grow, the SSL design space continues to expand. The Platonic view of SSL, following the Platonic Representation Hypothesis (PRH), suggests that despite different methods and engineering approaches, all representations converge to the same Platonic ideal. However, this phenomenon lacks precise theoretical explanation. By synthesizing evidence from Identifiability Theory (IT), we show that the PRH can emerge in SSL. However, current IT cannot explain SSL's empirical success. To bridge the gap between theory and practice, we propose expanding IT into what we term Singular Identifiability Theory (SITh), a broader theoretical framework encompassing the entire SSL pipeline. SITh would allow deeper insights into the implicit data assumptions in SSL and advance the field towards learning more interpretable and generalizable representations. We highlight three critical directions for future research: 1) training dynamics and convergence properties of SSL; 2) the impact of finite samples, batch size, and data diversity; and 3) the role of inductive biases in architecture, augmentations, initialization schemes, and optimizers.

Figures

Figures reproduced from arXiv: 2504.13101 by the authors.

Figure 1
Figure 1. Representational similarities between contrastive and non-contrastive methods trained on ImageNet-1k: (Left): the similarities are calculated with an RBF-kernel–based CKA with σ = 0.4. We include supervised baselines for reference. Reproduced from (Ciernik et al., 2024) with the authors’ permission. (Right): a two-dimensional UMAP embedding of similarity matrix colored by method types (contrastive, non-contrastive a… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Understanding Self-Supervised Learning via Latent Distribution Matching

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    Self-supervised learning is recast as latent distribution matching that unifies multiple SSL families and yields a sampling-free Kalman-based predictor plus an identifiability proof for predictive variants under mild ...

Reference graph

Works this paper leans on

12 extracted references · 10 linked inside Pith · cited by 1 Pith paper

  1. [1]

    4, 5 DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., Zhang, X., Yu, X., Wu, Y ., Wu, Z. F., Gou, Z., Shao, Z., Li, Z., Gao, Z., Liu, A., Xue, B., Wang, B., Wu, B., Feng, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Chen, D., Ji, D., Li, E., Lin, F., Dai, F., Luo, F., Hao, G., Chen...

    arXiv 2025

  2. [2]

    Func- tion Classes for Identifiable Nonlinear Independent Com- ponent Analysis

    1, 4 Buchholz, S., Besserve, M., and Sch ¨olkopf, B. Func- tion Classes for Identifiable Nonlinear Independent Com- ponent Analysis. pp. 25, 2022. URL https:// arxiv.org/abs/2208.06406. 18 Cabannes, V ., Kiani, B. T., Balestriero, R., LeCun, Y ., and Bietti, A. The SSL Interplay: Augmentations, Inductive Bias, and Generalization, February 2023. URLhttp://...

    arXiv 2022

  3. [9]

    URL http://ebooks.cambridge.org/ref/id/ CBO9780511803161

    doi: 10 .1017/CBO9780511803161. URL http://ebooks.cambridge.org/ref/id/ CBO9780511803161. 3 Power, A., Burda, Y ., Edwards, H., Babuschkin, I., and Misra, V . Grokking: Generalization Be- yond Overfitting on Small Algorithmic Datasets, January 2022. URL http://arxiv.org/abs/ 2201.02177. arXiv:2201.02177 [cs]. 5 Radford, A., Wu, J., Child, R., Luan, D., Am...

    arXiv 2022

  4. [2008]

    18 Huh, M., Cheung, B., Wang, T., and Isola, P

    URL https://proceedings.neurips.cc/ paper/2008/hash/ f7664060cc52bc6f3d620bcedc94a4b6- Abstract.html. 18 Huh, M., Cheung, B., Wang, T., and Isola, P. The Pla- tonic Representation Hypothesis, May 2024. URL http: //arxiv.org/abs/2405.07987. arXiv:2405.07987 [cs]. 1, 2, 3, 18 Hyvarinen, A. and Morioka, H. Unsupervised Feature Ex- traction by Time-Contrastiv...

    arXiv 2008

  5. [2009]

    Platonic ideals

    ISBN 978-0-521-86467-1 978-0-511-80047-4. doi: 10 .1017/CBO9780511800474. URL https: //www.cambridge.org/core/product/ identifier/9780511800474/type/book. 2, 4 Watanabe, S. Mathematical theory of Bayesian statistics . CRC Press, Taylor & Francis Group, Boca Raton London New York, first issued in paperback edition, 2020. ISBN 978-1-4822-3806-8 978-0-367-73...

    arXiv 2024

  6. [2012]

    D., Whitney, W

    2, 7 Kulkarni, T. D., Whitney, W. F., Kohli, P., and Tenenbaum, J. Deep convolutional inverse graphics network. Advances in neural information processing systems, 28, 2015. 2 Kunin, D., Ravent ´os, A., Domin ´e, C., Chen, F., Klindt, D., Saxe, A., and Ganguli, S. Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature l...

    arXiv 2015

  7. [2014]

    The bitter lesson

    3 Sutton, R. The bitter lesson. Incomplete Ideas (blog), 13(1): 38, 2019. 3, 8 Tenenbaum, J. B., Silva, V . d., and Langford, J. C. A global geometric framework for nonlinear dimen- sionality reduction. Science, 290(5500):2319–2323, December 2000. ISSN 1095-9203. doi: 10 .1126/ science.290.5500.2319. URL http://dx.doi.org/ 10.1126/science.290.5500.2319. 7...

    arXiv 2019

  8. [2017]

    ISSN: 2640-3498

    URL http://proceedings.mlr.press/ v54/hyvarinen17a.html. ISSN: 2640-3498. 3 Hyvarinen, A., Karhunen, J., and Oja, E. Independent component analysis. J. Wiley, New York, 2001. ISBN 978-0-471-40540-5. 3 Hyvarinen, A., Sasaki, H., and Turner, R. E. Nonlin- ear ICA Using Auxiliary Variables and Generalized Contrastive Learning. arXiv:1805.08651 [cs, stat] , F...

    arXiv 1984

Show all 12 references
  1. [2019]

    ISSN: 2640-3498

    URL http://proceedings.mlr.press/ v97/locatello19a.html. ISSN: 2640-3498. 3, 8, 18 Locatello, F., Poole, B., R¨atsch, G., Sch¨olkopf, B., Bachem, O., and Tschannen, M. Weakly-Supervised Disentan- glement Without Compromises. arXiv:2002.02886 [cs, stat], October 2020. URL http:...

    2002 arXiv

  2. [2021]

    Guillotine Regularization: Why removing layers is needed to improve generaliza- tion in Self-Supervised Learning

    2 Bordes, F., Balestriero, R., Garrido, Q., Bardes, A., and Vincent, P. Guillotine Regularization: Why removing layers is needed to improve generaliza- tion in Self-Supervised Learning. Transactions on Machine Learning Research , May 2023. ISSN 2835-8856. URL https://openrevie...

    2023 arXiv

  3. [2022]

    URL https: //arxiv.org/abs/2210.11464

    doi: 10.48550/ARXIV.2210.11464. URL https: //arxiv.org/abs/2210.11464. Publisher: arXiv Version Number: 1. 7 Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Sch¨olkopf, B., and Bachem, O. Challenging Common Assumptions in the Unsupervised Learning of Disen- tangle...

    doi arXiv

  4. [2023]

    4 Reizinger, P., Gresele, L., Brady, J., K¨ugelgen, J

    URL https://proceedings.neurips.cc/ paper files/paper/2023/hash/ 2e10b2c2e1aa4f8083c37dfe269873f8- Abstract-Conference.html. 4 Reizinger, P., Gresele, L., Brady, J., K¨ugelgen, J. V ., Zietlow, D., Sch¨olkopf, B., Martius, G., Brendel, W., and Besserve, M. Embrace the Gap: V A...

    2023 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.