Pith. sign in

REVIEW 3 cited by

The Hidden Uniform Cluster Prior in Self-Supervised Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2210.07277 v1 pith:XL42NLYE submitted 2022-10-13 cs.LG cs.AIcs.CV

Mahmoud Assran , Randall Balestriero , Quentin Duval , Florian Bordes , Ishan Misra , Piotr Bojanowski , Pascal Vincent , Michael Rabbat
show 1 more author
Nicolas Ballas
This is my paper · ORCID
classification cs.LGcs.AIcs.CV
keywords datapretrainingpriorclass-imbalanceddemonstratefeatureslearningpriors
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

A successful paradigm in representation learning is to perform self-supervised pretraining using tasks based on mini-batch statistics (e.g., SimCLR, VICReg, SwAV, MSN). We show that in the formulation of all these methods is an overlooked prior to learn features that enable uniform clustering of the data. While this prior has led to remarkably semantic representations when pretraining on class-balanced data, such as ImageNet, we demonstrate that it can hamper performance when pretraining on class-imbalanced data. By moving away from conventional uniformity priors and instead preferring power-law distributed feature clusters, we show that one can improve the quality of the learned representations on real-world class-imbalanced datasets. To demonstrate this, we develop an extension of the Masked Siamese Networks (MSN) method to support the use of arbitrary features priors.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Semantic Concentration for Self-Supervised Dense Representations Learning

    cs.CV 2025-09 conditional novelty 6.0 of 10

    A self-distillation framework with a continuous-target AP loss and an object-aware prototype filter improves dense representations by concentrating semantically similar patches.

  2. Hi-End-MAE: Hierarchical encoder-driven masked autoencoders are stronger vision learners for medical image segmentation

    cs.CV 2025-02 conditional novelty 6.0 of 10

    Hi-End-MAE pre-trains ViT encoders for medical image segmentation by having the decoder query encoder features from multiple layers, improving one-shot segmentation DSC by about 6 points over prior medical SSL methods.

  3. IConE: Batch Independent Collapse Prevention for Self-Supervised Representation Learning

    cs.CV 2026-03 conditional novelty 5.0 of 10

    IConE prevents representation collapse in self-supervised learning by aligning views to a globally regularized per-instance embedding table, making training stable down to batch size 1.

Pith tools