Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

Cluster and Predict Latent Patches for Improved Masked Image Modeling

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that pure masked image modeling can rival contrastive SSL if the loss predicts latent cluster assignments, reporting 83.8% on ImageNet and 32.1% mIoU on ADE20K for a ViT-L backbone.

desk verdict Real advance in pure masked image modeling, but the 'approaching DINOv2' headline leans on an attentive probe — with a linear probe the gap is ~4.7 points, not ~2.7. read the letter →

arxiv 2502.08769 v3 pith:WEXOUBOE submitted 2025-02-12 cs.CV cs.AI

Timothée Darcet , Federico Baldassarre , Maxime Oquab , Julien Mairal , Piotr Bojanowski This is my paper
classification cs.CVcs.AI
keywords maskedimagemodelingself-supervisedlearninglatentclusteringvisiontransformerEMAteacherSinkhorn-KnoppdensefeaturesNet
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that masked image modeling by itself, predicting the content of missing image patches, can produce visual representations that rival the current self-supervised state of the art, without contrastive or DINO-style auxiliary objectives. It introduces CAPI, a framework in which an EMA teacher encodes the full image, clusters its patch representations into pseudo-categories with an online clustering, and the student, seeing only a masked view, is trained to predict those cluster assignments for the missing patches. The authors report that a 300-million-parameter ViT-L trained this way reaches 83.8% top-1 accuracy on ImageNet-1k and 32.1% mIoU on ADE20K with frozen features and simple probes, outperforming previous masked-image-modeling methods and approaching DINOv2. A sympathetic reader would care because the result suggests the gap between reconstruction-based pretraining and contrastive or joint-embedding pretraining may be a matter of target design and loss stability rather than a fundamental limitation of masked modeling.

What carries the argument

The engine of the method is the clustering-based target loss. The teacher's patch embeddings are L2-normalized and projected by a learnable centroid matrix $C \in \mathbb{R}^{p \times d}$ into logits over $p = 16{,}384$ pseudo-categories; a softmax with temperature turns these into soft assignments, and Sinkhorn-Knopp rebalancing makes the assignments near-uniform. The paper's modification is to run Sinkhorn-Knopp separately for each spatial position, so the joint distribution of tokens over positions and clusters is uniform and the targets carry no positional information. The student predicts those assignments for masked patches using a cross-attention predictor that processes only mask tokens, making each prediction independent and cheap.

What would settle it

Train CAPI with the published recipe but replace the per-position Sinkhorn-Knopp normalization with the standard global version; if ImageNet-1k attentive-probe accuracy does not drop and no positional collapse appears, the paper's stated stability mechanism is not what carries the result. Alternatively, evaluate DINOv2 with the same attentive-probe protocol on its patch tokens and CAPI with a linear probe on a global pooled vector; if CAPI does not still approach DINOv2 under both protocols, the headline comparison rests on measurement choice.

Watch

Extended reading notes

Core claim

The central claim is that a pure masked-image-modeling objective can bootstrap a high-quality visual encoder if the reconstruction target is a latent clustering of patch tokens rather than pixels, raw latent vectors, or per-token MLP projections. Concretely, CAPI computes soft cluster assignments over the 196 patch tokens of an EMA teacher using a learned linear projection onto 16,384 prototypes, rebalanced by Sinkhorn-Knopp; the student, which sees only 69 of 196 patches, predicts these assignments for a sample of masked positions through a cross-attention predictor and a cross-entropy loss. The paper argues that this restores a valid clustering interpretation of the loss that iBOT's masked-token-versus-patch-token setup breaks, and that running Sinkhorn-Knopp separately per position removes positional information from the targets and prevents positional collapse. With this recipe, CAPI's ViT-L improves over all prior reconstruction-based models of comparable size on classification and segmentation, reduces the ImageNet gap to DINOv2 from 8.4 to 3.6 points when both are trained on large data, and even surpasses DINOv2+reg on ADE20K k-NN segmentation when trained on Places205.

Load-bearing premise

The paper's main comparison assumes that its chosen way of measuring frozen features, pooling patch-level outputs with a learned attention head rather than using each model's own global summary token, treats CAPI and DINOv2 even-handedly.

Editorial extensions

If this is right

  • CAPI narrows the gap between pure reconstruction-based methods and DINOv2 on ImageNet from 8.4 to 3.6 points when both are trained on large data, implying that masked modeling alone can account for most of the performance of a state-of-the-art SSL model.
  • Because the clustering loss trains stably without a DINO or contrastive term, large-scale SSL pretraining may no longer need to juggle multiple objectives whose heads can interfere.
  • Scaling model size, training length, and dataset size each improve CAPI's scores, so the recipe offers a path toward pushing reconstruction-based models further at larger scales.
  • CAPI's frozen local features exceed DINOv2+reg on ADE20K k-NN segmentation when trained on Places205, demonstrating that reconstruction-based models can lead on dense prediction even if they lag on global classification.
  • The cross-attention predictor's independence between mask-token predictions removes the need for multiple predictor passes, making the method more efficient as well as more accurate than fused or self-attention predictors.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One implication the authors leave implicit is that the per-position Sinkhorn-Knopp trick should transfer to other dense self-supervised targets, such as video, audio, or multi-view patch prediction, wherever positional information can leak into targets and collapse representations.
  • The measured gap may depend on probe choice: CAPI's local patch features could be unusually well matched to attentive pooling, so evaluating DINOv2 under the same patch-token attentive probe, or CAPI under a linear probe on a global token, would disentangle representation quality from evaluation protocol.
  • The learnable clustering can be read as a continuously improving tokenizer, unlike methods that rely on a fixed pretrained tokenizer; if so, the same EMA-plus-clustering scheme might bootstrap other token-based modalities from scratch.
  • Because the paper reports smooth, object-focused dense features, one plausible untested use is as an input to dense downstream tasks such as depth estimation, tracking, or keypoint matching.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes CAPI, a masked image modeling method that trains a student ViT to predict cluster assignments of masked patches, where the cluster assignments are computed online from the patch embeddings of an EMA teacher via a learned prototype layer and a modified Sinkhorn-Knopp rebalancing. The authors justify a cross-attention predictor architecture, a per-position Sinkhorn-Knopp step to remove positional information from the targets, and a clustering loss replacing direct latent or MLP-head losses. Extensive ablations on ImageNet-22k, multi-dataset training (IN-1k, IN-22k, Places205, LVD-142M), and evaluations on ImageNet variants, iNaturalist, Places, SUN, ADE20K, Pascal VOC, and Cityscapes support the claim that CAPI substantially outperforms prior pure-MIM baselines and narrows the gap to DINOv2.

Significance. If the central claim holds, the paper is a meaningful advance in pure masked image modeling: it shows that an online clustering target, combined with a cross-attention predictor and a positional-collapse mitigation, yields frozen representations that beat prior MIM methods and approach a strong contrastive baseline (DINOv2). The work is well structured around three design axes, and the experimental coverage is broad, including scaling runs, OOD robustness, dense prediction, resolution transfer, and a careful ablation of the predictor architecture, masking, crop, prototypes, registers, positional encoding, and Sinkhorn-Knopp variant. The authors also release code and models, which increases reproducibility. The main weakness is that the headline 'approaching DINOv2 with simple linear probes' relies on an attentive-probe evaluation for CAPI, while the linear-probe numbers in the paper show a larger gap.

major comments (3)
  1. [Abstract and Section 4.1 / Appendix E, Table 6] The abstract states that CAPI achieves 83.8% ImageNet accuracy 'with simple linear probes', but that number is obtained with an attentive probe, not a linear probe. Table 6 in Appendix E shows that with a fixed predictor pooling and a linear head, CAPI reaches 81.8% on ImageNet, while the attentive probe gives 83.8%. This difference of 2.0 points is material for the headline comparison with DINOv2: the 2.7-point gap implied by the attentive-probe comparison becomes a 4.7-point gap under a consistent linear-probe evaluation. The authors should either report the linear-probe number in the abstract and abstract-level claims, or explicitly state that the 83.8% figure is obtained with a learned attentive pooling head.
  2. [Section 4.1, Evaluation protocol] The paper justifies the attentive probe by saying the model 'does not learn a single global image representation, preventing the use of a linear probe'. This is contradicted by Table 6, where the predictor pooling produces a fixed global vector that supports linear probing at 81.8%. Since a linear probe is possible, the choice of the attentive probe as the main classification protocol inflates the reported numbers relative to standard linear-probe benchmarks. The authors should present the linear-probe results as the primary evaluation, or at minimum report both numbers in all headline tables, and temper the 'simple linear probes' wording in the abstract.
  3. [Section 4.3, Table 2, and Appendix H.4] It is unclear whether the grey-reference rows (iBOT, MIM-refiner, DINOv2+reg) in Table 2 were re-evaluated with the same attentive-probe protocol as the main baselines, or whether their scores are taken from the original publications. The text says 'we report in grey the performance of other SSL models', which suggests the latter, but the same table then compares CAPI's attentive-probe score against these numbers. If DINOv2's 86.5% is its published CLS linear-probe result, the comparison is not apples-to-apples. The authors must state explicitly, for every row of the table, which numbers were recomputed with the paper's evaluation protocol and which are cited from prior work, and provide attentive-probe results for DINOv2 if they exist.
minor comments (4)
  1. [Section 3.1] The sentence 'Uniformity of the joint distribution directly implies zero mutual information between the clustering and the targets' is ambiguous; the intended statement is that the modified Sinkhorn-Knopp makes the cluster assignments independent of patch positions, so the mutual information between assignments and positions is zero. Please rephrase.
  2. [Section 4.1, Implementation Details] The definition 'teacher EMA momentum to μ = 1−lr' is ambiguous because 'lr' could mean the peak learning rate or the current scheduled learning rate. State clearly which one is used.
  3. [Appendix H.1] There is a typo in 'two learned k anv v projections' (should be 'k and v') and the notation 'head sized//64' is non-standard; write the head dimension explicitly as d_model / 64.
  4. [Section 4.2, Predictor shape] The claim that shallow predictors 'are less stable in long schedules' is supported only by 'informal experiments'; please specify the experimental setting or remove the informal reference.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: EMA-bootstrapped cluster targets are the method's acknowledged design, and headline results come from external frozen-feature benchmarks, so the central claim is not equivalent to its inputs.

full rationale

The derivation chain is self-contained against external benchmarks. The training targets are cluster assignments produced by an online clustering over the EMA teacher's patch embeddings (Eqs. 1-5), and the student is trained to predict those assignments for masked patches. This is an acknowledged self-distillation/bootstrap design (intro: 'bootstrapping an informative latent space from scratch'; Appendix D: 'this interpretation still seems to hold in CAPI'), not a hidden equivalence: the paper makes no claim that the cluster assignments are external ground truth, and the value of the learned representations is established by frozen-feature evaluations on ImageNet-1k, ADE20K, and other datasets, which are not functions of the training loss. Hyperparameters were selected by ablations on IN-22k and validation splits, which is standard model selection rather than a fitted-input-called-prediction reduction. The few self-citations (registers from Darcet et al. 2024; DINOv2 as topline) are not load-bearing: registers are ablated in Table 1g, and DINOv2 is an external baseline. The abstract's 'simple linear probes' wording is inaccurate, since 83.8% is obtained with an attentive probe (Sec 4.1, Table 6) while linear probing with predictor pooling gives 81.8%, but that is a reporting/protocol concern, not circularity. No step in the paper reduces by construction to its own inputs.

Assumptions & free parameters 12 free parameters · 6 assumptions · 0 invented entities

The method introduces no external invented entity such as a new force or particle; the prototype matrix and pseudo-categories are internal learned parameters and are counted as free parameters. The main conceptual inputs are standard Sinkhorn-Knopp convergence, an information-theoretic fact about uniform joint distributions, and domain assumptions about EMA bootstrapping, transfer of ablations, and fairness of the evaluation protocol.

free parameters (12)
  • Number of clustering prototypes K = 16384
    Ablated in fig. 7; larger K improves ImageNet and ADE scores until saturation around 16384, chosen as a memory/performance tradeoff.
  • Student softmax temperature = 0.12
    Set in table 7; controls sharpness of predicted cluster assignments.
  • Teacher softmax temperature = 0.06
    Set in table 7; controls sharpness of the teacher assignment distribution used as target.
  • Sinkhorn-Knopp temperature tau' = not reported
    Equation (4) introduces tau' as the temperature for SK assignments, but table 7 does not list a value, leaving a tuning knob unspecified.
  • Masking ratio = 65%
    Ablated in table 1e; 65% beats 55% and 75% on ImageNet attentive probing.
  • Crop scale lower bound = 0.6
    Ablated in table 1d; [0.6, 1.0] is best among [0.2, 1.0], [0.6, 1.0] and [1.0, 1.0].
  • Predictor depth and width = 12 layers, 1024 width
    Ablated in table 1f; a shallower predictor scores higher but is less stable in long schedules, so 12 layers are chosen.
  • Number of register tokens = 16
    Ablated in table 1g; registers give +3.2 ADE and +2.1 ImageNet points and are used in all final models.
  • Batch size = 16384
    Appendix C says this is used throughout and that smaller batches may need recipe adjustments; no results at smaller batch size are provided.
  • Learning rate, teacher momentum, clustering learning rate = 1e-3, 1-lr, 0.5*lr
    Set in table 7; teacher momentum is tied to the learning rate schedule to simplify tuning.
  • Weight decay and stochastic depth = 0.1 and 0.2
    Fixed in table 7; not ablated in the paper.
  • RoPE frequencies = logspace(7e-4, 7), axial
    Fixed in table 7; chosen for ease of resolution transfer but not ablated.
assumptions (6)
  • standard math Sinkhorn-Knopp converges to a near-uniform doubly stochastic assignment matrix.
    Relled on in equation (4) and in the modified per-position SK used to make cluster assignments balanced.
  • standard math A uniform joint distribution over positions and clusters implies zero mutual information between cluster assignments and position.
    Used in Section 3.1 to argue that the modified SK removes positional information from targets and prevents positional collapse.
  • domain assumption The EMA teacher's patch representations contain enough semantic structure for online clustering to produce useful targets.
    Core bootstrap assumption of the method; if the teacher collapses or targets are trivial, the student would learn nothing useful.
  • domain assumption Ablation conclusions on ImageNet-22k with patch size 16 and 100k iterations transfer to the final ViT-L/14 configs trained on LVD-142M and other datasets.
    Final hyperparameters are inherited from the ablation configs, but most ablations are shorter and lower-resolution.
  • domain assumption The evaluation protocol with attentive probes, k-NN and standardized logistic-regression probes is a fair comparison across very different SSL models.
    Section 4.1 and Appendix H define a uniform protocol, but models such as DINOv2 were designed around a CLS token and linear probing; the protocol may favor local-feature models.
  • ad hoc to paper Removing positional mutual information from cluster assignments is sufficient to stop positional collapse without removing useful semantic signal.
    The modified per-position SK is proposed empirically to fix a failure mode observed during training; no proof is given that it preserves all useful content information.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cluster and Predict Latent Patches for Improved Masked Image Modeling." pith.science (2026). https://pith.science/paper/WEXOUBOE

@misc{pith2026250208769,
  author       = {Pith},
  title        = {Pith review of: Cluster and Predict Latent Patches for Improved Masked Image Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WEXOUBOE}},
  note         = {Machine review of arXiv:2502.08769}
}
read the original abstract

Masked Image Modeling (MIM) offers a promising approach to self-supervised representation learning, however existing MIM models still lag behind the state-of-the-art. In this paper, we systematically analyze target representations, loss functions, and architectures, to introduce CAPI - a novel pure-MIM framework that relies on the prediction of latent clusterings. Our approach leverages a clustering-based loss, which is stable to train, and exhibits promising scaling properties. Our ViT-L backbone, CAPI, achieves 83.8% accuracy on ImageNet and 32.1% mIoU on ADE20K with simple linear probes, substantially outperforming previous MIM methods and approaching the performance of the current state-of-the-art, DINOv2. We release all our code and models.

Figures

Figures reproduced from arXiv: 2502.08769 by the authors.

Figure 1
Figure 1. CAPI Method overview: image patches embedded by a EMA encoder are grouped into clusters. Their assignments are then used as the training signal for the online model. The loss is purely about predicting the content of missing patches and does not rely on augmentations or a contrastive loss. Evaluation scores: we evaluate frozen representations on ADE20K segmentation with a k-nn and linear probe and on ImageNet-1k cla… view at source ↗
Figure 2
Figure 2. Detailed overview of our method with reference tensor sizes for a ViT-L/16 model. We denote in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of the components of a reconstruction-based model. We identify three main choices [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: The target representations commonly used in MIM. (a) In the simplest case, the pixel values of the [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: The different loss formulations considered here. We depict in red the flow of the gradient. (a) The [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: The different predictor architectures discussed in the paper. Here, each box represents a transformer. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Additional ablation experiments. (Left) Influence of the number of prototypes. (center) Influence of the training length. Each point here is an independent training. (right) Influence of the training dataset. recent observations on tabular data, for which registers wer…
Figure 8
Figure 8. Figure 8: Visualization of the features of CAPI and baseline models. We apply a PCA to the features and [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: The loss curve of our CAPI ViT-L during training [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 11
Figure 11. Figure 11: PCA visualization of the patch embeddings of some early CAPI models, showing partial positional [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: PyTorch pseudo-code for the proposed modified Sinkhorn-Knopp algorithm. We normalize by the [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: Visualization of the features produced by CAPI and other vision models at various resolutions: [PITH_FULL_IMAGE:figures/full_fig_p025_13.png]
Figure 14
Figure 14. Figure 14: Visualization of the features produced by CAPI ViT-L/14 applied to images at 560 pixel resolution. [PITH_FULL_IMAGE:figures/full_fig_p026_14.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Separating Representation from Reconstruction Enables Scalable Text Encoders

    cs.CL 2026-07 accept novelty 6.5 of 10

    Separating representation from token reconstruction via a bipartite CrossBERT architecture restores scalable frozen text embeddings and enables high-masking complementary training.

  2. Feed-Forward SceneDINO for Unsupervised Semantic Scene Completion

    cs.CV 2025-07 conditional novelty 6.0 of 10

    SceneDINO performs semantic scene completion from a single image in a fully unsupervised way by lifting self-supervised DINO features into a 3D feature field trained with multi-view consistency.

Reference graph

Works this paper leans on

74 extracted references · 46 canonical work pages · cited by 2 Pith papers

  1. [1]

    Mim-refiner: A contrastive learning boost from intermediate pre-trained representations, 2024

    Benedikt Alkin, Lukas Miklautz, Sepp Hochreiter, and Johannes Brandstetter. Mim-refiner: A contrastive learning boost from intermediate pre-trained representations, 2024

    work page 2024

  2. [2]

    Self-labelling via simultaneous clustering and representation learning

    Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. In ICLR, 2020

    2020

  3. [3]

    Masked siamese networks for label-efficient learning

    Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. In ECCV, 2022

    work page 2022

  4. [4]

    Self-supervised learning from images with a joint-embedding predictive architecture

    Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In CVPR, 2023

    2023

  5. [5]

    data2vec: A general framework for self-supervised learning in speech, vision and language

    Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. data2vec: A general framework for self-supervised learning in speech, vision and language. In ICML, 2022

    work page 2022

  6. [6]

    Efficient self-supervised learning with contextualized target representations for vision, speech and language

    Alexei Baevski, Arun Babu, Wei-Ning Hsu, and Michael Auli. Efficient self-supervised learning with contextualized target representations for vision, speech and language. In ICML, 2023

    2023

  7. [7]

    Beit: Bert pre-training of image transformers

    Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. In ICLR, 2021

    work page 2021

  8. [8]

    Stochastic positional embeddings improve masked image modeling

    Amir Bar, Florian Bordes, Assaf Shocher, Mido Assran, Pascal Vincent, Nicolas Ballas, Trevor Darrell, Amir Globerson, and Yann LeCun. Stochastic positional embeddings improve masked image modeling. In ICML, 2024

    work page 2024

Show all 74 references
  1. [9]

    Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models

    Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In NeurIPS, 2019

    2019

  2. [10]

    V-jepa: Latent video prediction for visual representation learning, 2023

    Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mido Assran, and Nicolas Ballas. V-jepa: Latent video prediction for visual representation learning, 2023

    2023

  3. [11]

    H \' e naff, Alexander Kolesnikov, Xiaohua Zhai, and A \" a ron van den Oord

    Lucas Beyer, Olivier J. H \' e naff, Alexander Kolesnikov, Xiaohua Zhai, and A \" a ron van den Oord. Are we done with imagenet? CoRR, abs/2006.07159, 2020. URL https://arxiv.org/abs/2006.07159

    2006 arXiv

  4. [12]

    Depth pro: Sharp monocular metric depth in less than a second, 2024

    Aleksei Bochkovskii, Ama \"e l Delaunoy, Hugo Germain, Marcel Santos, Yichao Zhou, Stephan R Richter, and Vladlen Koltun. Depth pro: Sharp monocular metric depth in less than a second, 2024

    2024

  5. [13]

    A limited memory algorithm for bound constrained optimization

    Richard H Byrd, Peihuang Lu, Jorge Nocedal, and Ciyou Zhu. A limited memory algorithm for bound constrained optimization. SIAM, 1995

    1995

  6. [14]

    Deep clustering for unsupervised learning of visual features

    Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In ECCV, 2018

    2018

  7. [15]

    Unsupervised learning of visual features by contrasting cluster assignments

    Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS, 2020

    2020

  8. [16]

    Emerging properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv \'e J \'e gou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021

    2021

  9. [17]

    Generative pretraining from pixels

    Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML, 2020 a

    2020

  10. [18]

    Towards a general-purpose foundation model for computational pathology

    Richard J Chen, Tong Ding, Ming Y Lu, Drew FK Williamson, Guillaume Jaume, Andrew H Song, Bowen Chen, Andrew Zhang, Daniel Shao, Muhammad Shaban, et al. Towards a general-purpose foundation model for computational pathology. Nature Medicine, 2024

    2024

  11. [19]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020 b

    2020

  12. [20]

    An empirical study of training self-supervised vision transformers

    Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In ICCV, 2021

    2021

  13. [21]

    The cityscapes dataset for semantic urban scene understanding

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016

    2016

  14. [22]

    Vision transformers need registers

    Timoth \'e e Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In ICLR, 2024

    2024

  15. [23]

    Endodino: A foundation model for gi endoscopy, 2025

    Patrick Dermyer, Angad Kalra, and Matt Schwartz. Endodino: A foundation model for gi endoscopy, 2025

    2025

  16. [24]

    BERT: pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming - Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, 2018

    2018

  17. [25]

    Unsupervised visual representation learning by context prediction

    Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In ICCV, 2015

    2015

  18. [26]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021

    2021

  19. [27]

    o kman, M rten Wadenb \

    Johan Edstedt, Qiyu Sun, Georg B \"o kman, M rten Wadenb \"a ck, and Michael Felsberg. Roma: Robust dense feature matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 19790--19800, 2024

    2024

  20. [28]

    Scalable pre-training of large autoregressive image models

    Alaaeldin El-Nouby, Michal Klein, Shuangfei Zhai, Miguel Angel Bautista, Alexander Toshev, Vaishaal Shankar, Joshua M Susskind, and Armand Joulin. Scalable pre-training of large autoregressive image models. In ICML, 2024

    2024

  21. [29]

    The pascal visual object classes (voc) challenge

    Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV, 2010

    2010

  22. [30]

    Eva: Exploring the limits of masked visual representation learning at scale

    Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In CVPR, 2023

    2023

  23. [31]

    Eva-02: A visual representation for neon genesis

    Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. Image and Vision Computing, 2024

    2024

  24. [32]

    Rethinking patch dependence for masked autoencoders, 2024

    Letian Fu, Long Lian, Renhao Wang, Baifeng Shi, Xudong Wang, Adam Yala, Trevor Darrell, Alexei A Efros, and Ken Goldberg. Rethinking patch dependence for masked autoencoders, 2024

    2024

  25. [33]

    Unsupervised representation learning by predicting image rotations

    Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR, 2018

    2018

  26. [34]

    Moca: Self-supervised representation learning by predicting masked online codebook assignments

    Spyros Gidaris, Andrei Bursuc, Oriane Simeoni, Anton \' n Vobeck \`y , Nikos Komodakis, Matthieu Cord, and Patrick P \'e rez. Moca: Self-supervised representation learning by predicting masked online codebook assignments. TMLR, 2024

    2024

  27. [35]

    Momentum contrast for unsupervised visual representation learning

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020

    2020

  28. [36]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll \'a r, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022

    2022

  29. [37]

    Natural adversarial examples

    Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, 2021

    2021

  30. [38]

    Contrastive masked autoencoders are stronger vision learners

    Zhicheng Huang, Xiaojie Jin, Chengze Lu, Qibin Hou, Ming-Ming Cheng, Dongmei Fu, Xiaohui Shen, and Jiashi Feng. Contrastive masked autoencoders are stronger vision learners. TPAMI, 2023

    2023

  31. [39]

    J. D. Hunter. Matplotlib: A 2d graphics environment. Computing in Science & Engineering, 2007

    2007

  32. [40]

    Adam: A method for stochastic optimization

    Diederik P Kingma. Adam: A method for stochastic optimization. In ICLR, 2014

    2014

  33. [41]

    A path towards autonomous machine intelligence, 2022

    Yann LeCun. A path towards autonomous machine intelligence, 2022

    2022

  34. [42]

    Deepseek-v3 technical report, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report, 2024

    2024

  35. [43]

    Decoupled weight decay regularization, 2017

    I Loshchilov. Decoupled weight decay regularization, 2017

    2017

  36. [44]

    Torchvision: Pytorch's computer vision library

    maintainers and TorchVision contributors. Torchvision: Pytorch's computer vision library. https://github.com/pytorch/vision, 2016

    2016

  37. [45]

    Self-supervised learning of pretext-invariant representations

    Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. In CVPR, 2020

    2020

  38. [46]

    Advancing human-centric ai for robust x-ray analysis through holistic self-supervised learning, 2024

    Th \'e o Moutakanni, Piotr Bojanowski, Guillaume Chassagnon, C \'e line Hudelot, Armand Joulin, Yann LeCun, Matthew Muckley, Maxime Oquab, Marie-Pierre Revel, and Maria Vakalopoulou. Advancing human-centric ai for robust x-ray analysis through holistic self-supervised learning, 2024

    2024

  39. [47]

    Representation learning with contrastive predictive coding

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. preprint arXiv:1807.03748, 2018

    2018 arXiv

  40. [48]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth \'e e Darcet, Th \'e o Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. TMLR, 2024

    2024

  41. [49]

    Context encoders: Feature learning by inpainting

    Deepak Pathak, Philipp Kr\"ahenb\"uhl, Jeff Donahue, Trevor Darrell, and Alexei Efros. Context encoders: Feature learning by inpainting. In CVPR, 2016

    2016

  42. [50]

    Improving language understanding by generative pre-training, 2018

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018

    2018

  43. [51]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML, 2021

    2021

  44. [52]

    Machine learning in python: Main developments and technology trends in data science, machine learning, and artificial intelligence, 2020

    Sebastian Raschka, Joshua Patterson, and Corey Nolet. Machine learning in python: Main developments and technology trends in data science, machine learning, and artificial intelligence, 2020

    2020

  45. [53]

    Do imagenet classifiers generalize to imagenet? In ICML, 2019

    Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML, 2019

    2019

  46. [54]

    Imagenet large scale visual recognition challenge

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. IJCV, 2015

    2015

  47. [55]

    Sigma: Sinkhorn-guided masked video modeling

    Mohammadreza Salehi, Michael Dorkenwald, Fida Mohammad Thoker, Efstratios Gavves, Cees GM Snoek, and Yuki M Asano. Sigma: Sinkhorn-guided masked video modeling. In ECCV, 2024

    2024

  48. [56]

    Web-scale k-means clustering

    David Sculley. Web-scale k-means clustering. In WWW, 2010

    2010

  49. [57]

    Concerning nonnegative matrices and doubly stochastic matrices

    Richard Sinkhorn and Paul Knopp. Concerning nonnegative matrices and doubly stochastic matrices. Pacific Journal of Mathematics, 1967

    1967

  50. [58]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 2024

    2024

  51. [59]

    T-jepa: Augmentation-free self-supervised learning for tabular data

    Hugo Thimonier, Jos \'e Lucas De Melo Costa, Fabrice Popineau, Arpad Rimmel, and Bich-Li \^e n Doan. T-jepa: Augmentation-free self-supervised learning for tabular data. arXiv preprint arXiv:2410.05016, 2024

    2024 arXiv

  52. [60]

    Very high resolution canopy height maps from rgb imagery using self-supervised vision transformer and convolutional decoder trained on aerial lidar

    Jamie Tolan, Hung-I Yang, Benjamin Nosarzewski, Guillaume Couairon, Huy V Vo, John Brandt, Justine Spore, Sayantan Majumdar, Daniel Haziza, Janaki Vamaraju, et al. Very high resolution canopy height maps from rgb imagery using self-supervised vision transformer and convolution...

    2024

  53. [61]

    Dino-tracker: Taming dino for self-supervised point tracking in a single video

    Narek Tumanyan, Assaf Singer, Shai Bagon, and Tali Dekel. Dino-tracker: Taming dino for self-supervised point tracking in a single video. In ECCV, 2025

    2025

  54. [62]

    Benchmarking representation learning for natural world image collections

    Grant Van Horn, Elijah Cole, Sara Beery, Kimberly Wilber, Serge Belongie, and Oisin Mac Aodha. Benchmarking representation learning for natural world image collections. In CVPR, 2021

    2021

  55. [63]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017

    2017

  56. [64]

    Vincent, H

    P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol. Extracting and composing robust features with denoising autoencoders. In ICML, 2008

    2008

  57. [65]

    A foundation model for clinical-grade computational pathology and rare cancers detection

    Eugene Vorontsov, Alican Bozkurt, Adam Casson, George Shaikovski, Michal Zelechowski, Kristen Severson, Eric Zimmermann, James Hall, Neil Tenenholtz, Nicolo Fusi, et al. A foundation model for clinical-grade computational pathology and rare cancers detection. Nature medicine, 2024

    2024

  58. [66]

    Sun database: Large-scale scene recognition from abbey to zoo

    Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010

    2010

  59. [67]

    Simmim: A simple framework for masked image modeling

    Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. arXiv, 2021

    2021

  60. [68]

    A whole-slide foundation model for digital pathology from real-world data

    Hanwen Xu, Naoto Usuyama, Jaspreet Bagga, Sheng Zhang, Rajesh Rao, Tristan Naumann, Cliff Wong, Zelalem Gero, Javier Gonz \'a lez, Yu Gu, et al. A whole-slide foundation model for digital pathology from real-world data. Nature, 2024

    2024

  61. [69]

    Depth anything: Unleashing the power of large-scale unlabeled data

    Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR, 2024 a

    2024

  62. [70]

    Depth anything v2, 2024 b

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2, 2024 b

    2024

  63. [71]

    Colorful image colorization

    Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In ECCV, 2016

    2016

  64. [72]

    Cae v2: Context autoencoder with clip target, 2022

    Xinyu Zhang, Jiahui Chen, Junkun Yuan, Qiang Chen, Jian Wang, Xiaodi Wang, Shumin Han, Xiaokang Chen, Jimin Pi, Kun Yao, et al. Cae v2: Context autoencoder with clip target, 2022

    2022

  65. [73]

    Scene parsing through ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017

    2017

  66. [74]

    ibot: Image bert pre-training with online tokenizer

    Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. In ICLR, 2022

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.