Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Simplifying DINO via Coding Rate Regularization

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Most of DINO/DINOv2's hand-tuned anti-collapse machinery can be replaced by a single explicit coding-rate penalty, producing simpler models that train more stably and transfer better.

desk verdict Coding-rate regularization is a promising simplification of DINO/DINOv2, but the Pareto-improvement claim is confounded by simultaneous hyperparameter changes. read the letter →

arxiv 2502.10385 v1 pith:Z2INREGN submitted 2025-02-14 cs.CV cs.AI

Ziyang Wu , Jingyuan Zhang , Druv Pai , XuDong Wang , Chandan Singh , Jianwei Yang , Jianfeng Gao , Yi Ma This is my paper · ORCID
classification cs.CVcs.AI
keywords self-supervisedlearningrepresentationcollapsecodingrateDINODINOv2VisionTransformerImageNetpretrainingdistillation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the complexity of the DINO and DINOv2 pretraining pipelines is not doing essential representational work: most of their hand-tuned components exist to stop the student representation from collapsing. It replaces the heads, centering, temperature softmaxes, Sinkhorn-Knopp centering, and KoLeo entropy with a single explicit regularizer, the total coding rate of the student's class-token covariance, paired with plain squared-Euclidean alignment to the teacher. The resulting SimDINO and SimDINOv2 are claimed to be more robust to hyperparameter and architecture choices and to learn representations that transfer better to classification, segmentation, and video object segmentation, i.e., a Pareto improvement over the original families. The broader appeal is that making collapse-avoidance an explicit term in the loss, rather than an emergent effect of training mechanics, both stabilizes pretraining and gives theory a concrete object to analyze.

What carries the argument

The carrying object is the total coding rate regularizer applied to the student's class-token batch covariance, defined as $R_\varepsilon(\Gamma)=\tfrac12\log\det(I+\frac{d}{\varepsilon^2}\Gamma)$; conceptually it measures the size of the feature covariance, so penalizing its negative keeps features from collapsing to a point. It replaces the implicit anti-collapse role played by DINO's centering-plus-temperature tradeoff and by DINOv2's Sinkhorn-Knopp and KoLeo entropy. The companion mechanism is the plain squared Euclidean distance on $\ell^2$-normalized features, whose gradient norm is $O(1)$, while Theorem C.1 bounds the coding-rate gradient by $\sqrt{d\min\{d,n\}/n}/(4\varepsilon)$, giving the scaling rule $\gamma=\Theta(\varepsilon\sqrt{n/(d\min\{d,n\})})$ that balances the two loss terms.

What would settle it

Train SimDINO with the recommended $\gamma$ scaling on a ViT-B backbone while sweeping batch size from 64 to 1024 and the covariance sub-sampling frequency; if any setting inside the recommended range collapses to near-chance k-NN accuracy or shows a sharp cliff, the sufficiency claim fails. Separately, estimate the variance of the sub-sampled covariance estimator versus the naive batch plug-in at each batch size: if sub-sampling is not lower-variance, the paper's stated explanation for small-batch robustness is unsupported.

Watch

Extended reading notes

Core claim

The central claim is that the collapse-prevention machinery in DINO and DINOv2 can be deleted wholesale and replaced by one term: a penalization of the total coding rate $R_\varepsilon(\Gamma)=\tfrac12\log\det(I+\frac{d}{\varepsilon^2}\Gamma)$ of the student's class-token batch covariance $\Gamma$, which measures how spread out the features are and approximates the rate-distortion of a Gaussian with that covariance. SimDINO minimizes $\mathbb{E}[d_{\ell^2}(z^{cls}_c(\theta_s),z^{cls}_g(\theta_t))]-\gamma R_\varepsilon(\operatorname{Cov}[z^{cls}_c(\theta_s)])$ on $\ell^2$-normalized features, and SimDINOv2 adds the analogous patch-token Euclidean alignment under masking while keeping the same class-token coding rate. With the coding rate in the loss, the teacher's EMA centering, the high-dimensional output heads, the temperature-weighted softmaxes, the Sinkhorn-Knopp iterations, and the KoLeo entropy estimator are no longer needed to avoid collapse. Empirically, the simplified losses train stably where DINO diverges, support smaller batch sizes, and produce higher downstream accuracy on ImageNet-1K, ADE20K, COCO, and DAVIS-2017. The paper also derives a gradient-norm bound for the coding rate term and uses it to motivate the scaling rule $\gamma=\Theta(\varepsilon\sqrt{n/(d\min\{d,n\})})$.

Load-bearing premise

The load-bearing premise is empirical: a single coding-rate penalty on the student class-token covariance, plus plain squared-Euclidean alignment, is sufficient to avoid representation collapse once all the original anti-collapse machinery is removed, and the paper offers stability experiments rather than a proof that this holds across architectures and batch sizes.

Editorial extensions

If this is right

  • DINO-style pretraining can shed its weight-normalized projection heads, teacher centering and EMA centering vector, temperature schedules, and symmetrized cross-entropy loss without collapse, because the coding-rate penalty enforces non-collapse directly.
  • The same simplification transfers to the masked patch-prediction setting: SimDINOv2 keeps only the masked-view squared-Euclidean patch loss plus the class-token coding rate and outperforms DINOv2's full pipeline.
  • Training becomes robust enough that ViT-L can be trained with ViT-B's hyperparameters where DINO diverges, and batch sizes down to 256 still give usable representations.
  • Downstream tasks improve as well: SimDINO(v2) beats DINO(v2) on ImageNet k-NN and linear probes, ADE20K linear segmentation (up to +4.4 mIoU on ViT-B), DAVIS video object segmentation, and MaskCut detection and segmentation on COCO.
  • Because the objective is explicit, self-distillation itself can be dropped: with the student as its own teacher, SimDINO still learns non-trivial ViT-S representations while DINO collapses.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the sufficiency claim holds beyond ViT image backbones, the same recipe—alignment plus total coding rate on the joint embedding's covariance—could replace bespoke anti-collapse heads and schedulers in other SSL pipelines, such as video or multimodal joint-embedding models, and this is a direct testable extension the paper does not run.
  • The paper's variance-reduction conjecture for the sub-sampled covariance estimator implies a quantitative prediction: the relative benefit of sub-sampling over the naive batch plug-in should grow as batch size shrinks, and a Monte Carlo comparison of estimator variances across batch sizes would settle it.
  • An explicit coding-rate loss also makes the geometry of the solution testable: at convergence, the class-token covariance should have a spectrum consistent with a well-spread, low-dimensional feature distribution, so measuring the eigenvalues of $\operatorname{Cov}[z^{cls}]$ could check whether the regularizer is actually binding or merely a stability aid.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes SimDINO and SimDINOv2, simplified variants of DINO and DINOv2, in which the DINO/iBOT heads, centering, temperature softmaxes, Sinkhorn-Knopp centering, and KoLeo entropy are removed and replaced by a squared-Euclidean alignment term plus an explicit total coding rate regularizer on the student class-token covariance (losses (9) and (17)). The authors report that the simplified models match or exceed the original families on ImageNet-1k k-NN and linear classification, COCO MaskCut detection/segmentation, ADE20K linear segmentation, and DAVIS video object segmentation, and that they are more stable, including a no-self-distillation variant (Table 8). A gradient-norm bound (Theorem C.1) is provided to guide the selection of the regularization strength gamma.

Significance. If the empirical claim is upheld, the paper is a valuable conceptual contribution: it replaces a fragile cascade of heuristics with one explicit, interpretable regularizer, and the simplification could ease adaptation of DINO-style pretraining to new architectures and domains. The paper has concrete strengths: the coding-rate term is a principled collapse prevention mechanism with prior theoretical grounding; the no-self-distillation result (Table 8) is a striking demonstration that the coding-rate term can replace some of the EMA machinery; and Theorem C.1 is a correct, useful scaling bound, even if the constant is heuristic. However, the central empirical claim of a Pareto improvement is not currently cleanly supported because the comparisons in Table 4 change many hyperparameters and architectural details at once, so the reported gains cannot be attributed to the coding-rate simplification. The robustness claims are also supported only by asymmetric experiments. These issues are fixable with matched ablations, but they are load-bearing for the paper's main message.

major comments (3)
  1. [Table 4; Tables 1-3] The headline comparison is confounded: the SimDINOv2 and DINOv2 runs differ simultaneously in register tokens (4 vs 0), positional-embedding anti-alias (True vs False), init layer scale (0.1 vs 1e-5), drop path rate (0.3 vs 0.1), batch size (128x8 vs 64x8), learning rate (0.004 vs 0.002), gradient clip (3.0 vs 0.3), and layerwise lr decay (0.9 vs absent), while SimDINO also doubles the batch size and changes the learning rate relative to DINO. Since these factors are known to affect self-supervised representation quality substantially, Tables 1-3 cannot establish that the coding-rate regularizer, or the removal of the DINO-specific components, causes the observed improvements. The authors should provide matched comparisons that vary only the loss and pipeline components, and ideally also an ablation that adds the coding-rate regularizer to an otherwise unchanged DINOv2 configuration.
  2. [Section 3.2; Table 5; Figure 4] The robustness claim is supported only by asymmetric experiments. Table 5 shows that DINO diverges when teacher momentum, last-layer normalization, or teacher temperature are perturbed, but there is no corresponding sweep showing how SimDINO behaves under the same perturbed DINO settings, and Figure 4 changes the dataset while keeping the full SimDINO/DINO hyperparameter sets different. Thus the paper does not demonstrate that SimDINO is stable under DINO's original hyperparameters, nor that the DINO divergences would persist after matching the architectural and optimization choices. The authors should include stability sweeps over shared hyperparameters (e.g., learning rate, batch size, EMA momentum, drop path) for both pipelines under otherwise identical settings.
  3. [Tables 1-3] All reported accuracy numbers appear to be single runs without standard deviations, multiple seeds, or significance testing. Given that the paper's own narrative emphasizes the instability of DINO training, the small reported improvements (for example DAVIS J&F 62.0 vs 62.6 and ADE20K mIoU 41.0 vs 41.8) are not interpretable as robust gains. At minimum, the authors should report the seed variance for the smaller models, or clearly state the limitations of single-run comparisons in the context of the stability claim.
minor comments (5)
  1. [Algorithm 1] The teacher update line reads 'ft.params = lam * ft.params + (1 - 1am) * fs.params'; '1am' should be 'lam'.
  2. [Section 3.2] There is a typo in 'Specificlly' in the paragraph on training on COCO train2017; it should be 'Specifically'.
  3. [Throughout] The benchmark 'DA VIS-2017' should be written consistently as 'DAVIS-2017' both in the text and in Table 3.
  4. [Table 4] The row 'Weight normalize last layer removed True removed True' is difficult to parse; the table would be clearer if each method had its own column with explicit values rather than interleaved 'removed' indicators.
  5. [Table 8] The DINO row for 'without self-distillation' is shown as '–', but the text says it collapses; please state explicitly in the caption or text whether the run diverged or produced NaNs, since the claim of collapse is central to the ablation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the coding-rate simplification is tested against external benchmarks, and the only fitted quantity (the gamma scale) is a standard hyperparameter, not a prediction.

full rationale

The paper's claim is empirical: it proposes simplified losses (Eq. 9 and Eq. 17) that replace DINO/DINOv2's heads, centering, softmaxes, Sinkhorn-Knopp, and KoLeo entropy with an explicit total coding rate regularizer, and then evaluates the resulting representations on external benchmarks (ImageNet k-NN/linear accuracy, COCO MaskCut, ADE20K linear segmentation, DAVIS video segmentation). No target result is derived from a fitted quantity. The only fitted constant is the scaling of gamma, chosen to balance gradient norms via Theorem C.1; the paper explicitly calls this 'ultimately a heuristic' and states that 'the constant factor needs to be tuned,' which is ordinary hyperparameter selection, not a prediction forced by construction. The coding rate R_epsilon is defined in the paper itself (Eq. 8), so citations to Ma et al. (2007), Yu et al. (2020), and Li et al. (2022) are sources and related work rather than load-bearing self-citations; no uniqueness theorem or unverified prior result is invoked to forbid alternative design choices. The self-distillation-free experiment (Table 8) independently supports the collapse-avoidance mechanism. The skeptic's observation that Table 4 changes multiple hyperparameters between SimDINOv2 and DINOv2 (batch size, learning rate, register tokens, gradient clip) is a validity/confound concern about causal attribution, not a circularity in the derivation chain. The paper's explicit hypotheses and heuristic choices were weighed and do not create any step where an output is equivalent to an input by definition or by fitting.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim depends on no new physical or mathematical entities. It does depend on two hand-set hyperparameters (gamma, epsilon) and on three domain assumptions about the sufficiency of coding rate for non-collapse, the variance behavior of the covariance estimator, and the adequacy of Euclidean alignment.

free parameters (2)
  • gamma (coding rate regularization strength) = not specified numerically; constant factor tuned in practice
    Section 2.4 sets gamma by equating gradient norm orders and states that the constant factor needs to be tuned. This is a hand-set hyperparameter the method depends on.
  • epsilon (coding rate quantization level) = not reported in Table 4 or elsewhere
    Appears in the logdet term R_epsilon; its value is absent from the hyperparameter table, so reproduction would require guessing or contacting the authors.
assumptions (3)
  • standard math The total coding rate R_epsilon approximates the rate-distortion of a Gaussian and its logdet gradient is bounded by Theorem C.1.
    Invoked in Section 2.4 and Appendix C to justify the scaling of gamma. The theorem proof is correct, but the bound is an upper bound and the balancing argument is a heuristic.
  • domain assumption A plug-in estimate of Cov[zcls_c] over a sub-sample of global views has low enough variance to prevent collapse at smaller batch sizes.
    Stated in Section 2.2 as a conjecture ('we hypothesize might be a factor'). This underpins the batch-size robustness experiments and is load-bearing for the stability claim.
  • domain assumption Squared Euclidean distance between normalized features is a sufficient alignment objective for self-distillation.
    Section 2.2 argues d_l2 works at least as well as cross-entropy on softmaxed features, but no theory is given; it is an empirical choice supported by the experiments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Simplifying DINO via Coding Rate Regularization." pith.science (2026). https://pith.science/paper/Z2INREGN

@misc{pith2026250210385,
  author       = {Pith},
  title        = {Pith review of: Simplifying DINO via Coding Rate Regularization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z2INREGN}},
  note         = {Machine review of arXiv:2502.10385}
}
read the original abstract

DINO and DINOv2 are two model families being widely used to learn representations from unlabeled imagery data at large scales. Their learned representations often enable state-of-the-art performance for downstream tasks, such as image classification and segmentation. However, they employ many empirically motivated design choices and their training pipelines are highly complex and unstable -- many hyperparameters need to be carefully tuned to ensure that the representations do not collapse -- which poses considerable difficulty to improving them or adapting them to new domains. In this work, we posit that we can remove most such-motivated idiosyncrasies in the pre-training pipelines, and only need to add an explicit coding rate term in the loss function to avoid collapse of the representations. As a result, we obtain highly simplified variants of the DINO and DINOv2 which we call SimDINO and SimDINOv2, respectively. Remarkably, these simplified models are more robust to different design choices, such as network architecture and hyperparameters, and they learn even higher-quality representations, measured by performance on downstream tasks, offering a Pareto improvement over the corresponding DINO and DINOv2 models. This work highlights the potential of using simplifying design principles to improve the empirical practice of deep learning.

Figures

Figures reproduced from arXiv: 2502.10385 by the authors.

Figure 1
Figure 1. The DINO and DINOv2 pipelines are substantially simplified to the respective SimDINO and SimDINOv2 pipelines. (a) In the DINO pipeline, an input image is turned into patches. Then a global view vg and a local view vc are randomly sampled. The global view is pushed through the teacher encoder, while the other view is through the student encoder. (b) The SimDINO pipeline removes the need for expensive post-processing … view at source ↗
Figure 2
Figure 2. Evolution of k-NN accuracy of ViT-B trained for 100 epochs using DINO and SimDINO paradigms on ImageNet-1K. We omit earlier epochs of similar metrics for better visual clarity [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Visualization of MaskCut segmentation results from DINO ViT-B/16 (row 1), SimDINO ViT-B/16 (row 2) and SimDINO ViT-L/16 (row 3) on selected images. 10 20 30 40 50 60 70 80 90 100 Epoch 0 5 10 15 20 25 30 k-N N A c c ura c y (%) DINO SimDINO [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: k-NN accuracy on ImageNet-1K of ViT-B trained on COCO train2017 using DINO and SimDINO paradigms. samples but instead manipulates the gradient structure (akin to SimSiam) in order to prevent collapse, and it uses an extra (“prediction”) module appended to the student n…
Figure 5
Figure 5. Figure 5: Visualization of average self-attention maps obtained from both DINO(v2) and SimDINO(v2) algorithms. F.5. Visualization of Attention Maps Following (Oquab et al., 2023; Caron et al., 2021), we provide visualizations of self-attention maps of different models for qualit…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. BenchECG and xECG: a benchmark and baseline for ECG foundation models

    cs.LG 2025-09 conditional novelty 6.0 of 10

    xECG, an xLSTM-based ECG foundation model pretrained with SimDINOv2, achieves the best average performance on BenchECG, a new benchmark spanning eight datasets and ten tasks.

  2. M3Ret: Unleashing Zero-shot Multimodal Medical Image Retrieval via Self-Supervision

    cs.CV 2025-09 conditional novelty 6.0 of 10

    One self-supervised encoder trained on unpaired X-ray, ultrasound, endoscopy, and CT data gives competitive zero-shot retrieval and seems to generalize to unseen MRI tasks.

Reference graph

Works this paper leans on

47 extracted references · 16 canonical work pages · cited by 2 Pith papers

  1. [1]

    Self-supervised learning from images with a joint-embedding predictive architecture

    Assran, M., Duval, Q., Misra, I., Bojanowski, P., Vincent, P., Rabbat, M., LeCun, Y., and Ballas, N. Self-supervised learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 15619--15629, 2023

    2023

  2. [2]

    Towards general purpose vision foundation models for medical image analysis: An experimental study of dinov2 on radiology benchmarks

    Baharoon, M., Qureshi, W., Ouyang, J., Xu, Y., Phol, K., Aljouie, A., and Peng, W. Towards general purpose vision foundation models for medical image analysis: An experimental study of dinov2 on radiology benchmarks. arXiv preprint arXiv:2312.02366, 2023

    arXiv 2023

  3. [3]

    Vicreg: Variance-invariance-covariance regularization for self-supervised learning

    Bardes, A., Ponce, J., and LeCun, Y. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105.04906, 2021

    arXiv 2021

  4. [4]

    J., Gy \"o rfi, L., Van der Meulen, E

    Beirlant, J., Dudewicz, E. J., Gy \"o rfi, L., Van der Meulen, E. C., et al. Nonparametric entropy estimation: An overview. International Journal of Mathematical and Statistical Sciences, 6 0 (1): 0 17--39, 1997

    work page 1997

  5. [5]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  6. [6]

    Unsupervised learning of visual features by contrasting cluster assignments

    Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33: 0 9912--9924, 2020

    2020

  7. [7]

    Emerging properties in self-supervised vision transformers

    Caron, M., Touvron, H., Misra, I., J \'e gou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 9650--9660, 2021

    2021

  8. [8]

    A simple framework for contrastive learning of visual representations

    Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp.\ 1597--1607. PMLR, 2020

    2020

Show all 47 references
  1. [9]

    and He, K

    Chen, X. and He, K. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 15750--15758, 2021

    2021

  2. [10]

    Sinkhorn distances: Lightspeed computation of optimal transport

    Cuturi, M. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013

    2013

  3. [11]

    Dai, X., Tong, S., Li, M., Wu, Z., Psenka, M., Chan, K. H. R., Zhai, P., Yu, Y., Yuan, X., Shum, H.-Y., et al. Ctrl: Closed-loop transcription to an ldr via minimaxing rate reduction. Entropy, 24 0 (4): 0 456, 2022

    2022

  4. [12]

    and Fournier, N

    Delattre, S. and Fournier, N. On the kozachenko--leonenko entropy estimator. Journal of Statistical Planning and Inference, 185: 0 69--93, 2017

    2017

  5. [13]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Devlin, J. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  6. [14]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy, A. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020

    2010 arXiv

  7. [15]

    Masked autoencoders as spatiotemporal learners

    Feichtenhofer, C., Li, Y., He, K., et al. Masked autoencoders as spatiotemporal learners. Advances in neural information processing systems, 35: 0 35946--35958, 2022

    2022

  8. [16]

    Bootstrap your own latent-a new approach to self-supervised learning

    Grill, J.-B., Strub, F., Altch \'e , F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33: ...

    2020

  9. [17]

    Dimensionality reduction by learning an invariant mapping

    Hadsell, R., Chopra, S., and LeCun, Y. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR'06), volume 2, pp.\ 1735--1742. IEEE, 2006

    2006

  10. [18]

    Momentum contrast for unsupervised visual representation learning

    He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 9729--9738, 2020

    2020

  11. [19]

    Masked autoencoders are scalable vision learners

    He, K., Chen, X., Xie, S., Li, Y., Doll \'a r, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 16000--16009, 2022

    2022

  12. [20]

    and Marshall, A

    Kahn, H. and Marshall, A. W. Methods of reducing sample size in monte carlo computations. Journal of the Operations Research Society of America, 1 0 (5): 0 263--278, 1953

    1953

  13. [21]

    Li, Z., Chen, Y., LeCun, Y., and Sommer, F. T. Neural manifold clustering and embedding. arXiv preprint arXiv:2201.10000, 2022

    2022 arXiv

  14. [22]

    Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll \'a r, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp....

    2014

  15. [23]

    Decoupled weight decay regularization

    Loshchilov, I. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  16. [24]

    Segmentation of multivariate mixed data via lossy data coding and compression

    Ma, Y., Derksen, H., Hong, W., and Wright, J. Segmentation of multivariate mixed data via lossy data coding and compression. IEEE transactions on pattern analysis and machine intelligence, 29 0 (9): 0 1546--1562, 2007

    2007

  17. [25]

    and Tong, S

    Mo, S. and Tong, S. Connecting joint-embedding predictive architecture with contrastive self-supervised learning. arXiv preprint arXiv:2410.19560, 2024

    2024 arXiv

  18. [26]

    Oord, A. v. d., Li, Y., and Vinyals, O. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018

    2018 arXiv

  19. [27]

    Dinov2: Learning robust visual features without supervision

    Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023

    2023 arXiv

  20. [28]

    W., Buchanan, S., Yu, Y., and Ma, Y

    Pai, D., Wu, Z. W., Buchanan, S., Yu, Y., and Ma, Y. Masked completion via structured diffusion with white-box transformers. International Conference on Learning Representations, 2023

    2023

  21. [29]

    The 2017 davis challenge on video object segmentation

    Pont-Tuset, J., Perazzi, F., Caelles, S., Arbel\'aez, P., Sorkine-Hornung, A., and Van Gool , L. The 2017 davis challenge on video object segmentation. arXiv:1704.00675, 2017

    2017 arXiv

  22. [30]

    Improving language understanding by generative pre-training

    Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. Improving language understanding by generative pre-training. 2018

    2018

  23. [31]

    Language models are unsupervised multitask learners

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  24. [32]

    W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.\ 8748--8763. PMLR, 2021

    2021

  25. [33]

    and Kingma, D

    Salimans, T. and Kingma, D. P. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. Advances in neural information processing systems, 29, 2016

    2016

  26. [34]

    B., Carranza, A., Isik, B., Unell, A., Khona, M., Yerxa, T., LeCun, Y., Chung, S., et al

    Schaeffer, R., Lecomte, V., Pai, D. B., Carranza, A., Isik, B., Unell, A., Khona, M., Yerxa, T., LeCun, Y., Chung, S., et al. Towards an improved understanding and utilization of maximum manifold capacity representations. arXiv preprint arXiv:2406.09366, 2024

    2024 arXiv

  27. [35]

    Unsupervised learning of structured representations via closed-loop transcription

    Tong, S., Dai, X., Chen, Y., Li, M., Li, Z., Yi, B., LeCun, Y., and Ma, Y. Unsupervised learning of structured representations via closed-loop transcription. arXiv preprint arXiv:2210.16782, 2022

    2022 arXiv

  28. [36]

    Training data-efficient image transformers & distillation through attention

    Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and J \'e gou, H. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp.\ 10347--10357. PMLR, 2021

    2021

  29. [37]

    X., and Misra, I

    Wang, X., Girdhar, R., Yu, S. X., and Misra, I. Cut and learn for unsupervised object detection and instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 3124--3134, 2023

    2023

  30. [38]

    Stronger fewer & superior: Harnessing vision foundation models for domain generalized semantic segmentation

    Wei, Z., Chen, L., Jin, Y., Ma, X., Liu, T., Ling, P., Wang, B., Chen, H., and Zheng, J. Stronger fewer & superior: Harnessing vision foundation models for domain generalized semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogn...

    2024

  31. [39]

    X., and Lin, D

    Wu, Z., Xiong, Y., Yu, S. X., and Lin, D. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 3733--3742, 2018

    2018

  32. [40]

    Wu, Z., Ding, T., Lu, Y., Pai, D., Zhang, J., Wang, W., Yu, Y., Ma, Y., and Haeffele, B. D. Token statistics transformer: Linear-time attention via variational rate reduction. arXiv preprint arXiv:2412.17810, 2024

    2024 arXiv

  33. [41]

    Scaling white-box transformers for vision

    Yang, J., Li, X., Pai, D., Zhou, Y., Ma, Y., Yu, Y., and Xie, C. Scaling white-box transformers for vision. arXiv preprint arXiv:2405.20299, 2024

    2024 arXiv

  34. [42]

    Learning efficient coding of natural images with maximum manifold capacity representations

    Yerxa, T., Kuang, Y., Simoncelli, E., and Chung, S. Learning efficient coding of natural images with maximum manifold capacity representations. Advances in Neural Information Processing Systems, 36: 0 24103--24128, 2023

    2023

  35. [43]

    Yu, Y., Chan, K. H. R., You, C., Song, C., and Ma, Y. Learning diverse and discriminative representations via the principle of maximal coding rate reduction. Advances in neural information processing systems, 33: 0 9422--9434, 2020

    2020

  36. [44]

    White-box transformers via sparse rate reduction

    Yu, Y., Buchanan, S., Pai, D., Chu, T., Wu, Z., Tong, S., Haeffele, B., and Ma, Y. White-box transformers via sparse rate reduction. Advances in Neural Information Processing Systems, 36: 0 9422--9457, 2023

    2023

  37. [45]

    Scene parsing through ade20k dataset

    Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., and Torralba, A. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 633--641, 2017

    2017

  38. [46]

    ibot: Image bert pre-training with online tokenizer

    Zhou, J., Wei, C., Wang, H., Shen, W., Xie, C., Yuille, A., and Kong, T. ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:2111.07832, 2021

    2021 arXiv

  39. [47]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.