Pith. sign in

REVIEW 3 major objections 5 minor 24 references

Leveraging Registers in Vision Transformers for Robust Adaptation

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Concatenating CLS with register token means improves OOD accuracy and anomaly rejection across frozen DINOv2 ViTs.

desk verdict A simple and plausibly useful feature-concatenation recipe for frozen DINOv2, but the missing random-patch control and lack of error bars keep it from being fully convincing. read the letter →

arxiv 2501.04784 v1 pith:4IWSUFYH submitted 2025-01-08 cs.CV cs.LG

Srikar Yellapragada , Kowshik Thopalli , Vivek Narayanaswamy , Wesam Sakla , Yang Liu , Yamen Mubarka , Dimitris Samaras , Jayaraman J. Thiagarajan This is my paper · ORCID
classification cs.CVcs.LG
keywords visiontransformersregistertokensout-of-distributiongeneralizationanomalydetectionlinearprobingDINOv2robustadaptationfeatureconcatenation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the register tokens introduced to clean up high-norm artifacts in vision transformers are not just a training crutch: their embeddings carry global image information that survives averaging and can be reused at test time. Concretely, it claims that training a linear classifier on the concatenation of the [CLS] token and the mean register embedding yields 2–4% higher top-1 accuracy on ImageNet-A, ImageNet-R and ImageNet-Sketch, and a 2–3% lower false-positive rate in anomaly rejection, across DINOv2 ViT-B/L/G, while in-distribution accuracy stays essentially flat. The attraction is that this is a post-hoc feature choice: the backbone stays frozen, and only the input to the final linear layer changes, so the gains come with no added inference cost.

What carries the argument

The central object is the average-pooled register embedding $\mu_R = \frac{1}{M}\sum_{k=1}^{M} r_k$, concatenated with the [CLS] token $c$ to form the feature $[c; \mu_R]$ that feeds a linear classifier. Registers are extra tokens appended to the input sequence during pretraining; prior work showed they absorb high-norm patch tokens and also capture global image-level information. The argument works by exploiting that auxiliary information: replacing the mean patch tokens in the standard $[c; \mu_P]$ protocol with the mean register tokens gives the linear probe access to a complementary global signal, improving robustness at no extra compute because the backbone is frozen.

What would settle it

Train the same linear probes on $[c; \mu_R]$ and $[c; \mu_P]$ after randomly shuffling register embeddings across images; if the OOD accuracy gain persists, the effect is not due to register-specific global information. Alternatively, measure whether $\mu_R$ is linearly separable for ImageNet classes with a rank or mutual-information analysis; if it carries no more class-relevant variance than $\mu_P$, the claimed mechanism is unsupported.

Watch

Extended reading notes

Core claim

The central discovery is that register token embeddings, which earlier work discarded after pretraining, contain linearly accessible auxiliary information that complements the [CLS] token. On DINOv2 ViT-Base, ViT-Large and ViT-Giant, the paper shows that a linear probe trained on $f_i = [c_i; \mu_R^i]$ — the CLS token concatenated with the mean of the register tokens — outperforms the standard concatenation $[c_i; \mu_P^i]$ with the mean patch token on out-of-distribution generalization and anomaly rejection. The comparison holds against both the no-register backbone using $[c; \mu_P]$ and the register backbone using $[c; \mu_P]$, while in-distribution accuracy remains comparable. Register tokens alone are not sufficient: a probe trained on $\mu_R$ alone degrades OOD accuracy, so the benefit is specifically the combination.

Load-bearing premise

The load-bearing premise is that the average of all register embeddings from a frozen, register-trained checkpoint still contains global image information that is distinct from and complementary to the [CLS] token; that property is imported from earlier work on registers rather than established here.

Editorial extensions

If this is right

  • Linear-probe robustness of frozen vision transformer backbones can be improved by a simple feature change, with no fine-tuning or extra inference cost.
  • The improvement is consistent across ViT-Base, ViT-Large and ViT-Giant, suggesting it is not an artifact of one architecture.
  • Anomaly rejection improves under both MSP and energy scoring, so open-set and safety-critical applications inherit the benefit.
  • Since the linear layer input dimension is the same as for the $[CLS; \mu_P]$ baseline, deployment memory and compute are unchanged.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: if the register mean carries genuinely complementary global information, then learning per-register weights or a small pooling network could beat plain averaging; the paper does not test this.
  • Inference: the same concatenation trick may transfer to other architectures that append register-like tokens, such as Mamba-R, or to dense tasks where global context matters.
  • Inference: a causal test of the mechanism would be to remove register tokens from the pretrained backbone and see if the OOD gain vanishes; the paper only compares backbones trained with and without registers, and the without-register backbone cannot produce $\mu_R$.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies whether register tokens in DINOv2 Vision Transformers are useful beyond their original role in suppressing high-norm patch tokens. It proposes to concatenate the [CLS] token with the mean of the register embeddings, µR, and train a linear classifier on the resulting representation. This is compared with the standard [CLS; µP] representation, where µP is the mean of patch embeddings. Experiments with ViT-Base, ViT-Large, and ViT-Giant, evaluated on ImageNet-A/R/S for OOD generalization and on six datasets for anomaly rejection, report 2–4% top-1 OOD accuracy gains and 2–3% false-positive-rate reductions while claiming that in-distribution accuracy is maintained, all with no additional inference cost. The main evidence is in Table I.

Significance. If the central empirical claim is correct, the paper provides a zero-cost, post-hoc feature choice that improves OOD generalization and anomaly rejection for frozen DINOv2 backbones, which would be a practically useful finding. The strengths of the paper are that it uses standard linear-probing protocols, open-source checkpoints, three backbone scales, and it does not tune the method on the OOD test sets. However, the result is purely empirical, and its load-bearing comparison is confounded by the different numbers of tokens being averaged; the paper also reports single-run numbers without uncertainty estimates. The stated quantitative gains in the abstract are not fully consistent with Table I. The direction of the result is plausible, but the central attribution to 'registers' is not yet established.

major comments (3)
  1. [§IV, Table I] The central comparison [CLS; µR] versus [CLS; µP] varies two factors simultaneously: the token type (register vs patch) and the number of tokens averaged (M=4 registers vs roughly 196–256 patch tokens). A mean over four vectors has very different statistical properties from a mean over hundreds, so the OOD gains could be caused by the aggregation statistics rather than by any 'global image-level information' specific to registers. The paper does not run the control [CLS; mean of M randomly chosen patch tokens] (or an equivalent control that matches the number of averaged tokens), so the attribution of the gain to registers is not supported. Please add this control, or reframe the claim to a recipe-level statement with the confound acknowledged.
  2. [§IV, Table I] All numbers appear to be single-run estimates with no standard deviations or seeds. Several of the reported margins are small: for example, the ViT-G energy-score mean FPR is 21.25 for [CLS; µP] versus 20.86 for [CLS; µR], and the ID accuracy differences are below one point for all three backbones. Without variance or multiple runs, the claim that the improvements are consistent across backbones is not fully supported. In addition, the claim that ID accuracy is 'maintained' is contradicted by the printed numbers: ViT-G drops from 87.1 to 86.57, ViT-Large from 86.7 to 85.82, and ViT-Base from 84.21 to 83.84. Please report error bars or multiple runs, and explicitly acknowledge this small ID tradeoff.
  3. [Abstract and §IV-B] The abstract states a '2-3% reduction in false positive rates,' but Table I does not consistently show this. Mean FPR reductions relative to the stronger [CLS; µP] baseline are 3.91/4.65/3.58 points for MSP and 0.39/3.22/5.63 points for Energy across the three backbones; the ViT-G Energy improvement is only 0.39 points. The quantitative claim in the abstract needs to be restated precisely, specifying which baseline and scoring function it refers to.
minor comments (5)
  1. [§IV-B] The sentence 'for ViT-Large, our method reduces the mean FPR by 22.18 and 3.22 percentage points compared to the baselines respectively' is unclear and appears to hold only for the Energy score compared with the no-register baseline on one architecture; please rephrase and specify which baselines and which scoring functions are being compared.
  2. [Abstract and Conclusion] The abstract and conclusion say the method works on backbones 'trained with and without registers,' but the proposed [CLS; µR] representation requires a model that has register tokens. Please clarify that the no-register models are used only as baselines.
  3. [§IV-A] The last sentence of Section IV-A is garbled: 'the combination of by viewing register tokens as auxiliary features and combining it with CLS embeddings yield substantially superior capture more features in terms of generalization and robustness.' This needs to be rewritten.
  4. [Fig. 1] The caption says 'Each color indicates the token embeddings chosen,' but the figure legend is not described in the caption and the axes are not labeled. Please make the figure self-contained.
  5. [Throughout] There are minor language and formatting issues, including 'auxilliary' in the conclusion and inconsistent capitalization of 'DINOv2' (Dino-v2, Dinov2, DINO-v2).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's OOD and anomaly-rejection claims are empirical benchmark results from frozen external backbones, with no fitted-then-predicted quantity or self-citation chain.

full rationale

The central claim is that linear classifiers trained on [CLS; mu_R] improve OOD generalization and anomaly rejection relative to [CLS; mu_P] (Abstract, Table I). The classifier is fit on ImageNet-1K via the objective in Eq. 1 and then evaluated on external OOD and anomaly benchmarks; no parameter is tuned on the OOD test sets, and no quantity appearing in the loss or feature construction is defined in terms of the reported OOD or FPR metrics. The register embeddings come from frozen DINOv2 checkpoints produced externally (Darcet et al. [6] and Oquab et al. [8]), not from any fitting procedure in this paper, and those citations are external evidence rather than self-citation. The method is an empirical feature-concatenation recipe: whether registers contain global information is a premise imported from prior work, but the paper does not claim to mathematically derive that premise, so its empirical comparison remains independent of that premise. The main weakness is a potential confound, namely that mu_R averages M=4 register tokens while mu_P averages hundreds of patch tokens, and no control with mean-of-M-random-patch-tokens is reported; however, a confound is a validity threat, not a circularity, because the reported gains are not forced by the construction of the features or by an equation equating output with input. No self-citation is load-bearing, no uniqueness theorem is invoked, and no known result is renamed. Therefore the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no fitted constants and no invented entities. Its central claim rests on imported assumptions: that DINOv2 registers encode useful global information (from Darcet et al.), that redundant ID features can help OOD (from Zhang and Bottou), and that linear probing with fixed hyperparameters is a fair evaluation. These are domain assumptions, not circular definitions.

assumptions (4)
  • domain assumption Register tokens capture global image-level information similar to the CLS token.
    Imported from Darcet et al. [6], used in Section III to justify using the mean register embedding as an auxiliary feature.
  • domain assumption Representations that are redundant on in-distribution data can carry complementary information under distribution shift.
    Imported from Zhang and Bottou [7], used in Section III to motivate combining CLS with register tokens.
  • domain assumption Linear probing on frozen DINOv2 features is a valid and sufficient measure of representation quality for adaptation.
    All conclusions in Section IV are drawn from linear classifier accuracy and OOD detection scores, with no non-linear probe or fine-tuning check.
  • domain assumption MSP and energy scoring provide valid anomaly rejection signals in this evaluation protocol.
    Adopted from Hendrycks and Gimpel [24] and Liu et al. [23], used for all FPR and AUROC numbers in Table I.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Leveraging Registers in Vision Transformers for Robust Adaptation." pith.science (2026). https://pith.science/paper/4IWSUFYH

@misc{pith2026250104784,
  author       = {Pith},
  title        = {Pith review of: Leveraging Registers in Vision Transformers for Robust Adaptation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4IWSUFYH}},
  note         = {Machine review of arXiv:2501.04784}
}
read the original abstract

Vision Transformers (ViTs) have shown success across a variety of tasks due to their ability to capture global image representations. Recent studies have identified the existence of high-norm tokens in ViTs, which can interfere with unsupervised object discovery. To address this, the use of "registers" which are additional tokens that isolate high norm patch tokens while capturing global image-level information has been proposed. While registers have been studied extensively for object discovery, their generalization properties particularly in out-of-distribution (OOD) scenarios, remains underexplored. In this paper, we examine the utility of register token embeddings in providing additional features for improving generalization and anomaly rejection. To that end, we propose a simple method that combines the special CLS token embedding commonly employed in ViTs with the average-pooled register embeddings to create feature representations which are subsequently used for training a downstream classifier. We find that this enhances OOD generalization and anomaly rejection, while maintaining in-distribution (ID) performance. Extensive experiments across multiple ViT backbones trained with and without registers reveal consistent improvements of 2-4\% in top-1 OOD accuracy and a 2-3\% reduction in false positive rates for anomaly detection. Importantly, these gains are achieved without additional computational overhead.

Figures

Figures reproduced from arXiv: 2501.04784 by the authors.

Figure 1
Figure 1. Impact of token embedding choices on linear probing on frozen Dino-V2 ViT-G backbones. Each color indicates the token embeddings chosen for optimizing a lin￾ear classifier on ImageNet (IN)-1K along with the protocol adopted for pre-training the backbone (w/o registers or w regis￾ters). Here, [CLS] represents the classification token, µP and µR represents the mean patch and register token embeddings respectively. Whi… view at source ↗
Figure 2
Figure 2. Overview of our proposed method: For large-scale vision transformer backbones (e.g., DINO-v2) pre-trained with “registers”, we find that concatenating registers (Mean (R1, R2, R3, R4) along with [CLS] is critical for obtaining rich features that enable robust adaptation. In particular, we train a linear classifier on these concatenated features and observe improved generalization and anomaly rejection capabilities. … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 11 canonical work pages

  1. [1]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weis- senborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021

    2021

  2. [2]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  3. [3]

    Deit iii: Revenge of the vit

    Hugo Touvron, Matthieu Cord, and Herv ´e J ´egou. Deit iii: Revenge of the vit. In European conference on computer vision , pages 516–533. Springer, 2022

    work page 2022

  4. [4]

    GPT3.int8(): 8-bit matrix multiplication for transformers at scale

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. GPT3.int8(): 8-bit matrix multiplication for transformers at scale. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022

    work page 2022

  5. [5]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4195–4205, 2023

    2023

  6. [6]

    Vision transformers need registers

    Timoth ´ee Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In The Twelfth International Con- ference on Learning Representations , 2024

    work page 2024

  7. [7]

    Learning useful representations for shifting tasks and distributions

    Jianyu Zhang and L ´eon Bottou. Learning useful representations for shifting tasks and distributions. In International Conference on Machine Learning, pages 40830–40850. PMLR, 2023

    work page 2023

  8. [8]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 , 2023

    arXiv 2023

Show all 24 references
  1. [9]

    Residual stream norms grow exponentially over the forward pass

    Stefan Heimersheim and Alex Turner. Residual stream norms grow exponentially over the forward pass. In AI Alignment F orum, page 23, 2023

    2023

  2. [10]

    Bert busters: Outlier dimensions that disrupt transformers

    Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, and Anna Rumshisky. Bert busters: Outlier dimensions that disrupt transformers. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3392–3405, 2021

    2021

  3. [11]

    Unveiling a core linguistic region in large language models

    Jun Zhao, Zhihao Zhang, Yide Ma, Qi Zhang, Tao Gui, Luhui Gao, and Xuanjing Huang. Unveiling a core linguistic region in large language models. arXiv preprint arXiv:2310.14928 , 2023

    2023 arXiv

  4. [12]

    All bark and no bite: Rogue dimensions in transformer language models obscure representational quality

    William Timkey and Marten Van Schijndel. All bark and no bite: Rogue dimensions in transformer language models obscure representational quality. arXiv preprint arXiv:2109.04404 , 2021

    2021 arXiv

  5. [13]

    Mamba-r: Vision mamba also needs registers

    Feng Wang, Jiahao Wang, Sucheng Ren, Guoyizhe Wei, Jieru Mei, Wei Shao, Yuyin Zhou, Alan Yuille, and Cihang Xie. Mamba-r: Vision mamba also needs registers. arXiv preprint arXiv:2405.14858 , 2024

    2024 arXiv

  6. [14]

    Understanding and minimising outlier features in neural network training

    Bobby He, Lorenzo Noci, Daniele Paliotta, Imanol Schlag, and Thomas Hofmann. Understanding and minimising outlier features in neural network training. arXiv preprint arXiv:2405.19279 , 2024

    2024 arXiv

  7. [15]

    Massive activations in large language models

    Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. arXiv preprint arXiv:2402.17762 , 2024

    2024 arXiv

  8. [16]

    Scaling vision transformers

    Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR) , pages 12104–12113, June 2022

    2022

  9. [17]

    An inverse scaling law for clip training

    Xianhang Li, Zeyu Wang, and Cihang Xie. An inverse scaling law for clip training. Advances in Neural Information Processing Systems , 36, 2024

    2024

  10. [18]

    H-optimus-0, 2024

    Charlie Saillard, Rodolphe Jenatton, Felipe Llinares-L ´opez, Zelda Ma- riet, David Cahan ´e, Eric Durand, and Jean-Philippe Vert. H-optimus-0, 2024

    2024

  11. [19]

    Imagenet large scale visual recognition challenge

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision , 115(3):211–252, 2015

    2015

  12. [20]

    Natural adversarial examples

    Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 15262– 15271, 2021

    2021

  13. [21]

    The many faces of robustness: A critical analysis of out-of-distribution generalization

    Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. ICCV, 2021

    2021

  14. [22]

    Learning robust global representations by penalizing local predictive power

    Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems , pages 10506– 10518, 2019

    2019

  15. [23]

    Energy- based out-of-distribution detection

    Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy- based out-of-distribution detection. Advances in Neural Information Processing Systems, 33:21464–21475, 2020

    2020

  16. [24]

    A baseline for detecting misclassified and out-of-distribution examples in neural networks

    Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations , 2017

    2017

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.