pith. sign in

arxiv: 2606.03148 · v1 · pith:RYWEI2CXnew · submitted 2026-06-02 · 💻 cs.CV

A²: Smaller Self-Supervised ViTs Localize Better than Larger Ones

Sreehari Rammohan , Huy Ha , Carl Vondrick This is my paper

Pith reviewed 2026-06-28 11:12 UTC · model grok-4.3

classification 💻 cs.CV
keywords self-supervised learningvision transformersattention mapsobject localizationimage classificationdistribution shiftmodel scalingcropping
0
0 comments X

The pith

Smaller self-supervised vision transformers localize foreground objects better with their attention maps than larger ones.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that attention maps from smaller self-supervised ViTs highlight main foreground objects more accurately than those from larger ViTs. Larger models still produce richer per-patch representations, so the authors introduce A² to combine the two strengths by using the small model's attention peaks to crop the image and then embedding the resulting crops with the large model. The approach relies only on pretrained features, needs no labels or additional training, and remains competitive with supervised baselines on five benchmarks while handling distribution shifts. A reader would care because it offers a training-free way to reduce the impact of contextual distractors in visual classification.

Core claim

The central claim is that smaller self-supervised ViTs produce attention maps that localize foreground objects better than larger ViTs. A² exploits this inverse scaling by decoupling localization (via attention peaks from a small pretrained ViT) from representation extraction (via a larger pretrained ViT that embeds the cropped regions), yielding competitive results across benchmarks without any per-dataset training or group labels.

What carries the argument

A², the method that crops images around attention peaks from a small self-supervised ViT and embeds the crops with a larger ViT to separate localization from feature richness.

If this is right

  • A² matches the accuracy of backbone-matched loss-level methods like DFR across five benchmarks.
  • It outperforms end-to-end attention training when distribution shifts become stronger.
  • The method works with any pair of pretrained small and large self-supervised ViTs and requires no extra optimization.
  • Robustness gains come from improved foreground focus without retraining either model.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same small-to-large cropping pattern could be tested on other localization-heavy tasks such as detection or segmentation to check whether the inverse scaling holds beyond classification.
  • If the localization advantage of smaller models persists across different self-supervised objectives, it might indicate that model capacity trades off global structure against local texture detail in attention.
  • Practitioners could explore using an ensemble of small models for more stable attention peaks before feeding crops to the large embedder.

Load-bearing premise

That attention peaks from the small model reliably mark regions containing the main object so that cropping them lets the large model extract useful representations without losing essential context or introducing bad crops.

What would settle it

On images with ground-truth object bounding boxes, if the attention-based crops from the small model consistently exclude the primary object or cut away too much surrounding context, performance with the large model would drop below that of embedding the full uncropped image.

Figures

Figures reproduced from arXiv: 2606.03148 by Carl Vondrick, Huy Ha, Sreehari Rammohan.

Figure 1
Figure 1. Figure 1: Robust Recognition through Exclusion. Images contain spurious correlations that often mislead classifiers, e.g. a man holding a rein is usually associated with horses. A2 removes spatially spurious correlations by automatically selecting crops based on a pre-trained model’s attention map. While the original model is misled (left), our A2 trained classifier explicitly attends to only the animal body and cor… view at source ↗
Figure 2
Figure 2. Figure 2: Do Smaller ViTs Localize Better? We show the proportion of attention mass that is inside ground-truth bounding boxes versus model size across six pretraining families, on Waterbirds (left) and ImageNet Val (right). All five self-supervised families (DINOv1, DINOv2, DINOv3, MAE, iBOT) slope downward as model size grows; the contrastive image-text family (OpenCLIP, dashed) does not. progress in curating benc… view at source ↗
Figure 3
Figure 3. Figure 3: Overview of A2 . Given an input image, x, we use an attention model to select crops, Cropi (x; Attn(x)), which are resized and embedded using a (possibly different) embedding model, T . The concatenated embeddings [ T (c1), . . . , T (ck) ] are passed to a classifier f to predict yˆ. and hit rate as the ViT gets larger. For ImageNet Val, ViT-S puts 74.6% of its mass inside the ground truth object box vs. 6… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative example of A2 attention-guided crops. One to four 64×64 crops selected by ViT-S attention on a MetaShift Animals bear. Crop regions are at full brightness; the rest of the image is faded. A2 focuses on the bear and excludes the surrounding water and aquarium context. Full crop-count ablation in Appendix D [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: MetaShift Animals (ours): disjoint train/test contexts. One of our eight classes (giraffe) shown across several Visual Genome context tags. The top row of each pair contains training images and the bottom row contains test images; the per-image label is the context tag (spurious feature). Train tags (e.g., tree, building, man) and test tags (e.g., water, umbrella, dirt) are disjoint by construction. Full p… view at source ↗
Figure 6
Figure 6. Figure 6: Where and What Scaling Laws. Larger embedding models but smaller attention models improve performance. The best configuration pairs a small attention model with a large embedder. Each panel plots each dataset’s standard metric (test accuracy for Spawrious; worst-group accuracy for Waterbirds and Cat vs. Dog; worst-class accuracy for Animals) relative to the smallest model (thin colored) and the mean of tho… view at source ↗
Figure 7
Figure 7. Figure 7: Downstream Task Performance. A2 ZS tracks localization quality, not model size. We fix the embedder to OpenCLIP ViT-L/14 and vary the attention model. Each panel plots per-dataset performance relative to the smallest model in that family (thin colored) and the mean of those relative changes across 5 datasets (bold black). Left: With a DINOv2 crop selector, performance decreases as the attention model grows… view at source ↗
Figure 8
Figure 8. Figure 8: Visualizing the MLP Adapter. The MLP Adapter changes the attention map for the blond male class in the CelebA dataset. Default attention (lime box) focuses on the face; adapted attention (cyan box) focuses on the hair, which is the label-defining feature. whose input is the latent activations of the ViT, and whose output is added to the attention map from the ViT. The MLP is trained end-to-end, by backprop… view at source ↗
Figure 9
Figure 9. Figure 9: Per-layer CLS-to-patches attention mass for DINOv2 with register tokens [8]. Stars mark each model’s last block (the layer used in our other figures). ViT-S and ViT-B peak at the last block; ViT-L peaks at block 21 of 24 and ViT-G at block 32 of 40, dropping 12–13 points on Waterbirds (left) and 4–6 points on ImageNet val (right) by the final block. Larger ViTs’ final blocks drift away from foreground loca… view at source ↗
Figure 10
Figure 10. Figure 10: The dip in patch-side localization coincides with attention shifting to register tokens. For DINOv2 with registers on Waterbirds (top) and ImageNet val (bottom), we plot the per-layer fraction of CLS softmax mass on patch tokens (blue) vs. register tokens (red), alongside the patch-side mass-in-GT-bbox metric from [PITH_FULL_IMAGE:figures/full_fig_p009_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Attention hit rate vs. model size across six pretraining families, on Waterbirds (left) and [PITH_FULL_IMAGE:figures/full_fig_p016_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Per-layer attention hit rate for DINOv2 with register tokens [8]. Hit-rate counterpart to [PITH_FULL_IMAGE:figures/full_fig_p016_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Per-layer CLS-to-patches attention mass for DINOv2 without register tokens. Without registers to absorb the late-layer high-norm “artifact” tokens identified by Darcet et al. [8], the inverse scaling trend in [PITH_FULL_IMAGE:figures/full_fig_p017_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Per-layer attention hit rate for DINOv2 without register tokens. Hit-rate counterpart to [PITH_FULL_IMAGE:figures/full_fig_p017_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: MetaShift Animals: full context visualization. Companion to [PITH_FULL_IMAGE:figures/full_fig_p020_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Crop count ablation visualization. Each column shows 1–4 attention-guided [PITH_FULL_IMAGE:figures/full_fig_p021_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Learning the MLP attention adapter by differentiating through classifier fit using differen [PITH_FULL_IMAGE:figures/full_fig_p029_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Examples from the bottom 1% (left three) subset and top 1% (right three) subsets. Red [PITH_FULL_IMAGE:figures/full_fig_p030_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: MLP Adapter Learning curves on CelebA. Results averaged across 5 seeds. [PITH_FULL_IMAGE:figures/full_fig_p033_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: DINOv3 where-to-look scaling (mean across datasets) with fixed DINOv2 ViT-G embed [PITH_FULL_IMAGE:figures/full_fig_p034_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: DINOv3 where-to-look scaling per dataset (colored lines) with the mean trend (black), [PITH_FULL_IMAGE:figures/full_fig_p035_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: Worst-group accuracy version of Figure 7, also plotted relative to the smallest model in [PITH_FULL_IMAGE:figures/full_fig_p037_22.png] view at source ↗
read the original abstract

Robust visual classification often depends on localizing the main foreground objects in an image while ignoring contextual distractors. Surprisingly, we find that the attention maps of smaller self-supervised ViTs localize foreground objects better than those of larger ViTs. However, we still need large ViTs, because they extract richer representations from each patch. To get the best of both worlds, good localization and rich representations, we propose $A^2$, a simple method that leverages this inverse scaling finding by decoupling where to look (a small attention model) from what to extract (a large embedding model): we crop around the attention peaks of a small model and embed the crops with a larger model. $A^2$ uses entirely pretrained features, requires no group labels, and does not require per-dataset attention or backbone training. Across 5 benchmarks, $A^2$ is competitive with backbone-matched loss-level methods like DFR, and outperforms end-to-end attention training under stronger distribution shifts.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper claims that smaller self-supervised Vision Transformers produce attention maps that localize foreground objects better than those from larger ViTs (an inverse scaling effect), and proposes the A² method to exploit this by using a small model's attention peaks to crop images and a large model's embeddings on those crops. A² requires no additional training, group labels, or per-dataset fitting and is reported competitive with methods like DFR on five benchmarks while outperforming end-to-end attention training under strong distribution shifts.

Significance. If the inverse scaling finding on localization holds, the result would provide a practical, training-free way to combine the localization strengths of small SSL ViTs with the representation richness of large ones, potentially improving robustness to distribution shifts without the overhead of group-robust training or attention fine-tuning. The use of entirely off-the-shelf pretrained models and the decoupling of localization from embedding are notable strengths that could generalize to other vision tasks.

major comments (2)
  1. [Experiments] Experiments section: The central claim that smaller SSL ViTs localize foreground objects better is supported only indirectly by A²'s downstream benchmark performance; no direct localization metrics (e.g., IoU with ground-truth boxes or pointing-game accuracy) are reported on datasets equipped with bounding-box annotations such as ImageNet-1k validation or COCO. This leaves open whether attention peaks align with main objects or simply yield crop sizes that benefit the tested shifts.
  2. [§3] §3 (Method) and abstract: The description of how attention maps from the small model are converted into crops (peak selection, thresholding, crop sizing) lacks sufficient detail to verify that the localization step is the operative factor, which is load-bearing for attributing gains to the claimed inverse scaling rather than other aspects of the pipeline.
minor comments (2)
  1. [Abstract] Abstract: The notation $A^2$ is introduced without an explicit expansion or definition on first use.
  2. The manuscript would benefit from a table or figure explicitly comparing localization quality metrics (even if added in revision) alongside the downstream results to make the core finding directly falsifiable.

Simulated Author's Rebuttal

2 responses · 0 unresolved

Thank you for your review and the recommendation for major revision. We appreciate the feedback highlighting areas where additional evidence and clarity would strengthen the paper. We plan to address both major comments through revisions to the experiments and method sections.

read point-by-point responses

  1. Referee: [Experiments] Experiments section: The central claim that smaller SSL ViTs localize foreground objects better is supported only indirectly by A²'s downstream benchmark performance; no direct localization metrics (e.g., IoU with ground-truth boxes or pointing-game accuracy) are reported on datasets equipped with bounding-box annotations such as ImageNet-1k validation or COCO. This leaves open whether attention peaks align with main objects or simply yield crop sizes that benefit the tested shifts.

    Authors: We acknowledge that direct quantitative evaluation of localization quality would provide stronger evidence for the inverse scaling claim. The current manuscript relies on the downstream task performance as a proxy, which demonstrates the practical benefit but does not directly measure alignment with ground-truth objects. In the revision, we will add direct localization metrics, such as pointing game accuracy or IoU on the ImageNet validation set (which has bounding box annotations), to directly support the claim that smaller models localize better. revision: yes

  2. Referee: [§3] §3 (Method) and abstract: The description of how attention maps from the small model are converted into crops (peak selection, thresholding, crop sizing) lacks sufficient detail to verify that the localization step is the operative factor, which is load-bearing for attributing gains to the claimed inverse scaling rather than other aspects of the pipeline.

    Authors: We agree that more details are needed in the method description. In the revised version, we will expand the description in §3 to specify the exact procedure: how the attention map is processed (e.g., averaging over heads if applicable), peak selection (e.g., selecting the highest attention location), thresholding if used, and how the crop is determined from the peak (e.g., fixed size centered on the peak or adaptive based on attention values). This will allow readers to verify the localization mechanism. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical observation on pretrained models

full rationale

The paper reports an empirical finding that smaller self-supervised ViTs yield superior attention localization, then applies this observation via a simple cropping procedure (A²) that decouples a small attention model from a large embedding model. All components rely on off-the-shelf pretrained weights with no new parameter fitting, no self-referential predictions, and no load-bearing self-citations that justify the core claim. The derivation chain consists of direct experimental comparison and downstream benchmark evaluation rather than any reduction of outputs to inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

No free parameters or invented entities are introduced; the approach depends on standard domain assumptions about pretrained self-supervised models being directly usable for attention and embedding.

axioms (1)
  • domain assumption Pretrained self-supervised ViTs produce usable attention maps and embeddings without additional per-dataset training or labels.
    The method description states it uses entirely pretrained features and requires no group labels or backbone training.

pith-pipeline@v0.9.1-grok · 5705 in / 1143 out tokens · 26191 ms · 2026-06-28T11:12:47.168855+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 17 canonical work pages · 7 internal anchors

  1. [1]

    Zero-Shot Robustification of Zero-Shot Models.arXiv e-prints, art

    Dyah Adila, Changho Shin, Linrong Cai, and Frederic Sala. Zero-Shot Robustification of Zero-Shot Models.arXiv e-prints, art. arXiv:2309.04344, September 2023. doi: 10.48550/ arXiv.2309.04344

    arXiv 2023

  2. [2]

    Dantas, Dino Ienco, and Diego Marcos

    Ananthu Aniraj, Cassio F. Dantas, Dino Ienco, and Diego Marcos. Two-stage vision transformers and hard masking offer robust object representations. InInternational Conference on Pattern Recognition (ICPR), 2026

    2026

  3. [3]

    SAM 3: Segment Anything with Concepts

    Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, Jie Lei, Tengyu Ma, Baishan Guo, Arpit Kalla, Markus Marks, Joseph Greer, Meng Wang, Peize Sun, Roman Rädle, Triantafyllos Afouras, Effrosyni Mavroudi, Katherine Xu, Tsung-Han Wu, Yu Zhou, Liliane ...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2511.16719 2025

  4. [4]

    Emerging Properties in Self-Supervised Vision Transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties in Self-Supervised Vision Transformers.arXiv e-prints, art. arXiv:2104.14294, April 2021. doi: 10.48550/arXiv.2104.14294

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2104.14294 2021

  5. [5]

    Reproducible scaling laws for contrastive language-image learning.arXiv e-prints, art

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning.arXiv e-prints, art. arXiv:2212.07143, December 2022. doi: 10.48550/arXiv.2212.07143

    work page doi:10.48550/arxiv.2212.07143 2022

  6. [6]

    Invariant Causal Mechanisms through Distribution Matching.arXiv e-prints, art

    Mathieu Chevalley, Charlotte Bunne, Andreas Krause, and Stefan Bauer. Invariant Causal Mechanisms through Distribution Matching.arXiv e-prints, art. arXiv:2206.11646, June 2022. doi: 10.48550/arXiv.2206.11646

    work page doi:10.48550/arxiv.2206.11646 2022

  7. [7]

    Differentiable patch selection for image recognition

    Jean-Baptiste Cordonnier, Aravindh Mahendran, Alexey Dosovitskiy, Dirk Weissenborn, Jakob Uszkoreit, and Thomas Unterthiner. Differentiable patch selection for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2351–2360, 2021

    2021

  8. [8]

    Vision transformers need registers

    Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. InThe Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview.net/forum?id=2dnO3LLiJ1

    2024

  9. [9]

    Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, Rodolphe Jenatton, Lucas Beyer, Michael Tschannen, Anurag Arnab, Xiao Wang, Carlos Riquelme Ruiz, Matthias Minderer, Joan Puigcerver, Utku Evci, Manoj Kumar, Sjoerd Van 11 Steenkiste,...

    2023

  10. [10]

    ImageNet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009. doi: 10.1109/CVPR.2009.5206848

    work page doi:10.1109/cvpr.2009.5206848 2009

  11. [11]

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.arXiv e-prints, art

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.arXiv e-prints, art. arXiv:2010.11929, October 2020. doi: 10.48550/ arXiv...

    Pith/arXiv arXiv 2010

  12. [12]

    Wichmann, and Wieland Brendel

    Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness.arXiv e-prints, art. arXiv:1811.12231, November 2018. doi: 10.48550/arXiv.1811.12231

    work page doi:10.48550/arxiv.1811.12231 2018

  13. [13]

    Zemel, Wieland Brendel, Matthias Bethge, and Felix A

    Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut Learning in Deep Neural Networks.arXiv e-prints, art. arXiv:2004.07780, April 2020. doi: 10.48550/arXiv.2004.07780

    work page doi:10.48550/arxiv.2004.07780 2004

  14. [14]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000–16009, 2022

    2022

  15. [15]

    Supervised Contrastive Learning.arXiv e-prints, art

    Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised Contrastive Learning.arXiv e-prints, art. arXiv:2004.11362, April 2020. doi: 10.48550/arXiv.2004.11362

    work page doi:10.48550/arxiv.2004.11362 2004

  16. [16]

    Last layer re-training is sufficient for robustness to spurious correlations.arXiv preprint arXiv:2204.02937, 2022

    Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for robustness to spurious correlations.arXiv preprint arXiv:2204.02937, 2022

    arXiv 2022

  17. [17]

    Register and [cls] tokens induce a decoupling of local and global features in large vits

    Alexander Lappe and Martin Giese. Register and [cls] tokens induce a decoupling of local and global features in large vits. In D. Belgrave, C. Zhang, H. Lin, R. Pas- canu, P. Koniusz, M. Ghassemi, and N. Chen, editors,Advances in Neural Infor- mation Processing Systems, volume 38, pages 1009–1029. Curran Associates, Inc.,

  18. [18]

    URL https://proceedings.neurips.cc/paper_files/paper/2025/file/ 01bcbd34b02f3da9700a3ddd0480c156-Paper-Conference.pdf

    2025

  19. [19]

    MetaShift: A Dataset of Datasets for Evaluating Contextual Distribution Shifts and Training Conflicts.arXiv e-prints, art

    Weixin Liang and James Zou. MetaShift: A Dataset of Datasets for Evaluating Contextual Distribution Shifts and Training Conflicts.arXiv e-prints, art. arXiv:2202.06523, February

    arXiv

  20. [20]

    doi: 10.48550/arXiv.2202.06523

    work page doi:10.48550/arxiv.2202.06523

  21. [21]

    Just train twice: Improving group robustness without training group information

    Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In Marina Meila and Tong Zhang, editors,Proceedings of the 38th International Conference on Machine Learning, volume 139 ofProceedings of Machine Learning ...

    2021

  22. [22]

    Deep learning face attributes in the wild

    Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. InProceedings of International Conference on Computer Vision (ICCV), December 2015

    2015

  23. [23]

    Think twice: Test-time reasoning for robust clip zero-shot classification

    Shenyu Lu, Zhaoying Pan, and Xiaoqian Wang. Think twice: Test-time reasoning for robust clip zero-shot classification. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2919–2929, October 2025. 12

    2025

  24. [24]

    Spawrious: A benchmark for fine control of spurious correlation biases, 2023

    Aengus Lynch, Gbètondji J-S Dovonon, Jean Kaddour, and Ricardo Silva. Spawrious: A benchmark for fine control of spurious correlation biases, 2023

    2023

  25. [25]

    Intriguing properties of vision transformers

    Muzammal Naseer, Kanchana Ranasinghe, Salman Khan, Munawar Hayat, Fahad Khan, and Ming-Hsuan Yang. Intriguing properties of vision transformers. In A. Beygelzimer, Y . Dauphin, P. Liang, and J. Wortman Vaughan, editors,Advances in Neural Information Processing Systems,

  26. [26]

    URLhttps://openreview.net/forum?id=o2mbl-Hmfgd

  27. [27]

    Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193, 2023

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193, 2023

    Pith/arXiv arXiv 2023

  28. [28]

    Learning Transferable Visual Models From Natural Language Supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision.arXiv e-prints, art. arXiv:2103.00020, February 2021. doi: 10.48550/arXiv.2103.00020

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2103.00020 2021

  29. [29]

    Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.007...

    Pith/arXiv arXiv 2024

  30. [30]

    Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization

    Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization.arXiv e-prints, art. arXiv:1911.08731, November 2019. doi: 10.48550/arXiv. 1911.08731

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv 1911

  31. [31]

    When Do We Not Need Larger Vision Models?arXiv e-prints, art

    Baifeng Shi, Ziyang Wu, Maolin Mao, Xin Wang, and Trevor Darrell. When Do We Not Need Larger Vision Models?arXiv e-prints, art. arXiv:2403.13043, March 2024. doi: 10.48550/ arXiv.2403.13043

    arXiv 2024

  32. [32]

    Oriane Siméoni, Huy V . V o, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, Francisco Massa, Daniel Haziza, Luca Wehrstedt, Jianyuan Wang, Timothée Darcet, Théo Moutakanni, Leonel Sentana, Claire Roberts, Andrea Vedaldi, Jamie Tolan, John Brandt, Camille Couprie, Julie...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2508.10104 2025

  33. [33]

    Deep CORAL: Correlation Alignment for Deep Domain Adaptation

    Baochen Sun and Kate Saenko. Deep CORAL: Correlation Alignment for Deep Domain Adaptation.arXiv e-prints, art. arXiv:1607.01719, July 2016. doi: 10.48550/arXiv.1607.01719

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1607.01719 2016

  34. [34]

    Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization,

    Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting Unreasonable Effectiveness of Data in Deep Learning Era . In2017 IEEE International Conference on Computer Vision (ICCV), pages 843–852, Los Alamitos, CA, USA, October 2017. IEEE Computer Society. doi: 10.1109/ICCV .2017.97. URLhttps://doi.ieeecomputersociety. org/10.1109/ICCV.2017.97

    work page doi:10.1109/iccv 2017

  35. [35]

    Attention Is All You Need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need.arXiv e-prints, art. arXiv:1706.03762, June 2017. doi: 10.48550/arXiv.1706.03762

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1706.03762 2017

  36. [36]

    Show, Attend and Tell: Neural Image Caption Generation with Visual Attention.arXiv e-prints, art

    Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention.arXiv e-prints, art. arXiv:1502.03044, February 2015. doi: 10.48550/ arXiv.1502.03044

    Pith/arXiv arXiv 2015

  37. [37]

    Mitigating Spurious Correlations in Multi-modal Models during Fine-tuning.arXiv e-prints, art

    Yu Yang, Besmira Nushi, Hamid Palangi, and Baharan Mirzasoleiman. Mitigating Spurious Correlations in Multi-modal Models during Fine-tuning.arXiv e-prints, art. arXiv:2304.03916, April 2023. doi: 10.48550/arXiv.2304.03916

    work page doi:10.48550/arxiv.2304.03916 2023

  38. [38]

    Scaling vision transform- ers

    Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transform- ers. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12104–12113, June 2022. 13

    2022

  39. [39]

    Sohoni, Hongyang R

    Michael Zhang, Nimit S. Sohoni, Hongyang R. Zhang, Chelsea Finn, and Christopher Ré. Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correla- tions.arXiv e-prints, art. arXiv:2203.01517, March 2022. doi: 10.48550/arXiv.2203.01517

    work page doi:10.48550/arxiv.2203.01517 2022

  40. [40]

    artifact

    Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer.International Conference on Learning Representations (ICLR), 2022. 14 A Attention Localization and Hit Rate across model families Table 3: Attention localization vs. ground-truth bounding boxes across six pretraining fam...

    arXiv 2022