Pith. sign in

REVIEW 5 major objections 4 minor 93 references

DA-VPT: Semantic-Guided Visual Prompt Tuning for Vision Transformers

T0 review · 5 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding metric-learning losses to visual prompt tuning lets each prompt specialize in a cluster of semantically related classes, turning prompts into bridges that carry class-specific information to the classification token.

desk verdict Useful method-level contribution in prompt tuning; the empirical gains look real, but the semantic-mapping interpretation needs a random-assignment control before you trust the story. read the letter →

arxiv 2505.23694 v2 pith:WIHBHPZG submitted 2025-05-29 cs.CV

Li Ren , Chen Chen , Liqiang Wang , Kien Hua This is my paper
classification cs.CV
keywords visualprompttuningparameter-efficientfine-tuningmetriclearningproxy-anchorlossvisiontransformersemanticsegmentationclass-promptmappingtransfer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that visual prompt tuning leaves the relationship between prompts and image content to chance, and that deliberately shaping that relationship makes fine-tuning more accurate and more parameter-efficient. The authors add two metric-learning losses to the VPT-Deep recipe: one that pulls each prompt toward image patches of its assigned semantic class and pushes it away from other classes, and one that aligns the class token with the prompts in the same way. A k-means step, refreshed every epoch, maps the dataset's classes onto about 20 prompts so each prompt acts as a proxy for a cluster of related classes. The result is that prompts behave as semantic bridges: they collect class-specific information from image patches and hand it to the class token through attention. On 24 recognition tasks the method reports gains over VPT-Deep of 2.83 percentage points on FGVC and 4.18 on VTAB-1K with a supervised ViT-B, and it also improves semantic segmentation while tuning about 4 percent of the parameters.

What carries the argument

The load-bearing mechanism is a Proxy-Anchor-style metric-learning loss applied to the prompt tokens themselves, paired with a dynamic class-to-prompt mapping. In the loss, each prompt $p_k$ serves as a proxy for a cluster of classes: for a batch, $\mathcal{L}_{\mathrm{ML}}(X,P)$ raises the cosine similarity between $p_k$ and same-class visual tokens and lowers it for different-class tokens, and $\mathcal{L}_{\mathrm{ML}}(P,x_{\mathrm{cls}})$ does the same between prompts and the class token; the total loss is $\mathcal{L} = \mathcal{L}_{\mathrm{CE}} + \beta\mathcal{L}_{\mathrm{ML}}(X,P) + \lambda\mathcal{L}_{\mathrm{ML}}(P,x_{\mathrm{cls}})$. The class-to-prompt assignment comes from k-means clustering of mean-pooled class tokens from the pre-trained model, refreshed each epoch, and the loss is applied to the output of the attention layer so it compares saliency-aggregated token representations. A small auxiliary theorem, $\Delta a_i \approx a_i(1-a_i)\Delta s_i$, connects the metric-induced similarity changes to attention-weight changes, explaining why moving a prompt closer to a token in embedding space gives that token more attention.

What would settle it

Run DA-VPT on a recognition task but replace the semantic k-means mapping with a fixed random class-to-prompt assignment that is permuted each epoch; if accuracy stays at the same level as DA-VPT, the semantic clustering is not what drives the gains, whereas a drop back toward VPT-Deep would confirm the semantic assignment is load-bearing. Alternatively, measure the stability of k-means centroids across epochs on a dataset like CUB: if assignments flip heavily while accuracy remains high, the dynamic mapping is not the source of improvement.

Watch

Extended reading notes

Core claim

The central discovery is that the distribution of prompts, not just their position or connectivity, determines how well they transfer. VPT-Deep initializes prompts randomly and lets the classification loss shape them indirectly; DA-VPT instead imposes explicit semantic constraints. Each prompt is assigned, via k-means on mean-pooled pre-trained class tokens, to a cluster of classes, and two smoothed NCA (Proxy-Anchor) losses push prompts toward same-class visual tokens and away from different-class tokens, while the class token is pulled toward its matching prompt. The paper shows this makes attention maps concentrate on class-discriminative regions in deep layers, and that the guided loss works best on the final layer. With these losses, prompts capture class-specific information and the class token aggregates it, producing consistent gains across supervised and self-supervised backbones and on dense prediction tasks.

Load-bearing premise

The k-means clustering of pre-trained class-token representations must produce stable, semantically meaningful prompt-to-class assignments that stay valid while the model is being fine-tuned; if the clusters are unstable or classes are very few, the metric guidance can mislead the prompts.

Editorial extensions

If this is right

  • If prompts are shaped by class semantics, the number of prompts can be cut: DA-VPT uses about 20 prompts and still outperforms methods that rely on more prompts, meaning the limiting factor is not prompt count but prompt informativeness.
  • The gains are largest on self-supervised backbones (MAE, MoCo-v3), where DA-VPT+ surpasses even full fine-tuning on VTAB-1K, suggesting semantic guidance compensates for missing label structure during pre-training.
  • Because the metric losses make bias tuning helpful, PEFT methods that previously found bias tuning harmful can be combined with distribution guidance.
  • The approach transfers to dense prediction: on ADE20K and PASCAL Context, DA-VPT+ reaches near-full-tuning mIoU with 4.3 percent of the tunable parameters.
  • The attention-similarity relationship implies the guidance effect is local and layer-dependent, so applying the loss to deeper layers, where semantics are abstract, is where most of the benefit comes from.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the semantic-bridge mechanism is real, the same k-means-plus-metric recipe should transfer to other token-based architectures and modalities, such as frozen text or multimodal encoders, where prompts could be assigned to clusters of label semantics per task.
  • A testable extension: instead of re-clustering every epoch, one could learn the class-to-prompt assignment end-to-end with a differentiable assignment, which would remove the k-means refresh cost that the paper shows is non-negligible in early epochs.
  • The dependence on image-level class labels is a potential ceiling: in weakly labeled or open-set tasks, the metric positives would be noisy, and the method would likely need pseudo-labeling or outlier prompts to remain effective, a direction the paper only partially covers.
  • The attention-similarity theorem suggests a diagnostic: measuring per-prompt attention concentration on same-class patches could serve as a training-free proxy for whether the metric guidance is working.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes DA-VPT, an extension of VPT-Deep that adds two metric-learning losses to the fine-tuning objective: L_ML(X,P) between prompts and visual tokens and L_ML(P,x_cls) between prompts and the class token (Eq. 5). Classes are assigned to prompts by k-means clustering of pre-trained CLS representations, with the mapping updated each epoch (Section 3.4). DA-VPT+ additionally unfreezes bias terms in the attention Key and Value projections (Section 3.5). The authors report consistent gains over VPT-Deep and related methods on FGVC, VTAB-1K, and two segmentation benchmarks, and they argue via attention-map visualizations that prompts become semantic bridges between image patches and the class token.

Significance. If substantiated, the method is a useful contribution to parameter-efficient fine-tuning: it adds a principled distributional regularizer to visual prompts, is evaluated across 24 recognition tasks and two segmentation benchmarks on supervised, MAE, and MoCo-v3 backbones, ablates the main components, compares alternative metric losses and connection structures, and releases code. The strongest evidence is the consistent improvement of DA-VPT+ over VPT-Deep and E2VPT in Tables 1-3. However, the load-bearing interpretive claim - that the specific semantic class-to-prompt mapping, rather than the extra metric objective alone, drives the gains - is not tested, and the effect sizes are reported without variance. The paper is therefore promising but needs additional controls and reporting changes before the central claim is established.

major comments (5)
  1. [Section 3.4, Eq. (4), Table 4] The paper never ablates the class-to-prompt assignment itself. The Proxy-Anchor loss in Eq. (4) is a contrastive objective that pulls each prompt toward whatever subset of visual tokens it is assigned to and pushes it away from other tokens; this objective will improve discrimination even if the assignment is arbitrary. A control with a random or shuffled class-to-prompt mapping, and a comparison between the dynamic re-clustering in Section 3.4 and a static mapping computed once from pre-trained features, are needed. If a random mapping performs as well as the k-means mapping, the reported gains support only the weaker claim that an additional metric objective helps, not the 'semantic bridge' interpretation in the title and abstract.
  2. [Section 5.1, Appendix A.3] The per-task hyperparameter search is very large and its budget is not matched against baselines. Appendix A.3 searches over base learning rate {1e-3, 5e-4, 1e-4, 5e-5}, weight decay {0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0}, guided-layer position {12, 10, 8, 6, 4, 2, 0}, and prompt counts {5, 10, 20, 40}. The main tables compare DA-VPT to baselines whose own search protocols and budgets are not specified, and no per-dataset selected configurations are given. Because the reported advantage could partly reflect a larger search budget, the paper should report the search protocol used for each baseline, the number of trials per dataset, and the chosen hyperparameters, or provide a budget-matched comparison.
  3. [Appendix A.2, Tables 1, 4, 9] Appendix A.2 states that mean accuracy across three runs is reported, but no standard deviations or per-seed values appear in the main tables. Several reported differences are very small, for example Table 4 shows +0.02 and +0.06 pp for individual LML components on VTAB-Natural, and Table 9 contains numerous 0.1-0.3 pp differences between methods. These margins are within typical seed noise for VTAB-1K, so the statistical significance of several claimed gains cannot be evaluated. The authors should add standard deviations or release per-seed numbers for the main results.
  4. [Appendix A.2, segmentation paragraph; Table 2] For multi-class segmentation, the target class for prompt assignment is selected randomly per iteration, so the assigned class may not be present in the image at all. This makes the class-to-prompt mapping effectively decoupled from pixel content in the segmentation experiments, weakening the semantic-guidance interpretation of Table 2. At minimum, the authors should ablate this choice (e.g., selecting only classes present in the sample, or using a predicted dominant class) and report sensitivity, since the segmentation results are a stated contribution.
  5. [Appendix F vs Appendix A.2] The treatment of tasks with few classes is internally inconsistent. Appendix A.2 says extra unassigned prompts are 'not guided by semantic metrics loss,' while Appendix F recommends supplemental prompts that are 'guided by the metric learning loss.' Appendix F also lists Retinopathy as a limited-class task even though Table 6 reports five classes for it, which does not match the stated C < 5 criterion. This contradiction should be resolved because it affects reproduction of the VTAB-1K results on Patch Camelyon, Retinopathy, and KITTI-Dist.
minor comments (4)
  1. [Appendix E] The heading 'The Proof and Detial of theorem 1' contains a typo ('Detial' should be 'Detail'), and the appendix restates Theorem 1 as Theorem 2 without explaining the relationship between the two labels.
  2. [Figures 4d and 5] Figures 4d and Figure 5 appear to report the same prompt-initialization experiment with slightly different numbers for the VTAB-Natural baseline (80.4 vs 80.1). The duplication should be removed or the discrepancy explained.
  3. [Appendix E, Theorem 1 proof] The proof assumes that the weighted sum of off-diagonal perturbations is negligible, but this assumption is stated only in the appendix; it should be stated in the main text alongside Theorem 1 so that the approximation's scope is clear.
  4. [Section 3.1, Eq. (1)] Equation (1) uses the placeholder '[ ]' for the reserved prompt positions in the next layer, which is confusing; writing the output sequence with explicit indexed variables would improve readability.

Circularity Check

1 steps flagged · score 2.0 of 10

One self-definitional support for the 'semantic bridge' narrative; benchmark gains remain independent.

  1. self definitional [Section 4.2 'Analysis of Guided Attention Maps', with Eq. (5) from Section 3.2 and Section 3.4]
    "Our positive prompt (∗p) successfully identifies informative patches that are subsequently selected by the [CLS] token. ... These visualizations demonstrate that positively labeled prompts serve as effective 'bridges' for semantic information flow to the [CLS] token in deep layers."

    The 'positive prompt' is the prompt whose assigned class matches the image's class via the k-means mapping of Section 3.4. Eq. (5) trains LML(X,P) to make that prompt more similar to same-class visual tokens than to other classes, and LML(P,x_cls) to make the CLS token more similar to the corresponding prompt. Thus the attention-map finding that the positive prompt attends to class-discriminative patches and the CLS token then attends to them is a direct consequence of the optimization objective, not an independent empirical discovery of emergent bridging. The headline accuracy improvements, however, are benchmark comparisons against external baselines and are not fitted by these losses, so the circularity is limited to the interpretive 'bridge' claim.

full rationale

The paper's quantitative claims are self-contained: DA-VPT and DA-VPT+ are compared with VPT-Deep, E2VPT, and other methods on 24 recognition and 2 segmentation tasks, and the reported gains are not fitted values or renamed training losses. The metric loss hyperparameters (delta, tau) come from Proxy-Anchor defaults, and the class-to-prompt k-means mapping uses pre-trained features, not the evaluation labels. The only circular-adjacent step is interpretive: the 'semantic bridge' behavior shown in attention maps is largely enforced by Eq. (5), since LML(X,P) and LML(P,x_cls) directly optimize prompt-to-patch and CLS-to-prompt similarity. This makes the qualitative 'bridge' demonstration partly by construction, but it is not the load-bearing evidence for the headline improvement. The missing random-mapping control noted by the skeptic is a genuine underdetermination of the semantic-guidance interpretation, but it is a missing control, not a circular reduction. Self-citations ([64]-[66]) appear only in related-work discussion and are not load-bearing. No fitted parameter is renamed as a prediction, and no uniqueness claim is imported from the authors' prior work.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or ontological entities. Its dependencies are standard deep metric learning losses, the ViT backbone, and a k-means clustering procedure. The free parameters are all hyperparameters or design choices that the paper tunes per task, which is a source of soundness risk.

free parameters (5)
  • beta and lambda (loss weights in Eq. 5) = searched per task; optimal below 1.0 (Fig. 6)
    Controls the trade-off between cross-entropy and the two metric losses; the paper notes these require careful tuning per backbone and task (Appendix F).
  • Number of prompts M = approximately 20 for most tasks
    Set by search (Table 5); affects prompt capacity and the k-means class-to-prompt mapping. Tasks with very few classes need extra unguided prompts (Appendix F).
  • Number and location of guided layers = final layer is optimal in most cases (Fig. 4a)
    Selected per model; Figure 4a shows sensitivity to layer choice across pre-training methods.
  • Margin delta and temperature tau in Proxy-Anchor = delta = 32, tau = 10
    Adopted as defaults from Kim et al.; they are fixed, not searched, but still influence the metric loss geometry.
  • Choice of bias terms to unfreeze (bK, bV) = bK and bV only
    Selected empirically via Figure 4b; a design choice that affects DA-VPT+ performance.
assumptions (4)
  • domain assumption Pre-trained [CLS] representations clustered with k-means yield a stable, semantically meaningful class-to-prompt assignment.
    Section 3.4 builds the class-prompt mapping on k-means of mean-pooled pre-trained CLS tokens; if clusters are unstable, the guidance can be misdirected.
  • ad hoc to paper Visual tokens after the attention layer (MHSA output) serve as a sufficient saliency aggregation for positive/negative sample selection.
    Section 3.3 claims to select saliency patches but directly uses all attention outputs; no explicit selection mechanism is implemented.
  • domain assumption Cosine similarity between prompts and tokens is a valid proxy for attention match in the optimized attention map.
    Section 3.2 argues that closer cosine distance implies higher attention probability; Theorem 1 gives a first-order approximation but does not establish the claim as a theorem.
  • domain assumption Proxy-Anchor loss is stable and appropriate under the M << N asymmetry between prompts and visual tokens.
    Section 4.3 asserts this on theoretical intuition and supports it with an empirical comparison in Appendix D, not with a proof.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DA-VPT: Semantic-Guided Visual Prompt Tuning for Vision Transformers." pith.science (2026). https://pith.science/paper/WIHBHPZG

@misc{pith2026250523694,
  author       = {Pith},
  title        = {Pith review of: DA-VPT: Semantic-Guided Visual Prompt Tuning for Vision Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WIHBHPZG}},
  note         = {Machine review of arXiv:2505.23694}
}
read the original abstract

Visual Prompt Tuning (VPT) has become a promising solution for Parameter-Efficient Fine-Tuning (PEFT) approach for Vision Transformer (ViT) models by partially fine-tuning learnable tokens while keeping most model parameters frozen. Recent research has explored modifying the connection structures of the prompts. However, the fundamental correlation and distribution between the prompts and image tokens remain unexplored. In this paper, we leverage metric learning techniques to investigate how the distribution of prompts affects fine-tuning performance. Specifically, we propose a novel framework, Distribution Aware Visual Prompt Tuning (DA-VPT), to guide the distributions of the prompts by learning the distance metric from their class-related semantic data. Our method demonstrates that the prompts can serve as an effective bridge to share semantic information between image patches and the class token. We extensively evaluated our approach on popular benchmarks in both recognition and segmentation tasks. The results demonstrate that our approach enables more effective and efficient fine-tuning of ViT models by leveraging semantic information to guide the learning of the prompts, leading to improved performance on various downstream vision tasks.

Figures

Figures reproduced from arXiv: 2505.23694 by the authors.

Figure 1
Figure 1. Comparison between VPT-Deep and DA-VPT. Left (VPT-Deep): Prompts are guided solely by the recognition task, leading to unconstrained distributions between prompts and vi￾sual tokens. This allows prompts to attract features from arbitrary classes, potentially hindering the class token’s ability to aggregate class-specific information. Right (DA-VPT): Prompts are jointly optimized by the main task and semantic metric … view at source ↗
Figure 2
Figure 2. Framework Overview. Our method establishes semantic prompt-class mappings by clustering class representations into M clusters (M = number of prompts). Prompts are guided through a metric space using smoothed proxy NCA loss LML between prompts and attention-based output tokens, enabling each prompt to capture information from its assigned semantic cluster. A similar metric guides [CLS] token-prompt relationships. The… view at source ↗
Figure 3
Figure 3. Comparison of attention patterns between VPT-Deep and our method on CUB dataset samples. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: 4a Illustrates the impact of the number and position of the layers to which the proposed metric learning loss is applied. 4c This figure shows the latency of the k-means calculation in each epoch. 4b Illustrates the importance of each category of efficient bias measure…
Figure 5
Figure 5. Figure 5: Impact of Prompt Initialization Strategies. Com￾parative analysis of model performance under different prompt initialization schemes. Results demonstrate that the background value initialization method proposed in Self-SPT [82], which uses mean pooled visual tokens, sh…
Figure 6
Figure 6. Figure 6: Impact of Hyperparameters in Three Pre-trained Models on CUB-200-2011: This figure illustrates the impact of hyper￾parameters on the performance of our proposed method across three pre-trained models (Supervised ViT, MAE, and MoCo-v3) on the CUB-200-2011 dataset. The h…
Figure 7
Figure 7. Figure 7: Visualization of Class-Specific Attention Maps. 7a,7b,7c: Examples from CUB-200-2011 showing fine-grained bird features. 7d: Examples from Stanford Dogs highlighting breed-specific characteristics. These visualizations illustrate the model’s ability to capture class-re…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

93 extracted references · 58 canonical work pages

  1. [1]

    Deepmind lab

    Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Si- mon Green, Víctor Valdés, Amir Sadik, et al. Deepmind lab. arXiv preprint arXiv:1612.03801, 2016. 13

    arXiv 2016

  2. [2]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS, 33:1877–1901, 2020. 2

    1901

  3. [3]

    One-for-all: Generalized lora for parameter- efficient fine-tuning

    Arnav Chavan, Zhuang Liu, Deepak Gupta, Eric Xing, and Zhiqiang Shen. One-for-all: Generalized lora for parameter- efficient fine-tuning. arXiv preprint arXiv:2306.07967, 2023. 2

    arXiv 2023

  4. [4]

    Adaptformer: Adapting vision transformers for scalable visual recognition

    Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yib- ing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. NeurIPS,

  5. [5]

    An empiri- cal study of training self-supervised vision transformers

    Xinlei Chen, Saining Xie, and Kaiming He. An empiri- cal study of training self-supervised vision transformers. In CVPR, pages 9640–9649, 2021. 6, 7

    2021

  6. [6]

    Person re-identification by multi-channel parts-based cnn with improved triplet loss function

    De Cheng, Yihong Gong, Sanping Zhou, Jinjun Wang, and Nanning Zheng. Person re-identification by multi-channel parts-based cnn with improved triplet loss function. In CVPR, pages 1335–1344, 2016. 3, 5

    2016

  7. [7]

    Remote sensing image scene classification: Benchmark and state of the art

    Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, pages 1865–1883, 2017. 13

    2017

  8. [8]

    Learning a similarity metric discriminatively, with application to face verification

    Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. In CVPR. IEEE, 2005. 3

    2005

Show all 93 references
  1. [9]

    Describing textures in the wild

    Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, pages 3606–3613, 2014. 13

    2014

  2. [10]

    Multi-head attention: Collaborate instead of concatenate

    Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. Multi-head attention: Collaborate instead of concatenate. arXiv preprint arXiv:2006.16362, 2020. 5

    2006 arXiv

  3. [11]

    Vision transformers need registers

    Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. ICLR, 2024. 5

    2024

  4. [12]

    Scaling vision transformers to 22 billion pa- rameters

    Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdul- mohsin, et al. Scaling vision transformers to 22 billion pa- rameters. In ICML, pages 7480–7512. PMLR, 2023. 2

    2023

  5. [13]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248–255. Ieee, 2009. 1, 6

    2009

  6. [14]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint a...

    2010 arXiv

  7. [15]

    Diabetic retinopathy detection, 2015

    Emma Dugas, Jorge Jared, and Will Cukierski. Diabetic retinopathy detection, 2015. 13

    2015

  8. [16]

    Hyperbolic vision transform- ers: Combining improvements in metric learning

    Aleksandr Ermolov, Leyla Mirvakhabova, Valentin Khrulkov, Nicu Sebe, and Ivan Oseledets. Hyperbolic vision transform- ers: Combining improvements in metric learning. In CVPR, pages 7409–7419, 2022. 3

    2022

  9. [17]

    One-shot learn- ing of object categories

    Li Fei-Fei, Robert Fergus, and Pietro Perona. One-shot learn- ing of object categories. IEEE TPAMI, 28(4):594–611, 2006. 13

    2006

  10. [18]

    Compositional prompt tuning with motion cues for open-vocabulary video relation detection

    Kaifeng Gao, Long Chen, Hanwang Zhang, Jun Xiao, and Qianru Sun. Compositional prompt tuning with motion cues for open-vocabulary video relation detection. ICLR, 2023. 2

    2023

  11. [19]

    Tuning pre-trained model via moment probing

    Mingze Gao, Qilong Wang, Zhenyi Lin, Pengfei Zhu, Qinghua Hu, and Jingbo Zhou. Tuning pre-trained model via moment probing. In CVPR, pages 11803–11813, 2023. 7, 12

    2023

  12. [20]

    Vi- sual prompt tuning for test-time domain adaptation

    Yunhe Gao, Xingjian Shi, Yi Zhu, Hao Wang, Zhiqiang Tang, Xiong Zhou, Mu Li, and Dimitris N Metaxas. Vi- sual prompt tuning for test-time domain adaptation. arXiv preprint arXiv:2210.04831, 2022. 2

    2022 arXiv

  13. [21]

    Fine-grained car detection for visual census estimation

    Timnit Gebru, Jonathan Krause, Yilun Wang, Duyun Chen, Jia Deng, and Li Fei-Fei. Fine-grained car detection for visual census estimation. In AAAI, 2017. 12, 13

    2017

  14. [22]

    Vision meets robotics: The kitti dataset

    Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The In- ternational Journal of Robotics Research, pages 1231–1237,

  15. [23]

    Dimension- ality reduction by learning an invariant mapping

    Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimension- ality reduction by learning an invariant mapping. In CVPR. IEEE, 2006. 3

    2006

  16. [24]

    Eˆ 2vpt: An effec- tive and efficient approach for visual prompt tuning

    Cheng Han, Qifan Wang, Yiming Cui, Zhiwen Cao, Wenguan Wang, Siyuan Qi, and Dongfang Liu. Eˆ 2vpt: An effec- tive and efficient approach for visual prompt tuning. arXiv preprint arXiv:2307.13770, 2023. 2, 7

    2023 arXiv

  17. [25]

    Sensitivity-aware visual parameter-efficient fine- tuning

    Haoyu He, Jianfei Cai, Jing Zhang, Dacheng Tao, and Bohan Zhuang. Sensitivity-aware visual parameter-efficient fine- tuning. In CVPR, pages 11825–11835, 2023. 7

    2023

  18. [26]

    Momentum contrast for unsupervised visual repre- sentation learning

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual repre- sentation learning. In CVPR, pages 9729–9738, 2020. 1

    2020

  19. [27]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, pages 16000–16009, 2022. 1, 6, 7

    2022

  20. [28]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, pages 16000–16009, 2022. 2

    2022

  21. [29]

    Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification

    Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, pages 2217–2226, 2019. 13

    2019

  22. [30]

    In defense of the triplet loss for person re-identification

    Alexander Hermans, Lucas Beyer, and Bastian Leibe. In defense of the triplet loss for person re-identification. arXiv preprint arXiv:1703.07737, 2017. 5

    2017 arXiv

  23. [31]

    Parameter-efficient transfer learning for nlp

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In ICML, pages 2790–2799. PMLR, 2019. 1, 2, 7, 17

    2019

  24. [32]

    Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021. 1

    2021 arXiv

  25. [33]

    Visual prompt tuning

    Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, pages 709–727. Springer, 2022. 1, 2, 3, 6, 7, 12, 13, 17

    2022

  26. [34]

    Clevr: A diagnostic dataset for compositional language and elementary visual reasoning

    Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, pages 2901–2910, 2017. 13

    2017

  27. [35]

    Novel dataset for fine-grained image cat- egorization: Stanford dogs

    Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li. Novel dataset for fine-grained image cat- egorization: Stanford dogs. In Proc. CVPR workshop on fine-grained visual categorization (FGVC). Citeseer, 2011. 12, 13

    2011

  28. [36]

    Proxy anchor loss for deep metric learning

    Sungyeon Kim, Dongwon Kim, Minsu Cho, and Suha Kwak. Proxy anchor loss for deep metric learning. In CVPR, pages 3238–3247, 2020. 3, 4, 5

    2020

  29. [37]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In CVPR, pages 4015–4026, 2023. 1

    2023

  30. [38]

    Do better imagenet models transfer better? In CVPR, pages 2661–2671,

    Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In CVPR, pages 2661–2671,

  31. [39]

    Cross-image-attention for conditional embeddings in deep metric learning

    Dmytro Kotovenko, Pingchuan Ma, Timo Milbich, and Björn Ommer. Cross-image-attention for conditional embeddings in deep metric learning. In CVPR, pages 11070–11081, 2023. 3

    2023

  32. [40]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 13

    2009

  33. [41]

    M-adda: Unsuper- vised domain adaptation with deep metric learning

    Issam H Laradji and Reza Babanezhad. M-adda: Unsuper- vised domain adaptation with deep metric learning. Domain adaptation for visual understanding, pages 17–31, 2020. 3

    2020

  34. [42]

    Learning methods for generic object recognition with invariance to pose and lighting

    Yann LeCun, Fu Jie Huang, and Leon Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In CVPR. IEEE, 2004. 13

    2004

  35. [43]

    The power of scale for parameter-efficient prompt tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021. 2

    2021 arXiv

  36. [44]

    Prefix-tuning: Optimiz- ing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimiz- ing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021. 1, 2

    2021 arXiv

  37. [45]

    Scaling & shifting your features: A new baseline for efficient model tuning

    Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Scaling & shifting your features: A new baseline for efficient model tuning. Neurips, 35:109–123, 2022. 2, 7, 12

    2022

  38. [46]

    Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing

    Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hi- roaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1–35,

  39. [47]

    P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks

    Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengx- iao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602, 2021. 2

    arXiv

  40. [48]

    Sgdr: Stochastic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983,

    arXiv

  41. [49]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 12

    2017 arXiv

  42. [50]

    Exploring the limits of weakly supervised pretraining

    Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaim- ing He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV, pages 181–196, 2018. 1

    2018

  43. [51]

    dsprites: Disentanglement testing sprites dataset,

    Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset,

  44. [52]

    The role of context for object detection and semantic segmentation in the wild

    Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In CVPR, pages 891–898,

  45. [53]

    No fuss distance metric learning using proxies

    Yair Movshovitz-Attias, Alexander Toshev, Thomas K Leung, Sergey Ioffe, and Saurabh Singh. No fuss distance metric learning using proxies. In CVPR, pages 360–368, 2017. 3

    2017

  46. [54]

    Reading digits in natural images with unsupervised feature learning

    Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, page 7. Granada, Spain, 2011. 13

    2011

  47. [55]

    Toward under- standing catastrophic forgetting in continual learning

    Cuong V Nguyen, Alessandro Achille, Michael Lam, Tal Has- sner, Vijay Mahadevan, and Stefano Soatto. Toward under- standing catastrophic forgetting in continual learning. arXiv preprint arXiv:1908.01091, 2019. 1

    1908 arXiv

  48. [56]

    A visual vocabulary for flower classification

    M-E Nilsback and Andrew Zisserman. A visual vocabulary for flower classification. In CVPR, pages 1447–1454. IEEE,

  49. [57]

    Automated flower classification over a large number of classes

    Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP. IEEE, 2008. 12, 13

    2008

  50. [58]

    Cats and dogs

    Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR, pages 3498–3505. IEEE, 2012. 13

    2012

  51. [59]

    Recall@ k surro- gate loss with large batches and similarity mixup

    Yash Patel, Giorgos Tolias, and Jiˇrí Matas. Recall@ k surro- gate loss with large batches and similarity mixup. In CVPR, pages 7502–7511, 2022. 3

    2022

  52. [60]

    Sa 2vp: Spatially aligned-and- adapted visual prompt

    Wenjie Pei, Tongqi Xia, Fanglin Chen, Jinsong Li, Jiandong Tian, and Guangming Lu. Sa 2vp: Spatially aligned-and- adapted visual prompt. In AAAI, 2024. 2

    2024

  53. [61]

    Adapterhub: A framework for adapting transformers

    Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Ka- math, Ivan Vuli ´c, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. Adapterhub: A framework for adapting transformers. arXiv preprint arXiv:2007.07779, 2020. 1

    2007 arXiv

  54. [62]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748–8763. PMLR, 2021. 2

    2021

  55. [63]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748–8763. PMLR, 2021. 1

    2021

  56. [64]

    Beyond the deep metric learning: enhance the cross-modal matching with adversarial discriminative domain regularization

    Li Ren, Kai Li, LiQiang Wang, and Kien Hua. Beyond the deep metric learning: enhance the cross-modal matching with adversarial discriminative domain regularization. In ICPR, pages 10165–10172. IEEE, 2021. 3

    2021

  57. [65]

    To- wards improved proxy-based deep metric learning via data- augmented domain adaptation

    Li Ren, Chen Chen, Liqiang Wang, and Kien Hua. To- wards improved proxy-based deep metric learning via data- augmented domain adaptation. In AAAI, 2024. 3

    2024

  58. [66]

    Li Ren, Chen Chen, Liqiang Wang, and Kien A. Hua. Learn- ing semantic proxies from visual prompts for parameter- efficient fine-tuning in deep metric learning. In ICLR, 2024. 1, 3

    2024

  59. [67]

    Non-isotropy regularization for proxy-based deep metric learning

    Karsten Roth, Oriol Vinyals, and Zeynep Akata. Non-isotropy regularization for proxy-based deep metric learning. InCVPR, pages 7420–7430, 2022. 3

    2022

  60. [68]

    Neighbourhood component analysis

    Sam Roweis, Geoffrey Hinton, and Ruslan Salakhutdinov. Neighbourhood component analysis. NeurIPS, 2004. 3

    2004

  61. [69]

    Revisiting unreasonable effectiveness of data in deep learning era

    Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In ICCV, pages 843–852, 2017. 1

    2017

  62. [70]

    Prox- ynca++: Revisiting and revitalizing proxy neighborhood com- ponent analysis

    Eu Wern Teh, Terrance DeVries, and Graham W Taylor. Prox- ynca++: Revisiting and revitalizing proxy neighborhood com- ponent analysis. In ECCV, pages 448–464. Springer, 2020. 3, 5

    2020

  63. [71]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Am- jad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. 2

    2023 arXiv

  64. [72]

    Convolu- tional visual prompt for robust visual perception

    Yun-Yun Tsai, Chengzhi Mao, and Junfeng Yang. Convolu- tional visual prompt for robust visual perception. Advances in Neural Information Processing Systems, 36, 2024. 3

    2024

  65. [73]

    Visual query tuning: Towards effective usage of intermediate representa- tions for parameter and memory efficient transfer learning

    Cheng-Hao Tu, Zheda Mai, and Wei-Lun Chao. Visual query tuning: Towards effective usage of intermediate representa- tions for parameter and memory efficient transfer learning. In CVPR, pages 7725–7735, 2023. 2

    2023

  66. [74]

    Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection

    Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Be- longie. Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection. In CVPR, pages 595–...

    2015

  67. [75]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 30, 2017. 2

    2017

  68. [76]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Neurips, 30, 2017. 3

    2017

  69. [77]

    Rotation equivariant cnns for digital pathology

    Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Co- hen, and Max Welling. Rotation equivariant cnns for digital pathology. In MICCAI, pages 210–218. Springer, 2018. 13

    2018

  70. [78]

    It takes two to tango: Mixup for deep metric learning

    Shashanka Venkataramanan, Bill Psomas, Yannis Avrithis, Ewa Kijak, Laurent Amsaleg, and Konstantinos Karantzalos. It takes two to tango: Mixup for deep metric learning. ICLR,

  71. [79]

    The caltech-ucsd birds-200-2011 dataset

    Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset

    2011

  72. [80]

    Adversarial cross-modal retrieval

    Bokun Wang, Yang Yang, Xing Xu, Alan Hanjalic, and Heng Tao Shen. Adversarial cross-modal retrieval. In Multi- media, pages 154–162, 2017. 3

    2017

  73. [81]

    Adapting shortcut with normalizing flow: An efficient tuning framework for visual recognition

    Yaoming Wang, Bowen Shi, Xiaopeng Zhang, Jin Li, Yuchen Liu, Wenrui Dai, Chenglin Li, Hongkai Xiong, and Qi Tian. Adapting shortcut with normalizing flow: An efficient tuning framework for visual recognition. In CVPR. IEEE, 2023. 7

    2023

  74. [82]

    Revisiting the power of prompt for visual tuning

    Yuzhu Wang, Lechao Cheng, Chaowei Fang, Dingwen Zhang, Manni Duan, and Meng Wang. Revisiting the power of prompt for visual tuning. arXiv preprint arXiv:2402.02382,

    arXiv

  75. [83]

    Sun database: Large-scale scene recog- nition from abbey to zoo

    Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recog- nition from abbey to zoo. In CVPR, pages 3485–3492. IEEE,

  76. [84]

    Difffit: Unlocking transferability of large diffusion models via simple parameter- efficient fine-tuning

    Enze Xie, Lewei Yao, Han Shi, Zhili Liu, Daquan Zhou, Zhaoqiang Liu, Jiawei Li, and Zhenguo Li. Difffit: Unlocking transferability of large diffusion models via simple parameter- efficient fine-tuning. In CVPR, pages 4230–4239, 2023. 2

    2023

  77. [85]

    Improving visual prompt tuning for self- supervised vision transformers

    Seungryong Yoo, Eunji Kim, Dahuin Jung, Jungbeom Lee, and Sungroh Yoon. Improving visual prompt tuning for self- supervised vision transformers. InICML, pages 40075–40092. PMLR, 2023. 2, 7, 14

    2023

  78. [86]

    Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models

    Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10199,

  79. [87]

    A large-scale study of representation learning with the visual task adaptation benchmark

    Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, An- dre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv pr...

    1910 arXiv

  80. [88]

    Mosa: Mixture of sparse adapters for visual efficient tuning

    Qizhe Zhang, Bocheng Zou, Ruichuan An, Jiaming Liu, and Shanghang Zhang. Mosa: Mixture of sparse adapters for visual efficient tuning. arXiv preprint arXiv:2312.02923 ,

    arXiv

  81. [89]

    Neural prompt search

    Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search. arXiv preprint arXiv:2206.04673, 2022. 2

    2022 arXiv

  82. [90]

    Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers

    Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xi- ang, Philip HS Torr, et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In CVPR, pages 6881–6890, 2021. 12

    2021

  83. [91]

    Semantic understand- ing of scenes through the ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understand- ing of scenes through the ade20k dataset. IJCV, 127:302–321,

  84. [93]

    Following established proto- cols [19, 33, 45], we report mean accuracy across three runs with different random seeds

    We determine optimal hyperparameters through cross- validation on the validation set. Following established proto- cols [19, 33, 45], we report mean accuracy across three runs with different random seeds. Segmentation Tasks. We implement our experiments us- ing the SETR framew...

  85. [2019]

    Details About the Experiments A.1

    6, 13 Appendix A. Details About the Experiments A.1. Datasets Classification Datasets. FGVC encompasses five fine- grained visual classification datasets: CUB-200-2011 [79], NABirds [74], Oxford Flowers [ 57], Stanford Dogs [ 35], and Stanford Cars [21]. Following Jia et al. [...

    2011

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.