Pith. sign in

REVIEW 3 major objections 4 minor 51 references

Attacking Attention of Foundation Models Disrupts Downstream Tasks

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single label-free perturbation can disrupt five vision tasks built on CLIP, ViT, and DINOv2 backbones.

desk verdict The empirical picture is probably real, but the paper's stated objective (minimize an L2 embedding distance) would preserve embeddings, so the reported attack success cannot come from the math as written; fix the sign, add EmbedAttack, and it becomes a credible contribution. read the letter →

arxiv 2506.05394 v3 pith:BDTU7AU7 submitted 2025-06-03 cs.CR cs.LG

Hondamunige Prasanna Silva , Federico Becattini , Lorenzo Seidenari This is my paper · ORCID
classification cs.CRcs.LG
keywords adversarialattackvisiontransformerCLIPDINOv2attentionmanipulationtask-agnosticembeddingdisruptionfoundationmodelrobustness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a single adversarial perturbation computed purely from a vision transformer's attention and embeddings can degrade a broad set of downstream tasks, without access to labels or text. The authors target CLIP, ViT, and DINOv2 backbones and evaluate on classification, image/text retrieval, captioning, depth estimation, and semantic segmentation. If the claim holds, it would show that foundation models inherit a shared vulnerability: whatever task is built on top, damaging attention and the embedding degrades it. The paper also reports that attention-focused disruption hits fine-grained spatial tasks harder, while embedding disruption hits semantic tasks harder.

What carries the argument

The attack uses two losses. The attention loss $\mathcal{L}_{\mathrm{atn}}$ is the dot product, summed over heads and token positions, between the clean attention matrix and the perturbed attention matrix at the last layer, so minimizing it makes perturbed attention orthogonal to clean attention and redistributes focus. The embedding loss $\mathcal{L}_{\mathrm{emb}}$ is written in Eq.\ 5 as the $L_2$ norm between clean and adversarial embeddings, while the accompanying text describes the goal as minimizing the dot product to push embeddings apart. The combined objective $\mathcal{L}_{\mathrm{comb}} = \alpha \mathcal{L}_{\mathrm{atn}} + \beta \mathcal{L}_{\mathrm{emb}}$, with $\beta = |\mathcal{L}_{\mathrm{atn}}/\mathcal{L}_{\mathrm{emb}}|$ at each iteration, is optimized with Weighted Adam and gradient descent over 250 iterations under a budget $\epsilon$.

What would settle it

Run the released code or re-implement the attack exactly as written, minimizing $\|E_{\mathrm{gt}} - E_{\mathrm{adv}}\|_2$, and measure downstream classification or retrieval accuracy; if the accuracy does not drop substantially, the central claim fails. Alternatively, inspect the optimizer update sign in the code: gradient descent on an $L_2$ distance would decrease the distance, contradicting the reported attack success.

Watch

Extended reading notes

Core claim

The central discovery is a task-agnostic adversarial attack on vision foundation models. The perturbation is optimized to make the attention matrix of the last transformer layer nearly orthogonal to the clean attention map, while also shifting the final embedding; the combined loss is minimized with projected gradient descent under an $\ell_\infty$ budget. With this single image-only perturbation the authors report near-total accuracy collapse in zero-shot classification, high attack success in cross-modal retrieval, severe caption degradation, and large RMSE and mIoU drops in depth and segmentation models based on DINOv2. The attack needs no downstream labels and no text branch, unlike prior multimodal attacks.

Load-bearing premise

The attack's reported success rests on the assumption that the implemented embedding loss actually pushes adversarial embeddings away from clean ones; if the $L_2$ norm in Eq.\ 5 is minimized literally, the optimizer would pull the embeddings together and the claimed disruption would not occur.

Editorial extensions

If this is right

  • A single image-only perturbation can degrade multiple tasks built on the same backbone, meaning a downstream deployer cannot assume task specificity protects against adversarial tampering.
  • Attacks that require neither labels nor text lower the barrier to attacking real deployments of CLIP-like models, since the adversary needs only white-box access to the visual encoder.
  • Attention-targeted attacks are more effective on fine-grained spatial tasks such as segmentation and depth, while embedding-targeted attacks are more effective on semantic tasks such as classification and retrieval, suggesting task-aware attack selection.
  • The method's transfer to other vision-language models such as ALBEF and TCL is limited, implying that while task-agnostic, the attack is not fully model-agnostic across different pretraining objectives.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the sign discrepancy in Eq.\ 5 is confirmed, the reported results likely come from an unstated maximization or a negative dot-product loss; the qualitative conclusion about attention vulnerability may still hold, but the exact mechanism would differ from what the equations describe.
  • The task-dependent split between attention and embedding attacks suggests a general principle: tasks that rely on spatial layout are more sensitive to attention redistribution, while tasks that rely on global semantics are more sensitive to representation shifts, which could inform robustness testing of future backbones.
  • A testable extension is to ablate the combined loss with a corrected sign to isolate which component carries the effect; the paper's separate $\mathcal{L}_{\mathrm{atn}}$ and $\mathcal{L}_{\mathrm{emb}}$ results already hint that the attention term alone drives captioning collapse.
  • The white-box threat model could be probed against black-box scenarios by querying only attention maps or using transfer from a surrogate encoder; the limited cross-model transfer reported suggests attention-based attacks on self-supervised encoders may behave differently across downstream heads.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a task-agnostic adversarial attack on vision foundation models, targeting both the attention mechanism and the image embedding of CLIP and Vision Transformer backbones. The attack is optimized with a combined loss and is claimed to transfer to multiple downstream tasks, including zero-shot classification, image/text retrieval, captioning, semantic segmentation, and depth estimation, without access to labels or text. Experiments report large performance drops on all these tasks, and the paper positions the method as an improvement over multimodal attack baselines while attacking only the image modality.

Significance. If the stated objective is corrected and the empirical results hold, the paper would make a useful contribution: it demonstrates a single image-only perturbation that disrupts several downstream tasks built on shared vision backbones, and it explicitly tests transferability. The paper is also commendable for evaluating five task families and for making code available. The main concern is that the optimization objective as written in Eq. (5) is inconsistent with the reported attack success; this must be resolved before the empirical claims can be accepted. The absence of a comparison to the closest task-agnostic baseline, EmbedAttack, also weakens the claimed state-of-the-art status.

major comments (3)
  1. [Section 3.3, Eq. (5) and Eq. (8)] The embedding loss is defined as an L2 norm to be minimized under z* = arg min_z Lcomb. Since z = 0 already attains Lemb = 0, the stated objective cannot increase the distance between clean and adversarial embeddings; it should instead preserve or even reduce the perturbation's effect on the embedding. The reported 97.9% attack success for Ours(Lemb) in Table 1 and the large drops in Table 2 are therefore inconsistent with the printed math. The prose around Eq. (5) refers to 'minimizing the dot product' between embeddings, which is a different, similarity-decreasing objective. Please state the exact objective actually optimized (e.g., maximizing ||E_gt - E_adv||_2 or minimizing a negative cosine similarity), update Eq. (5) and Eq. (8) accordingly, or provide the optimization code. This is load-bearing because all downstream-task claims derive from this objective.
  2. [Section 2, Tables 1 and 2] EmbedAttack [22] is introduced as the first task-agnostic attack on ViT backbones and is the closest baseline to the proposed method, but it is never compared on classification or retrieval, despite the paper claiming state-of-the-art results. The experiments should include EmbedAttack (or at least an image-only embedding-perturbation baseline equivalent to Ours(Lemb)) on the same tasks and budgets. Without this, the claimed advantage over existing task-agnostic attacks is not established.
  3. [Section 4, Table 3] The captioning experiments state that 'Adversarially perturbed images are generated using the CLIP-ViT-L model' while evaluating BLIP-2. It is unclear whether the attack is optimized on CLIP and then transferred to BLIP-2, or whether BLIP-2's own visual encoder is attacked. These are different threat models (transfer versus white-box), and the distinction matters for interpreting the captioning results. Please specify the exact protocol and, if the former, clarify how the attack is aligned with BLIP-2's backbone.
minor comments (4)
  1. [Abstract and Section 1] The term 'Vision Transfomers' should be corrected to 'Vision Transformers'.
  2. [Section 3.4] The balancing rule beta = alpha * |L_atn / L_emb| is undefined if L_emb is zero early in optimization; please state a numerical safeguard (e.g., adding a small epsilon to the denominator).
  3. [Table 3] In the FLICKR row for Ours(Lcomb), the value '0.03598' appears to be a typo for BLEU-1 (likely 0.3598); please verify all entries for consistency.
  4. [Table 7] The table formatting is hard to read because numbers are run together (e.g., 'Sep-Attack2.50 0.40 0.10'), and the dataset name is misspelled as 'Flicr30K'. Please fix the formatting and ensure all column entries are clearly separated.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports an empirical attack with independently measured downstream metrics; the sign inconsistency in Lemb is a correctness issue, not a circular reduction.

full rationale

This paper is an empirical attack paper, not a derivation. It proposes an optimization objective (Eq. 8) that combines an attention loss (Eq. 4) and an embedding loss (Eq. 5), then evaluates the resulting perturbation on standard downstream benchmarks (ImageNet, COCO, Flickr30K, NYU-Depth v2, ADE20K). The downstream metrics are not used as inputs to the optimization; there is no fitted parameter renamed as a prediction, and no claimed first-principles result that reduces to its own assumptions. The adaptive balancing of α and β is a normalization computed from current loss magnitudes, not a fit to downstream task performance. The method is white-box in the sense that the perturbation is optimized on the same model used for evaluation, but this is standard adversarial-attack methodology and does not make the downstream attack success circular: the attack success is measured on external datasets and task heads that are not part of the loss definition. No load-bearing self-citations appear in the paper; the references are to prior external work such as CLIP, DINOv2, SGA, Co-Attack, and EmbedAttack. There is an internal inconsistency in the stated embedding objective: Eq. 5 defines Lemb as an L2 norm to be minimized, while the surrounding text says the goal is minimizing the dot product between adversarial and clean embeddings; minimizing an L2 distance would pull embeddings together and should preserve downstream performance, yet the reported results require the opposite effect. This is a sign or formulation inconsistency that affects the correctness or reproducibility of the method, but it is not circularity in the sense of the derivation being equivalent to its inputs by construction. Therefore the circularity score is 0.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The attack uses existing model components and standard optimization hyperparameters. The hyperparameters epsilon, learning rate, iterations, alpha, beta, and the chosen attention layer are hand-picked; they are not fitted to downstream labels. No new entities are introduced. The axioms are standard assumptions about transformer architectures, pretrained representations, and the L_inf threat model.

free parameters (6)
  • epsilon (L_inf budget) = 2/255 for classification and retrieval, 8/255 for captioning, zero-shot classification, depth estimation, segmentation
    Perturbation budget chosen by the authors; larger budgets generally increase attack strength, so the comparison across tasks is not at a fixed budget.
  • learning rate = 0.01
    Initial learning rate for Weighted Adam; higher than typical for adversarial attacks, with no reported ablation.
  • optimization iterations = 250
    Number of optimization steps; no ablation showing convergence behavior.
  • alpha = 1
    Weight for attention loss in the combined objective.
  • beta = alpha * |Latn / Lemb| at each iteration
    Adaptive balance that normalizes the two loss scales; no separate search over beta.
  • attention layer = last layer
    Chosen after preliminary experiments; not ablated across tasks or layer depths.
assumptions (4)
  • domain assumption Backpropagation through attention maps and embeddings of pretrained transformers is a valid way to optimize input perturbations.
    The attack relies on gradients of Latn and Lemb with respect to the input through the full CLIP/ViT stack, as described in Section 3.1.
  • domain assumption Pretrained foundation model representations are shared across the downstream tasks evaluated, so perturbing the backbone transfers.
    The paper attacks CLIP/DINOv2 backbones and evaluates on tasks that use the same or related backbones, as laid out in Section 4.
  • domain assumption The L_inf threat model with budgets of 2/255 and 8/255 is the appropriate measure of imperceptibility for these tasks.
    All experiments use L_inf clipping; no comparison with other threat models such as L2 or patch-based attacks is provided.
  • domain assumption Image embeddings are aligned with text embeddings in CLIP, so disturbing image embeddings alone disrupts multimodal alignment.
    Section 3.3 states this alignment is 'by construction' in CLIP and uses it to justify not needing text access.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Attacking Attention of Foundation Models Disrupts Downstream Tasks." pith.science (2026). https://pith.science/paper/BDTU7AU7

@misc{pith2026250605394,
  author       = {Pith},
  title        = {Pith review of: Attacking Attention of Foundation Models Disrupts Downstream Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BDTU7AU7}},
  note         = {Machine review of arXiv:2506.05394}
}
read the original abstract

Foundation models represent the most prominent and recent paradigm shift in artificial intelligence. Foundation models are large models, trained on broad data that deliver high accuracy in many downstream tasks, often without fine-tuning. For this reason, models such as CLIP , DINO or Vision Transfomers (ViT), are becoming the bedrock of many industrial AI-powered applications. However, the reliance on pre-trained foundation models also introduces significant security concerns, as these models are vulnerable to adversarial attacks. Such attacks involve deliberately crafted inputs designed to deceive AI systems, jeopardizing their reliability. This paper studies the vulnerabilities of vision foundation models, focusing specifically on CLIP and ViTs, and explores the transferability of adversarial attacks to downstream tasks. We introduce a novel attack, targeting the structure of transformer-based architectures in a task-agnostic fashion. We demonstrate the effectiveness of our attack on several downstream tasks: classification, captioning, image/text retrieval, segmentation and depth estimation. Code available at:https://github.com/HondamunigePrasannaSilva/attack-attention

Figures

Figures reproduced from arXiv: 2506.05394 by the authors.

Figure 1
Figure 1. Our method generates adversarial noise z ∗ by attacking attention and embeddings in visual backbones of foundation mod￾els, without any knowledge of downstream tasks. Unlike other attacks, access to other modalities (e.g., text) is not required. embeddings, and it is widely used in downstream tasks in￾cluding classification, retrieval and captioning [25]. Like most foundation models, CLIP relies on attention, which … view at source ↗
Figure 2
Figure 2. Our attack disrupts both the attention mechanism and [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Attention on clean (left) and perturbed (right) image. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Comparison of adversarial and original COCO images with predicted and adversarial captions from BLIP-2. Adversarial [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Estimated depths with unattacked (second row) and at [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Semantic segmentations with unattacked (second row) [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 28 canonical work pages

  1. [22]

    Benchmarking Robust Self-Supervised Learning Across Diverse Downstream Tasks

    Antoni Kowalczuk, Jan Dubi ´nski, Atiyeh Ashari Ghomi, Yi Sui, George Stein, Jiapeng Wu, Jesse C Cresswell, Franziska Boenisch, and Adam Dziedzic. Benchmarking robust self- supervised learning across diverse downstream tasks.arXiv preprint arXiv:2407.12588, 2024. 3, 7, 8

    work page Pith review arXiv 2024

  2. [1]

    Reveal of vision transformers robustness against ad- versarial attacks.arXiv preprint arXiv:2106.03734, 2021

    Ahmed Aldahdooh, Wassim Hamidouche, and Olivier De- forges. Reveal of vision transformers robustness against ad- versarial attacks.arXiv preprint arXiv:2106.03734, 2021. 2

    arXiv 2021

  3. [2]

    Are transformers more robust than cnns?Advances in neural information processing systems, 34:26831–26843, 2021

    Yutong Bai, Jieru Mei, Alan L Yuille, and Cihang Xie. Are transformers more robust than cnns?Advances in neural information processing systems, 34:26831–26843, 2021. 3

    work page 2021

  4. [3]

    Under- standing robustness of transformers for image classification

    Srinadh Bhojanapalli, Ayan Chakrabarti, Daniel Glasner, Daliang Li, Thomas Unterthiner, and Andreas Veit. Under- standing robustness of transformers for image classification. InProceedings of the IEEE/CVF international conference on computer vision, pages 10231–10241, 2021. 3

    work page 2021

  5. [4]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. 1

    arXiv 2005

  6. [5]

    Towards evaluating the robustness of neural networks

    Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In2017 ieee symposium on security and privacy (sp), pages 39–57. Ieee, 2017. 3

    work page 2017

  7. [6]

    Poisoning web-scale training datasets is practical.arXiv preprint arXiv:2302.10149, 2023

    Nicholas Carlini, Matthew Jagielski, Christopher A Choquette-Choo, Daniel Paleka, Will Pearce, Hyrum An- derson, Andreas Terzis, Kurt Thomas, and Florian Tram `er. Poisoning web-scale training datasets is practical.arXiv preprint arXiv:2302.10149, 2023. 3

    arXiv 2023

  8. [7]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. InPro- ceedings of the IEEE/CVF international conference on com- puter vision, pages 9650–9660, 2021. 2

    2021

Show all 51 references
  1. [8]

    Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805, 2018

    Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805, 2018. 1

    2018 arXiv

  2. [9]

    Boosting adversarial at- tacks with momentum

    Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial at- tacks with momentum. InProceedings of the IEEE con- ference on computer vision and pattern recognition, pages 9185–9193, 2018. 7

    2018

  3. [10]

    An image is worth 16x16 words: Trans- formers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale.arXiv preprint ar...

    2010 arXiv

  4. [11]

    Adversarial examples for the openai clip in its zero-shot classification regime and their semantic gener- alization, 2021

    Stanislav Fort. Adversarial examples for the openai clip in its zero-shot classification regime and their semantic gener- alization, 2021. 3

    2021

  5. [12]

    Pixels still beat text: Attacking the openai clip model with text patches and adversarial pixel perturbations,

    Stanislav Fort. Pixels still beat text: Attacking the openai clip model with text patches and adversarial pixel perturbations,

  6. [13]

    Patch-fool: Are vision transformers always robust against adversarial perturbations?arXiv preprint arXiv:2203.08392, 2022

    Yonggan Fu, Shunyao Zhang, Shang Wu, Cheng Wan, and Yingyan Lin. Patch-fool: Are vision transformers always robust against adversarial perturbations?arXiv preprint arXiv:2203.08392, 2022. 3

    2022 arXiv

  7. [14]

    Explaining and harnessing adversarial examples.arXiv preprint arXiv:1412.6572, 2014

    Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples.arXiv preprint arXiv:1412.6572, 2014. 2, 3, 5, 7

    2014 arXiv

  8. [15]

    Are vision trans- formers robust to patch perturbations? InEuropean Con- ference on Computer Vision, pages 404–421

    Jindong Gu, V olker Tresp, and Yao Qin. Are vision trans- formers robust to patch perturbations? InEuropean Con- ference on Computer Vision, pages 404–421. Springer, 2022. 3

    2022

  9. [16]

    Sa-attack: Improving adversar- ial transferability of vision-language pre-training models via self-augmentation.arXiv preprint arXiv:2312.04913, 2023

    Bangyan He, Xiaojun Jia, Siyuan Liang, Tianrui Lou, Yang Liu, and Xiaochun Cao. Sa-attack: Improving adversar- ial transferability of vision-language pre-training models via self-augmentation.arXiv preprint arXiv:2312.04913, 2023. 3, 5, 6

    2023 arXiv

  10. [17]

    Black-box adversarial attacks with limited queries and information

    Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. InInternational conference on machine learn- ing, pages 2137–2146. PMLR, 2018. 7

    2018

  11. [18]

    Scal- ing up vision-language pretraining

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V Le, Tom Duerig, and Claire Song. Scal- ing up vision-language pretraining. InInternational Confer- ence on Machine Learning, pages 4904–4916. PMLR, 2021. 2

    2021

  12. [19]

    Analyzing adversarial ro- bustness of vision transformers against spatial and spectral attacks.arXiv preprint arXiv:2208.09602, 2022

    Gihyun Kim and Jong-Seok Lee. Analyzing adversarial ro- bustness of vision transformers against spatial and spectral attacks.arXiv preprint arXiv:2208.09602, 2022. 2

    2022 arXiv

  13. [20]

    Curved representation space of vision transformers

    Juyeop Kim, Junha Park, Songkuk Kim, and Jong-Seok Lee. Curved representation space of vision transformers. InPro- ceedings of the AAAI Conference on Artificial Intelligence, pages 13142–13150, 2024. 3

    2024

  14. [21]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. InProceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 4015–4026, 2023. 2

    2023

  15. [23]

    Ad- versarial examples in the physical world

    Alexey Kurakin, Ian J Goodfellow, and Samy Bengio. Ad- versarial examples in the physical world. InArtificial in- telligence safety and security, pages 99–112. Chapman and Hall/CRC, 2018. 7

    2018

  16. [24]

    Align before fuse: Vision and language representation learn- ing with momentum distillation.Advances in neural infor- mation processing systems, 34:9694–9705, 2021

    Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learn- ing with momentum distillation.Advances in neural infor- mation processing systems, 34:9694–9705, 2021. 3

    2021

  17. [25]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InIn- ternational conference on machine learning, pages 19730– 19742. PMLR, 2023. 1, 6

    2023

  18. [26]

    Lawrence Zitnick, and Piotr Doll ´ar

    Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Doll ´ar. Microsoft coco: Common objects in context, 2015. 5

    2015

  19. [27]

    Exploring the rela- tionship between architecture and adversarially robust gen- eralization.arXiv preprint arXiv:2209.14105, 2022

    Aishan Liu, Shiyu Tang, Siyuan Liang, Ruihao Gong, Boxi Wu, Xianglong Liu, and Dacheng Tao. Exploring the rela- tionship between architecture and adversarially robust gen- eralization.arXiv preprint arXiv:2209.14105, 2022. 3

    2022 arXiv

  20. [28]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017. 4

    2017 arXiv

  21. [29]

    Set-level guidance at- tack: Boosting adversarial transferability of vision-language pre-training models

    Dong Lu, Zhiqiang Wang, Teng Wang, Weili Guan, Hongchang Gao, and Feng Zheng. Set-level guidance at- tack: Boosting adversarial transferability of vision-language pre-training models. InProceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 102–111,

  22. [30]

    Towards deep learn- ing models resistant to adversarial attacks.arXiv preprint arXiv:1706.06083, 2017

    Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learn- ing models resistant to adversarial attacks.arXiv preprint arXiv:1706.06083, 2017. 3, 5

    2017 arXiv

  23. [31]

    On the robustness of vision transformers to adversarial ex- amples

    Kaleel Mahmood, Rigel Mahmood, and Marten Van Dijk. On the robustness of vision transformers to adversarial ex- amples. InProceedings of the IEEE/CVF international con- ference on computer vision, pages 7838–7847, 2021. 3

    2021

  24. [32]

    Deepfool: a simple and accurate method to fool deep neural networks

    Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: a simple and accurate method to fool deep neural networks. InProceedings of the IEEE con- ference on computer vision and pattern recognition, pages 2574–2582, 2016. 7

    2016

  25. [33]

    On improving adversarial transferability of vision transformers.arXiv preprint arXiv:2106.04169, 2021

    Muzammal Naseer, Kanchana Ranasinghe, Salman Khan, Fahad Shahbaz Khan, and Fatih Porikli. On improving adversarial transferability of vision transformers.arXiv preprint arXiv:2106.04169, 2021. 2

    2021 arXiv

  26. [34]

    Reading isn’t believing: Adversarial attacks on multi-modal neurons

    David A Noever and Samantha E Miller Noever. Reading isn’t believing: Adversarial attacks on multi-modal neurons. arXiv preprint arXiv:2103.10480, 2021. 3

    2021 arXiv

  27. [35]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 1, 2

    2023 arXiv

  28. [36]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, p...

    2021

  29. [37]

    Berg, and Li Fei-Fei

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San- jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Chal- lenge.International Journal of Computer Vision (...

    2015

  30. [38]

    On the adversarial robustness of vision trans- formers.arXiv preprint arXiv:2103.15670, 2021

    Rulin Shao, Zhouxing Shi, Jinfeng Yi, Pin-Yu Chen, and Cho-Jui Hsieh. On the adversarial robustness of vision trans- formers.arXiv preprint arXiv:2103.15670, 2021. 2

    2021 arXiv

  31. [39]

    Cnn features off-the-shelf: an astound- ing baseline for recognition

    Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. Cnn features off-the-shelf: an astound- ing baseline for recognition. InProceedings of the IEEE con- ference on computer vision and pattern recognition work- shops, pages 806–813, 2014. 1

    2014

  32. [40]

    Indoor segmentation and support inference from rgbd images

    Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. InComputer Vision–ECCV 2012: 12th Eu- ropean Conference on Computer Vision, Florence, Italy, Oc- tober 7-13, 2012, Proceedings, Part V 12, pages 746–760. ...

    2012

  33. [41]

    Adversarial risk and the dangers of eval- uating against weak attacks

    Jonathan Uesato, Brendan O’donoghue, Pushmeet Kohli, and Aaron Oord. Adversarial risk and the dangers of eval- uating against weak attacks. InInternational conference on machine learning, pages 5025–5034. PMLR, 2018. 7

    2018

  34. [42]

    Benchmarking zero-shot robustness of multimodal founda- tion models: A pilot study.arXiv preprint arXiv:2403.10499,

    Chenguang Wang, Ruoxi Jia, Xin Liu, and Dawn Song. Benchmarking zero-shot robustness of multimodal founda- tion models: A pilot study.arXiv preprint arXiv:2403.10499,

    arXiv

  35. [43]

    Exploring transferability of multimodal adversarial sam- ples for vision-language pre-training models with contrastive learning.arXiv preprint arXiv:2308.12636, 2023

    Youze Wang, Wenbo Hu, Yinpeng Dong, and Richang Hong. Exploring transferability of multimodal adversarial sam- ples for vision-language pre-training models with contrastive learning.arXiv preprint arXiv:2308.12636, 2023. 2, 3

    2023 arXiv

  36. [44]

    Towards transferable adversarial attacks on vision transformers

    Zhipeng Wei, Jingjing Chen, Micah Goldblum, Zuxuan Wu, Tom Goldstein, and Yu-Gang Jiang. Towards transferable adversarial attacks on vision transformers. InProceedings of the AAAI Conference on Artificial Intelligence, pages 2668– 2676, 2022. 2

    2022

  37. [45]

    Vision-language pre-training with triple contrastive learning

    Jinyu Yang, Jiali Duan, Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng, Trishul Chilimbi, and Junzhou Huang. Vision-language pre-training with triple contrastive learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 156...

    2022

  38. [46]

    Peter Young, Alice Lai, Micah Hodosh, and Julia Hocken- maier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descrip- tions.Transactions of the Association for Computational Linguistics, 2:67–78, 2014. 5

    2014

  39. [47]

    Florence: A new foundation model for computer vision.arXiv preprint arXiv:2111.11432, 2021

    Lu Yuan, Qibin Hou, Zihang Jiang, Zhe Feng, Mingfei Cheng, Abner Wan, Jiadong Xie, Varun Kumar, Hongyu Shi, Dongdong Yu, et al. Florence: A new foundation model for computer vision.arXiv preprint arXiv:2111.11432, 2021. 2

    2021 arXiv

  40. [48]

    Towards adversarial attack on vision-language pre-training models

    Jiaming Zhang, Qi Yi, and Jitao Sang. Towards adversarial attack on vision-language pre-training models. InProceed- ings of the 30th ACM International Conference on Multime- dia, pages 5005–5013, 2022. 3, 5, 6, 7, 8

    2022

  41. [49]

    Transferable adversarial attacks on vision transform- ers with token gradient regularization

    Jianping Zhang, Yizhan Huang, Weibin Wu, and Michael R Lyu. Transferable adversarial attacks on vision transform- ers with token gradient regularization. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16415–16424, 2023. 2

    2023

  42. [50]

    Univer- sal adversarial perturbations for vision-language pre-trained models

    Peng-Fei Zhang, Zi Huang, and Guangdong Bai. Univer- sal adversarial perturbations for vision-language pre-trained models. InProceedings of the 47th International ACM SI- GIR Conference on Research and Development in Informa- tion Retrieval, pages 862–871, 2024. 6

    2024

  43. [51]

    Semantic under- standing of scenes through the ade20k dataset.International Journal of Computer Vision, 127(3):302–321, 2019

    Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fi- dler, Adela Barriuso, and Antonio Torralba. Semantic under- standing of scenes through the ade20k dataset.International Journal of Computer Vision, 127(3):302–321, 2019. 7

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.