Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustness

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that adversarially training vision-language models at both the contrastive pre-training stage and the visual instruction-tuning stage produces state-of-the-art robustness against visual adversarial attacks with little…

desk verdict A serious attempt at robust VLMs with a plausible two-stage recipe, but the headline SOTA margin is a confounded cross-paper comparison; worth peer review, not desk rejection. read the letter →

arxiv 2501.09446 v2 pith:GJSDYKC3 submitted 2025-01-16 cs.CV

Zeyu Wang , Cihang Xie , Brian Bartoldson , Bhavya Kailkhura This is my paper · ORCID
classification cs.CV
keywords adversarialtrainingvision-languagemodelsCLIPLLaVAzero-shotrobustnessvisualinstructiontuninghallucinationweb-scalepre-training
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vision-language models inherit a serious vulnerability: small, invisible perturbations to an input image can make a CLIP or LLaVA model misclassify, hallucinate, or emit attacker-chosen text. Previous defenses tried to fix this by lightly fine-tuning an already-trained CLIP on ImageNet, which restored some robustness but degraded clean accuracy and generalization. This paper argues for a different strategy: build robustness into the model at both stages of training. It adversarially pre-trains a new CLIP, called ∆CLIP, from scratch on web-scale data, then adversarially tunes a LLaVA on top of it, producing ∆2LLaVA. Across zero-shot recognition, captioning, VQA, targeted attacks, reasoning, and hallucination benchmarks, the resulting models are claimed to set a new state of the art in adversarial defense while keeping clean-data performance close to that of the non-robust originals.

What carries the argument

The machinery is the two-stage 'double visual defense'. Stage one is adversarial contrastive pre-training: $\min_{\theta_I}\max_{\|\delta\|_p\leq\epsilon} \mathcal{L}_{con}(x+\delta,y)$ run at increasing resolution and attack radius (112px with PGD-2 and $\epsilon=4/255$, then 224px with PGD-3, then 336px with PGD-4 and $\epsilon=8/255$) on DataComp-1B with a frozen text encoder and an added CoCa captioning loss. Stage two is adversarial visual instruction tuning: $\min_{\phi}\max_{\|\delta\|_p\leq\epsilon} \mathcal{L}_{inst}(x+\delta,y)$ on the LLaVA-1.5 recipe with LoRA, where the visual encoder is fine-tuned at one-twentieth of the base learning rate. The argument is that the first stage prevents the catastrophic clean-performance loss seen in post-hoc ImageNet fine-tuning, and the second stage adds strength specifically under large perturbations.

What would settle it

Train the identical H/14-336 architecture and DataComp-1B recipe with the same compute, but with the adversarial perturbations removed, or with only one of the two stages made adversarial, and evaluate on the exact fixed 1000-sample subsets used in [42]; if the robustness margin over the earlier models mostly disappears, the attribution of the gain to the double visual defense fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that adversarial training should happen at every stage of vision-language model creation, not only as a post-hoc patch. The authors replace CLIP's contrastive objective with a min-max objective in which images are perturbed to break image-text matching, and train this adversarially pre-trained model, ∆CLIP, from scratch on web-scale data. They then add a second layer of defense by adversarially tuning LLaVA with perturbed images optimized to cause next-token mispredictions, yielding ∆2LLaVA. The claimed result is that ∆CLIP and ∆2LLaVA exceed the prior robust VLM baselines by roughly 20% on ImageNet-1k zero-shot robustness, 30% on captioning robustness, and 20% on VQA robustness, while matching vanilla CLIP and LLaVA on clean inputs and hallucinating less than prior robust models.

Load-bearing premise

The load-bearing assumption is that the large robustness gains come from the two adversarial training stages rather than from the larger vision tower and much larger pre-training dataset, and that the paper's unshared 1000-sample evaluation subsets, which differ from those used for the prior results it compares against, are representative enough for direct state-of-the-art comparisons.

Editorial extensions

If this is right

  • Adversarial visual hijacking becomes far harder: at $\epsilon=16/255$, the average targeted-attack success rate for ∆2LLaVA8 is 3.3%, versus 65.0% for the FARE4-based LLaVA and 100% for a vanilla LLaVA.
  • Robustness transfers across zero-shot domains: ∆CLIP improves over prior robust CLIP models on both ImageNet-variant datasets and non-ImageNet datasets such as Stanford Cars, where its adversarial accuracy is roughly seven times the second-best model.
  • The second stage earns its cost mainly at large attack radii: on VQAv2 at $\epsilon=8/255$, ∆2LLaVA8 raises accuracy to 52.4% from 41.1% for ∆LLaVA.
  • Because clean performance and helpfulness stay close to vanilla LLaVA, the resulting models could serve as drop-in replacements for unprotected CLIP and LLaVA in applications that need both safety and utility.
  • The paper's own limitation statement scopes the result to visual attacks on CLIP-based architectures; text-side threats and non-CLIP VLMs are not covered by the double visual defense.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: because the new model uses a larger vision tower (H/14-336) and a far larger pre-training corpus (DataComp-1B) than the L/14 WIT-400M-plus-ImageNet models it is compared with, the share of the robustness gain due specifically to double-defense training, rather than to scale and data, is not isolated by an ablation.
  • Beyond the paper: the paper's own observation that typographic text attacks 'emerge' from naive $\ell_\infty$ attacks against its robust models suggests a testable extension: adversarially training on typographic prompts could close the remaining attack surface and may generalize to unseen text-in-image attacks.
  • Beyond the paper: if these gains replicate with a controlled comparison, adversarial pre-training from scratch could become the default route to robust vision-language models, and the community could compare it against lightweight fine-tuning on shared evaluation subsets.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a two-stage adversarial training recipe for vision-language models: (i) adversarial contrastive pre-training of CLIP on DataComp-1B with Recap synthetic captions and a CoCa captioning loss, yielding DeltaCLIP, and (ii) adversarial visual instruction tuning of LLaVA-1.5, yielding Delta2LLaVA. The authors evaluate clean and adversarial zero-shot classification on standard and distribution-shift benchmarks, captioning and VQA robustness, targeted attacks, visual reasoning, and hallucination. They report large robustness gains over the TeCoA and FARE baselines and argue that, unlike prior post-hoc adversarial fine-tuning, their models preserve clean performance close to non-robust baselines. The central claim is that this 'double visual defense' sets a new state of the art in VLM adversarial robustness.

Significance. If the headline results survive a controlled comparison, the paper is a substantive empirical contribution: it introduces adversarial visual instruction tuning as a defense stage, evaluates with AutoAttack and a broad set of benchmarks, and provides an internal comparison showing that the second training stage adds robustness, especially at larger attack radii. The paper also documents an interesting phenomenon in which typographic attacks emerge from plain L-infinity attacks on adversarially trained models. However, the claimed state-of-the-art margins are currently based on confounded cross-paper comparisons on non-shared random evaluation subsets, so the significance is conditional on the authors supplying a same-setting ablation and a reproducible evaluation protocol.

major comments (3)
  1. [Section 4.1, Tables 1-2] The headline ~20% ImageNet robustness improvement over TeCoA and FARE is not attributable to the proposed training method from these tables alone: DeltaCLIP-H/14-336 is trained on DataComp-1B with Recap synthetic captions and an additional CoCa captioning loss, whereas TeCoA and FARE are L/14 models fine-tuned from OpenAI CLIP on WIT-400M plus ImageNet at 224 resolution. Architecture, training data, resolution, and auxiliary losses vary simultaneously. A same-setting ablation, such as a non-adversarial CLIP trained on the same DataComp-1B recipe or an adversarially trained L/14 with the prior setup, is needed to support the attribution.
  2. [Sections 4.1 and 4.2] The paper states that its random 1000-sample (CLIP) and 500-sample (LLaVA) adversarial evaluation subsets differ from those used by Schlarmann et al. and that the results are 'not directly comparable' to prior work. Because the abstract and conclusion present ~20% and ~30% improvements as state of the art, the authors should either re-evaluate prior baselines under the identical protocol, release the exact random subsets and seeds, or substantially soften the SOTA claims. Confidence intervals would also help, since some reported numbers are based on only 1000 samples.
  3. [Section 3.3] The paper claims that adversarial visual instruction tuning on a vanilla CLIP-based LLaVA 'completely crashed,' but it provides no quantitative detail or experimental description. Since the necessity of the double defense is a load-bearing claim, please report the failure mode and the relevant numbers, or clearly mark this as a preliminary observation that was not systematically evaluated.
minor comments (5)
  1. [Table 2 caption] The caption reads 'TeCoA and FAR' but should read 'TeCoA and FARE'.
  2. [Training Details] The text cites reference [5] for both JAX and PyTorch implementations, but [5] is the JAX reference; a separate PyTorch reference should be cited for the LLaVA implementation.
  3. [Section 4.4] The ScienceQA setup description contains a duplicated word: 'we use the the subset with images'.
  4. [Limitations] The Limitations section begins with 'In this paper, We focus solely...' and should be reworded; it also does not mention the lack of public code or model weights at the time of submission, which is relevant for reproducibility.
  5. [Section 4.3 and Table 4] The targeted-attack evaluation relies on 10 random COCO samples per target and human judgement of attack success; the paper should specify the human judgement protocol and whether multiple annotators were used, because Table 6 reports CIDEr/ASR combinations that are not self-explanatory.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; double-defense claims rest on external benchmark evaluations, with comparability caveats rather than constructed predictions.

full rationale

No circular step is present. The paper's load-bearing claims are empirical results of a training recipe: Eq. (3) is the standard adversarial CLIP objective and Eq. (5) is adversarial autoregressive instruction tuning; both are evaluated on held-out external benchmarks (ImageNet variants, COCO, Flickr30k, VQAv2, TextVQA, POPE, MME, etc.), so no quantity is fitted and then relabeled as a prediction. The self-citations used in the setup—Recap-DataComp-1B [21], efficient CLIP training [22], and adversarial training at scale [49]—are data and recipe building blocks with independent standing, not arguments that the double-defense result holds; thus they are not load-bearing circular references. The paper even flags the main evaluation caveat in Sections 4.1 and 4.2, stating that its randomly selected adversarial subsets differ from Schlarmann et al. and that previous results are 'not directly comparable'; that concern is about comparability and attribution of the ~20% SOTA gain, not about a derivational circle. Accordingly, the correct circularity finding is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central result is empirical and rests on a small set of hand-chosen hyperparameters, mainly attack radii, PGD steps, data budgets, and learning-rate ratios. No new entities are introduced. The key axiomatic load is that L-infinity PGD/APGD attacks define the relevant threat model and that DataComp-1B is an appropriate pretraining distribution.

free parameters (4)
  • training perturbation radius epsilon = 4/255 for CLIP stages 1 to 2 and LLaVA PGD-3; 8/255 for CLIP stage 3 and LLaVA PGD-5
    Chosen by hand; defines the threat model and is central to all robustness results. Appears in Sections 3.2, 3.3, and Training Details.
  • PGD steps per training stage = CLIP: PGD-2, PGD-3, PGD-4; LLaVA: PGD-3 and PGD-5
    Attack strength during adversarial training; directly affects the robustness-clean performance tradeoff. Described in Training Details.
  • stage-wise training data budgets = 5.12B, 512M, and 128M samples across the three CLIP stages
    Amount of web-scale data per stage, selected following prior efficient CLIP practice; influences both clean and robust performance. Described in Training Details.
  • vision encoder learning-rate scale = 1/20 of the base learning rate
    The vision encoder is unfrozen during adversarial instruction tuning at a reduced learning rate; this ratio is chosen by hand and affects the final model. Described in Training Details.
assumptions (4)
  • domain assumption L-infinity bounded perturbations are the relevant threat model for the visual channel.
    All training objectives (Eqs. 3 and 5) and all robustness evaluations use L-infinity epsilons. The Limitations section explicitly leaves other threat models for future work.
  • standard math Inner maximization by PGD during training and by APGD/AutoAttack during evaluation approximates the worst-case perturbation.
    Adversarial training in Section 3.1 relies on approximate inner maximization; the paper does not prove that the inner optimization is solved exactly.
  • domain assumption DataComp-1B with Recap-DataComp captions is a suitable data distribution for robust VLM pre-training.
    The paper provides no controlled comparison against WIT-400M or another data source, so the data choice is a premise of the reported results.
  • domain assumption Keeping the text encoder frozen during adversarial visual pre-training preserves image-text alignment.
    Training Details states that the text model is pre-trained with clean data and kept frozen during adversarial training; if this assumption fails, clean zero-shot performance could degrade.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustness." pith.science (2026). https://pith.science/paper/GJSDYKC3

@misc{pith2026250109446,
  author       = {Pith},
  title        = {Pith review of: Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustness},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GJSDYKC3}},
  note         = {Machine review of arXiv:2501.09446}
}
abstract

This paper investigates the robustness of vision-language models against adversarial visual perturbations and introduces a novel ``double visual defense" to enhance this robustness. Unlike previous approaches that resort to lightweight adversarial fine-tuning of a pre-trained CLIP model, we perform large-scale adversarial vision-language pre-training from scratch using web-scale data. We then strengthen the defense by incorporating adversarial visual instruction tuning. The resulting models from each stage, $\Delta$CLIP and $\Delta^2$LLaVA, show substantially enhanced zero-shot robustness and set a new state-of-the-art in adversarial defense for vision-language models. For example, the adversarial robustness of $\Delta$CLIP surpasses that of the previous best models on ImageNet-1k by ~20%. %For example, $\Delta$CLIP surpasses the previous best models on ImageNet-1k by ~20% in terms of adversarial robustness. Similarly, compared to prior art, $\Delta^2$LLaVA brings a ~30% robustness improvement to image captioning task and a ~20% robustness improvement to visual question answering task. Furthermore, our models exhibit stronger zero-shot recognition capability, fewer hallucinations, and superior reasoning performance compared to baselines. Our project page is https://doublevisualdefense.github.io/.

Figures

Figures reproduced from arXiv: 2501.09446 by the authors.

Figure 1
Figure 1. (a) Our Double Visual Defense framework, which involves an adversarial contrastive pre-training stage and an adversarial visual [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Output from various models under targeted attacks from Table [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Visualization of adversarial samples generated with different target models and attack radii. Note that typographic attacks “emerge” from naive ℓ∞-adversarial attacks when applied to the proposed robust models, especially with larger attack radii. FARE4 -LLaVA No TeCoA4 -LLaVA No Q: Is there a traffic light in the image? GT-Answer: Yes ∆ 2LLaVA8 Yes Q: Is there a bench in the image? Q: Is there a person in the image… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Visual examples from the POPE hallucination benchmark. GT-Answer is the ground truth response to the question, the red [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Grounding-Driven Attack: Improving Encoder-based Adversarial Transferability against Large Vision-Language Models

    cs.CR 2026-02 conditional novelty 6.0 of 10

    A grounding-guided attack that concentrates perturbation on text-matched image regions and disrupts global and local semantic alignment consistently improves adversarial transferability across multiple vision-language models.

Reference graph

Works this paper leans on

58 extracted references · 40 canonical work pages · cited by 1 Pith paper

  1. [1]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision , pages 2425– 2433, 2015. 6

    work page 2015

  2. [2]

    Obfus- cated gradients give a false sense of security: Circumventing defenses to adversarial examples

    Anish Athalye, Nicholas Carlini, and David Wagner. Obfus- cated gradients give a false sense of security: Circumventing defenses to adversarial examples. In International confer- ence on machine learning, pages 274–283. PMLR, 2018. 3

    work page 2018

  3. [3]

    Image hijacking: Adversarial images can control generative models at runtime

    Luke Bailey, Euan Ong, Stuart Russell, and Scott Emmons. Image hijacking: Adversarial images can control generative models at runtime. arXiv e-prints, pages arXiv–2309, 2023. 1, 3, 4, 5

    work page 2023

  4. [4]

    Adversarial robustness limits via scaling-law and human-alignment studies

    Brian R Bartoldson, James Diffenderfer, Konstantinos Parasyris, and Bhavya Kailkhura. Adversarial robustness limits via scaling-law and human-alignment studies. arXiv preprint arXiv:2404.09349, 2024. 3, 9

    arXiv 2024

  5. [5]

    Jax: composable transformations of python+ numpy programs

    James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclau- rin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, et al. Jax: composable transformations of python+ numpy programs. 2018. 6

    work page 2018

  6. [6]

    Towards evaluating the robustness of neural networks

    Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp), pages 39–57. Ieee, 2017. 1, 3

    work page 2017

  7. [7]

    Are aligned neural networks adversarially aligned? Advances in Neural Information Processing Systems, 36, 2024

    Nicholas Carlini, Milad Nasr, Christopher A Choquette- Choo, Matthew Jagielski, Irena Gao, Pang Wei W Koh, Daphne Ippolito, Florian Tramer, and Ludwig Schmidt. Are aligned neural networks adversarially aligned? Advances in Neural Information Processing Systems, 36, 2024. 1, 3, 4

    work page 2024

  8. [8]

    Reliable evalua- tion of adversarial robustness with an ensemble of diverse parameter-free attacks

    Francesco Croce and Matthias Hein. Reliable evalua- tion of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on ma- chine learning, pages 2206–2216. PMLR, 2020. 6

    work page 2020

Show all 58 references
  1. [9]

    InstructBLIP: Towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. In Thirty- seventh Conference on Neural Information Processing Sys- tems...

    2023

  2. [10]

    Robust physical-world attacks on deep learning visual classification

    Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust physical-world attacks on deep learning visual classification. In 2018 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pag...

    2018

  3. [11]

    Mme: A comprehensive evaluation bench- mark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation bench- mark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023. 8

    2023 arXiv

  4. [12]

    Dat- acomp: In search of the next generation of multimodal datasets

    Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Dat- acomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Sys...

    2024

  5. [13]

    Figstep: Jailbreaking large vision-language models via typo- graphic visual prompts

    Yichen Gong, Delong Ran, Jinyuan Liu, Conglei Wang, Tianshuo Cong, Anyu Wang, Sisi Duan, and Xiaoyun Wang. Figstep: Jailbreaking large vision-language models via typo- graphic visual prompts. arXiv preprint arXiv:2311.05608 ,

    arXiv

  6. [14]

    Explaining and harnessing adversarial examples

    Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR,

  7. [15]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913...

    2017

  8. [16]

    Agent smith: A single image can jailbreak one million multimodal llm agents exponentially fast

    Xiangming Gu, Xiaosen Zheng, Tianyu Pang, Chao Du, Qian Liu, Ye Wang, Jing Jiang, and Min Lin. Agent smith: A single image can jailbreak one million multimodal llm agents exponentially fast. arXiv preprint arXiv:2402.08567, 2024. 3

    2024 arXiv

  9. [17]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608–3617,

  10. [18]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. InIn- ternational Conference on Learning Representations , 2022. 5

    2022

  11. [19]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 6700–6709, 2019. 8

    2019

  12. [20]

    Collecting a large-scale dataset of fine-grained cars.(2013)

    Jonathan Krause, Jia Deng, Michael Stark, and Li Fei-Fei. Collecting a large-scale dataset of fine-grained cars.(2013). URL https://api. semanticscholar. org/CorpusID, 16632981,

    2013

  13. [21]

    What if we recaption billions of web images with llama-3? arXiv preprint arXiv:2406.08478,

    Xianhang Li, Haoqin Tu, Mude Hui, Zeyu Wang, Bingchen Zhao, Junfei Xiao, Sucheng Ren, Jieru Mei, Qing Liu, Huangjie Zheng, et al. What if we recaption billions of web images with llama-3? arXiv preprint arXiv:2406.08478,

    arXiv

  14. [22]

    An inverse scal- ing law for clip training

    Xianhang Li, Zeyu Wang, and Cihang Xie. An inverse scal- ing law for clip training. Advances in Neural Information Processing Systems, 36, 2024. 3, 5

    2024

  15. [23]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In The 2023 Conference on Empir- ical Methods in Natural Language Processing, 2023. 8

    2023

  16. [24]

    Scaling language-image pre-training via masking

    Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichten- hofer, and Kaiming He. Scaling language-image pre-training via masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23390– 23400, 2023. 3, 6 13

    2023

  17. [25]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceeding...

    2014

  18. [26]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 1, 5

    2024

  19. [27]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 1, 3, 4

    2024

  20. [28]

    AutoDAN: Generating stealthy jailbreak prompts on aligned large language models

    Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. AutoDAN: Generating stealthy jailbreak prompts on aligned large language models. In The Twelfth International Confer- ence on Learning Representations, 2024. 1

    2024

  21. [29]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems , 35:2507–2521,

  22. [30]

    Towards deep learning models resistant to adversarial attacks

    Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018. 1, 3

    2018

  23. [31]

    Understanding zero-shot adversarial robust- ness for large-scale models

    Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, and Carl V ondrick. Understanding zero-shot adversarial robust- ness for large-scale models. In The Eleventh International Conference on Learning Representations, 2023. 1, 2, 3, 4, 5, 6

    2023

  24. [32]

    Deepfool: a simple and accurate method to fool deep neural networks

    Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: a simple and accurate method to fool deep neural networks. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 2574–2582, 2016. 3

    2016

  25. [33]

    Bag of tricks for adversarial training

    Tianyu Pang, Xiao Yang, Yinpeng Dong, Hang Su, and Jun Zhu. Bag of tricks for adversarial training. In International Conference on Learning Representations, 2021. 1

    2021

  26. [34]

    Learning to see but forgetting to fol- low: Visual instruction tuning makes llms more prone to jail- break attacks

    Georgios Pantazopoulos, Amit Parekh, Malvina Nikandrou, and Alessandro Suglia. Learning to see but forgetting to fol- low: Visual instruction tuning makes llms more prone to jail- break attacks. arXiv preprint arXiv:2405.04403, 2024. 4

    2024 arXiv

  27. [35]

    Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models

    Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazeb- nik. Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models. In Pro- ceedings of the IEEE international conference on computer ...

    2015

  28. [36]

    Visual adversarial exam- ples jailbreak large language models

    Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Mengdi Wang, and Prateek Mittal. Visual adversarial exam- ples jailbreak large language models. arXiv preprint arXiv:2306.13213, 2023. 1, 3, 4

    2023 arXiv

  29. [37]

    Visual adversarial examples jailbreak aligned large language models

    Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Peter Hen- derson, Mengdi Wang, and Prateek Mittal. Visual adversarial examples jailbreak aligned large language models. In Pro- ceedings of the AAAI Conference on Artificial Intelligence , pages 21527–21536, 2024. 1

    2024

  30. [38]

    Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference on Learning Representations, 2024

    Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference on Learning Representations, 2024. 4

    2024

  31. [39]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, 2021. 1, 3, 6

    2021

  32. [40]

    When do universal image jailbreaks transfer between vision-language models? arXiv preprint arXiv:2407.15211, 2024

    Rylan Schaeffer, Dan Valentine, Luke Bailey, James Chua, Crist´obal Eyzaguirre, Zane Durante, Joe Benton, Brando Mi- randa, Henry Sleight, John Hughes, et al. When do universal image jailbreaks transfer between vision-language models? arXiv preprint arXiv:2407.15211, 2024. 3, 4

    2024 arXiv

  33. [41]

    on the adversar- ial robustness of multi-modal foundation models

    Christian Schlarmann and Matthias Hein. on the adversar- ial robustness of multi-modal foundation models. In 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), 2023. 3, 4

    2023

  34. [42]

    Robust clip: Unsupervised ad- versarial fine-tuning of vision embeddings for robust large vision-language models

    Christian Schlarmann, Naman Deep Singh, Francesco Croce, and Matthias Hein. Robust clip: Unsupervised ad- versarial fine-tuning of vision embeddings for robust large vision-language models. arXiv preprint arXiv:2402.12336,

    arXiv

  35. [43]

    Adversarial training for free! NeurIPS, 32, 2019

    Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! NeurIPS, 32, 2019. 1, 3

    2019

  36. [44]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. 6, 8

    2019

  37. [45]

    Eva-clip: Improved training techniques for clip at scale

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389, 2023. 3

    2023 arXiv

  38. [46]

    In- triguing properties of neural networks

    Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. In- triguing properties of neural networks. In ICLR, 2014. 3

    2014

  39. [47]

    Robustness may be at odds with accuracy

    Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. arXiv preprint arXiv:1805.12152,

    arXiv

  40. [48]

    Cider: Consensus-based image description evalua- tion

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evalua- tion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015. 6

    2015

  41. [49]

    Revisiting adversarial training at scale

    Zeyu Wang, Xianhang Li, Hongru Zhu, and Cihang Xie. Revisiting adversarial training at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24675–24685, 2024. 1, 3, 5

    2024

  42. [50]

    Zico Kolter

    Eric Wong, Leslie Rice, and J. Zico Kolter. Fast is better than free: Revisiting adversarial training. In ICLR, 2020. 3

    2020

  43. [51]

    On the safety concerns of deploying 14 llms/vlms in robotics: Highlighting the risks and vulnerabil- ities

    Xiyang Wu, Ruiqi Xian, Tianrui Guan, Jing Liang, Souradip Chakraborty, Fuxiao Liu, Brian M Sadler, Dinesh Manocha, and Amrit Bedi. On the safety concerns of deploying 14 llms/vlms in robotics: Highlighting the risks and vulnerabil- ities. In First Vision and Language for Auton...

    2024

  44. [52]

    Feature denoising for improving ad- versarial robustness

    Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan L Yuille, and Kaiming He. Feature denoising for improving ad- versarial robustness. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 501–509, 2019. 3

    2019

  45. [53]

    Coca: Contrastive captioners are image-text foundation models

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mo- jtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research, 2022. 3, 5

    2022

  46. [54]

    Towards robust resnet: A small step but a giant leap

    Jingfeng Zhang, Bo Han, Laura Wynter, Bryan Kian Hsiang Low, and Mohan Kankanhalli. Towards robust resnet: A small step but a giant leap. In Proceedings of the Twenty- Eighth International Joint Conference on Artificial Intelli- gence, IJCAI-19, 2019. 3

    2019

  47. [55]

    Attacks which do not kill training make adversarial learning stronger

    Jingfeng Zhang, Xilie Xu, Bo Han, Gang Niu, Lizhen Cui, Masashi Sugiyama, and Mohan Kankanhalli. Attacks which do not kill training make adversarial learning stronger. In International conference on machine learning, pages 11278– 11287. PMLR, 2020. 1

    2020

  48. [56]

    Geometry-aware instance-reweighted adversarial training

    Jingfeng Zhang, Jianing Zhu, Gang Niu, Bo Han, Masashi Sugiyama, and Mohan Kankanhalli. Geometry-aware instance-reweighted adversarial training. In International Conference on Learning Representations, 2021. 3

    2021

  49. [57]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 1, 3, 4

    2023 arXiv

  50. [58]

    Universal and transferable ad- versarial attacks on aligned language models

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable ad- versarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023. 1, 3 15

    2023 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.