Pith. sign in

REVIEW 3 major objections 7 minor 1 cited by

On the robustness of multimodal language model towards distractions

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Most vision-language models, including GPT-4o, lose accuracy when faced with irrelevant distractions in images or text.

desk verdict Useful new distraction benchmark for VLMs, but the missing appendix and unreleased data make the central evaluation unverifiable; the qualitative finding is plausible, the model ranking is not. read the letter →

arxiv 2502.09818 v1 pith:R3LBS46A submitted 2025-02-13 cs.CV

Ming Liu , Hao Chen , Jindong Wang , Wensheng Zhang This is my paper · ORCID
classification cs.CV
keywords vision-languagemodelsdistractionrobustnessbenchmarkScienceQAvisualquestionansweringmultimodalevaluationpromptengineeringexactmatchdegradation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that current vision-language models (VLMs) are not resilient to irrelevant visual and textual information, a weakness that existing clean benchmarks fail to expose. It introduces I-ScienceQA, a benchmark built on ScienceQA that injects four types of distractions into images and text, and evaluates fourteen state-of-the-art VLMs. The results show widespread performance degradation, with textual distractions causing larger drops than visual ones. Larger models tend to resist better, but prompt engineering and robust vision encoders recover only part of the lost accuracy. If the findings hold, real-world deployment of VLMs should treat distraction robustness as a first-class evaluation criterion.

What carries the argument

The central object is the I-ScienceQA benchmark, containing 8,100 samples built from ScienceQA test sets and divided into four distraction scenarios: Add Image, Insert Image, Add Hint, and Insert Hint. Visual distractions are generated with Stable Diffusion and images from Unsplash and PromeAI, while textual distractions are produced with GPT-3.5-turbo. Evaluation relies on exact-match accuracy and exact-match degradation, defined as the difference in accuracy between the original and distracted sets. This machinery isolates the impact of each distraction modality on model reasoning.

What would settle it

Human annotators could verify all 8,100 distracted samples to confirm that the correct answer is unchanged; if even a small fraction of distractors alter the correct answer, the reported degradation values would need to be recomputed and would not cleanly reflect distraction robustness.

Watch

Extended reading notes

Core claim

The central discovery is that most state-of-the-art VLMs, including GPT-4o, show measurable exact-match degradation on science questions when irrelevant content is added to the image or the text. The largest drops occur in the 'Add Hints' scenario, where unrelated text replaces a missing hint: Qwen2-VL-2B falls by 6.35 points and InternVL2-2B by 9.05 points. Textual distractions are consistently more harmful than visual ones, and model scale helps but does not guarantee robustness, with InternVL2-8B showing the smallest degradation in most scenarios. The paper reports these findings through a new benchmark and a degradation metric that quantifies the effect of each distraction type.

Load-bearing premise

The entire degradation measure rests on the assumption that the added images and texts are irrelevant to the question and never change the correct answer, so any accuracy drop must stem from distraction rather than from a changed ground truth.

Editorial extensions

If this is right

  • VLM benchmarks should include distraction robustness as a standard evaluation axis, since clean benchmarks overstate real-world capability.
  • Textual distractions are a particular weak point, so future training should focus on ignoring irrelevant text more aggressively.
  • Model scaling reduces but does not eliminate distraction vulnerability; architecture and training data matter as much as size.
  • Simple defenses such as instruction prompts and robust vision encoders recover only part of the lost accuracy, leaving room for stronger mitigation strategies.
  • The exact-match degradation metric can be reused as a standard measure for distraction robustness in future evaluations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the benchmark only uses science questions, degradation may be even larger in tasks demanding tighter visual-text alignment, such as medical or technical image interpretation.
  • The paper's assumption that distractors are answer-neutral is not verifiable in the submitted text; if some distractors alter the correct answer, the reported degradation would overstate true vulnerability.
  • A testable extension is to fine-tune models on distraction-augmented samples and measure whether the lost accuracy is recovered, since the same models score high on clean data.
  • The 'Add Hints' scenario being the most harmful suggests these models over-weight textual priors; probing attention weights could test whether the models fixate on inserted text.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper presents I-ScienceQA, a robustness benchmark derived from the ScienceQA dataset, in which visual and textual distractions are injected into test samples across four scenarios (Add Image, Insert Image, Add Hint, Insert Hint). Fourteen vision-language models ranging from 1B to 34B parameters, plus GPT-4o, are evaluated with an Exact Match degradation metric. The main findings are that most VLMs degrade under distractions, textual distractions are more harmful than visual ones, InternVL2 (8B) shows the least degradation, larger models tend to be more robust, and prompt-engineering or robust-CLIP defenses provide only partial relief. The authors explicitly disclose in Section 6.2 that InternVL2 and LLaVA were trained on ScienceQA, but they nonetheless draw model-level robustness conclusions from the same benchmark.

Significance. If the experimental claims could be fully verified, I-ScienceQA would be a useful and reasonably large robustness benchmark with an easy-to-reproduce evaluation protocol, covering well-known open models and a proprietary model (GPT-4o), a clear performance metric, and a natural visual-versus-textual distraction taxonomy. Credit is due for including GPT-4o alongside open models, for openly discussing training-data contamination in Section 6.2, and for quantifying the limited effect of two defenses. However, two issues are currently load-bearing for the paper's claims: the benchmark is built on ScienceQA, which Section 6.2 admits was part of the training data for InternVL2 and LLaVA, so the headline robustness ranking may reflect memorization rather than distraction handling; and the validity of every degradation number depends on the assumption that no generated distraction changes the correct answer, an assumption whose validation is deferred to an appendix that is not present in the submitted text. The contribution is therefore significant only conditionally, pending a deconfounding analysis and release of the validation material.

major comments (3)
  1. [§6.2, Table 1] The paper's headline conclusion that "models such as InternVL2 demonstrate a higher degree of robustness" is drawn from Table 1, which is evaluated on a benchmark derived from ScienceQA, and Section 6.2 states that InternVL2 and LLaVA were trained on ScienceQA. The authors acknowledge that "their performance metrics might be artificially inflated" but do not act on this disclosure: the model-level ranking is still presented without decontamination. Because the central claim is that InternVL2 is robust, the comparison is currently confounded. The authors should either restrict the robustness ranking to models not trained on ScienceQA or provide additional evidence that contamination does not drive the InternVL2 advantage, for example by measuring degradation on a subset of I-ScienceQA built from non-ScienceQA sources or by comparing against a held-out ScienceQA split that was unseen by the trained models.
  2. [§3.1, §3.2, Figure 1] Every degradation figure in Table 1 is computed by comparing the distracted sample against the original ScienceQA label, so the benchmark's validity rests on the assumption that each generated visual or textual distraction is irrelevant to the question and never changes the correct answer. The definitions of the visual distraction subtypes, the GPT-3.5-generated textual distraction types, dataset statistics, and sample images are repeatedly deferred to "the Appendix" (Sections 3.1, 3.2, 6.2, Figure 1), but no appendix is present in the submitted text. A reader therefore cannot inspect a single distracted example to confirm label preservation. The manuscript must include the appendix material as well as explicit validation of the label-preservation property, such as human or automated checks with a reported label-change rate, before the magnitudes of the reported degradations can be interpreted.
  3. [§3.2, Scenario III (Add Hint)] The Add Hint scenario replaces the original "N/A" hint with GPT-3.5-generated content, so the "Original" condition contains no hint at all while the "Distraction" condition introduces a new textual clue. Any new hint changes the semantics of the input, and the paper states that "contradictory information" is one of the generated distraction types; a contradictory hint can conflict with the ground-truth answer, may change which answer is correct under the new context, or may help some models answer correctly for reasons unrelated to distraction handling. The reported degradation for this scenario therefore conflates genuine robustness with input-semantics changes. The authors should clarify how the contradictory category is constrained so that the original label remains correct, and should report accuracy on a subset of Add Hint samples whose label preservation has been explicitly verified.
minor comments (7)
  1. [Abstract and §4.1] The abstract states that the evaluation covers "ten state-of-the-art VLMs, including GPT-4o", while Section 4.1 says 14 models are employed and Table 1 lists 14 model rows; these counts should be aligned.
  2. [§3.1, §3.2] The dataset statistics state 8,100 samples total with 4,000 text-based and 4,100 image-based distractions, whereas Section 3.2 describes four scenarios of 2,000 randomly selected samples each (8,000 in total); the source of the extra 100 samples, and their relationship to the diffusion-inpainted subset of Scenario II, should be clarified so that samples are not double-counted.
  3. [§6.4, Table 4] The bi-modal results in Table 4 (e.g., Qwen2-VL-2B, Add Hint, "No Img" 57.68) differ from the corresponding Add Hints values in Table 1 (54.45), and the selection of the subset used for Table 4 is not described; this discrepancy should be explained.
  4. [§4.2, Eq. (2)] The claim that ΔAccuracy(F) is "always less than or equal to zero" is presented as a mathematical fact, but the definition does not guarantee it: a distraction could in principle move a model from a wrong to a correct answer. The non-positive values in Table 1 should be reported as an empirical observation rather than a necessity.
  5. [§5] The section contains grammatical slips such as "a exact match scores" (twice) and should be copy-edited; similar issues appear in Section 6.1 ("theAdd Hint") and Section 2 ("Mecently", "papr").
  6. [Table 3] The table caption reads "Exact Math Scores" and should be "Exact Match Scores".
  7. [Abstract, §1, §3.1, reference [25]] There are naming inconsistencies between the abstract ("GPT-4") and the main text ("GPT-4o"), and between "ProMeAI" in Section 3.1 and "PromeAI" in reference [25]; these should be made consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No construction-level circularity; the paper's benchmark-contamination and missing-appendix concerns are validity gaps, not circular reductions.

full rationale

The paper does not contain a derivation in which a predicted quantity is algebraically equal to a fitted input or in which a central claim is forced by a self-citation. I-ScienceQA is an empirical augmentation of ScienceQA, and every degradation number in Table 1 is a direct accuracy difference between two evaluations, not the output of a model fitted to those numbers. Section 6.2 explicitly flags that InternVL2 and LLaVA were trained on ScienceQA and that this 'may' inflate performance metrics; that is a data-contamination caveat about external validity, not a circularity, because robustness is defined as an accuracy change rather than as inclusion in a training set. The missing appendix and the unverified assumption that GPT-3.5 and stable-diffusion distractions never change the correct answer (Sections 3.1 and 3.2, Figure 1) are serious verification gaps, but they concern label integrity and benchmark validity, not a self-referential reduction. There are no load-bearing self-citations, no imported uniqueness theorems, no ansatz smuggled in by citation, and no fitted parameter renamed as a prediction. The central empirical finding may be confounded by ScienceQA training overlap, but the paper itself acknowledges this limitation, and the claimed derivation chain is not circular by construction.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The work is empirical, so there are no fitted parameters or invented physical entities. The key assumptions are the invariance of ground-truth answers under distraction, the cleanliness of the ScienceQA test split, and the validity of exact-match degradation as a robustness measure. The second assumption is contradicted by the authors' own contamination discussion in Section 6.2.

assumptions (3)
  • domain assumption The correct answer to each ScienceQA question is unchanged after adding visual or textual distractions.
    All degradation metrics in Section 4.2 compare against the original ScienceQA labels; if any generated image or text alters the correct answer, the measured degradation is not purely due to distraction. This assumption is asserted in Section 3 but never validated with examples.
  • domain assumption The ScienceQA test partition used to build I-ScienceQA is unseen by the evaluated models.
    Clean evaluation requires that models have not memorized the test samples. Section 6.2 and Figure 3 show that InternVL2 and LLaVA were trained on ScienceQA, so this assumption is violated for several headline models, directly threatening the robustness ranking.
  • domain assumption Exact match on a single run is a sufficient measure of reasoning capability and robustness.
    Section 4.2 defines robustness as exact-match degradation with no variance estimation, no repeated runs, and no confidence intervals, despite small absolute differences such as -0.50 percentage points.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On the robustness of multimodal language model towards distractions." pith.science (2026). https://pith.science/paper/R3LBS46A

@misc{pith2026250209818,
  author       = {Pith},
  title        = {Pith review of: On the robustness of multimodal language model towards distractions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R3LBS46A}},
  note         = {Machine review of arXiv:2502.09818}
}
read the original abstract

Although vision-language models (VLMs) have achieved significant success in various applications such as visual question answering, their resilience to prompt variations remains an under-explored area. Understanding how distractions affect VLMs is crucial for improving their real-world applicability, as inputs could have noisy and irrelevant information in many practical scenarios. This paper aims to assess the robustness of VLMs against both visual and textual distractions in the context of science question answering. Built on the ScienceQA dataset, we developed a new benchmark that introduces distractions in both the visual and textual contexts to evaluate the reasoning capacity of VLMs amid these distractions. Our findings reveal that most-of-the-art VLMs, including GPT-4, are vulnerable to various types of distractions, experiencing noticeable degradation in reasoning capabilities when confronted with distractions. Notably, models such as InternVL2 demonstrate a higher degree of robustness to these distractions. We also found that models exhibit greater sensitivity to textual distractions than visual ones. Additionally, we explored various mitigation strategies, such as prompt engineering, to counteract the impact of distractions. While these strategies improved solution accuracy, our analysis shows that there remain significant opportunities for improvement.

Figures

Figures reproduced from arXiv: 2502.09818 by the authors.

Figure 1
Figure 1. Diagram illustrating various scenario of distraction we [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison of Exact Match Score for Internvl2(top) and [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Training datasets, vision encoders, and language models for LLaVA, CogVLM2, InstructBLIP, and InternVL2. Non-QA datasets [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Can Large Multimodal Models Actively Recognize Faulty Inputs? A Systematic Evaluation Framework of Their Input Scrutiny Ability

    cs.CV 2025-08 unverdicted novelty 6.0 of 10

    Large multimodal models mostly fail to proactively detect flawed textual premises, and their performance depends on error type and on how they weight text versus images.

Reference graph

Works this paper leans on

37 extracted references · 24 canonical work pages · cited by 1 Pith paper

  1. [1]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proc. of International Conference on Computer Vision, pages 2425–2433, 2015. 2

    work page 2015

  2. [2]

    Choquette- Choo, Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Florian Tramer, and Ludwig Schmidt

    Nicholas Carlini, Milad Nasr, Christopher A. Choquette- Choo, Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Florian Tramer, and Ludwig Schmidt. Are aligned neural networks adversarially aligned?, 2024. 2

    work page 2024

  3. [3]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024. 4

    arXiv 2024

  4. [4]

    Halc: Object hallucination re- duction via adaptive focal-contrast decoding

    Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. Halc: Object hallucination re- duction via adaptive focal-contrast decoding. arXiv preprint arXiv:2403.00425, 2024. 1

    arXiv 2024

  5. [5]

    Instructblip: Towards general- purpose vision-language models with instruction tuning,

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning,

  6. [6]

    How Robust is Google’s Bard to Adversarial Image Attacks? arXiv preprint arXiv:2309.11751, 2023

    Yinpeng Dong, Huanran Chen, Jiawei Chen, Zhengwei Fang, Xiao Yang, Yichi Zhang, Yu Tian, Hang Su, and Jun Zhu. How Robust is Google’s Bard to Adversarial Image Attacks? arXiv preprint arXiv:2309.11751, 2023. 2

    arXiv 2023

  7. [7]

    Mme: A comprehensive evaluation benchmark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023. 2

    arXiv 2023

  8. [8]

    Phi3v-finetuning, 2023

    Shao Zhenwei Gai Zhenbiao. Phi3v-finetuning, 2023. GitHub repository. 4 8

    work page 2023

Show all 37 references
  1. [9]

    Hallusionbench: An advanced diag- nostic suite for entangled language hallucination & visual illusion in large vision-language models

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: An advanced diag- nostic suite for entangled language hallucination & visual illusion in large vision-language models. arXiv pr...

    arXiv

  2. [10]

    Cogvlm2: Visual language models for image and video understanding, 2024

    Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, Lei Zhao, Zhuoyi Yang, Xiaotao Gu, Xiaohan Zhang, Guanyu Feng, Da Yin, Zihan Wang, Ji Qi, Xixuan Song, Peng Zhang, Debing Liu, Bin Xu, Juanzi Li, Yuxiao Dong, a...

    2024

  3. [11]

    Large multilingual models pivot zero-shot multimodal learning across languages

    Jinyi Hu, Yuan Yao, Chongyi Wang, Shan Wang, Yinxu Pan, Qianyu Chen, Tianyu Yu, Hanghao Wu, Yue Zhao, Haoye Zhang, Xu Han, Yankai Lin, Jiao Xue, Dahai Li, Zhiyuan Liu, and Maosong Sun. Large multilingual models pivot zero-shot multimodal learning across languages. arXiv prepri...

    2023 arXiv

  4. [12]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6700–6709, 2019. 2

    2019

  5. [13]

    Open- clip, 2021

    Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Han- naneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Open- clip, 2021. If you use this software, please cite it as below. 6

    2021

  6. [14]

    Adversarial examples for evaluating math word problem solvers

    Vivek Kumar, Rishabh Maheshwary, and Vikram Pudi. Adversarial examples for evaluating math word problem solvers. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2705–2712, 2021. 2

    2021

  7. [15]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. arXiv preprint arXiv:2308.00692,

    arXiv

  8. [16]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In EMNLP, pages 292–305,

  9. [17]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 36, 2023. 1, 4

    2023

  10. [18]

    Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023. 2

    2023 arXiv

  11. [19]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. NeurIPS, 35: 2507–2521, 2022. 2, 3

    2022

  12. [20]

    Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts, 2024

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts, 2024. 2

    2024

  13. [21]

    Understanding zero-shot adversarial robust- ness for large-scale models

    Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, and Carl V ondrick. Understanding zero-shot adversarial robust- ness for large-scale models. In The Eleventh International Conference on Learning Representations, 2023. 2

    2023

  14. [22]

    Gpt-4v(ision) system card

    OpenAI. Gpt-4v(ision) system card. https://cdn. openai.com/papers/GPTV_System_Card.pdf ,

  15. [23]

    Gpt-3.5 turbo

    OpenAI. Gpt-3.5 turbo. https://platform.openai. com/docs/models/gpt- 3- 5- turbo , 2024. Ac- cessed: yyyy-mm-dd. 2

    2024

  16. [24]

    Are nlp models really able to solve simple math word problems? In NAACL-HLT, 2021

    Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able to solve simple math word problems? In NAACL-HLT, 2021. 2

    2021

  17. [25]

    Homepage

    ProMeAI. Homepage. https://www.promeai.pro/,

  18. [26]

    Visual adversarial examples jailbreak aligned large language models

    Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Peter Hen- derson, Mengdi Wang, and Prateek Mittal. Visual adversarial examples jailbreak aligned large language models. In Pro- ceedings of the AAAI Conference on Artificial Intelligence , pages 21527–21536, 2024. 2

    2024

  19. [27]

    Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer

    Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10674–10685, 2021. 2, 3

    2022

  20. [28]

    On the adversarial robustness of multi-modal foundation models, 2023

    Christian Schlarmann and Matthias Hein. On the adversarial robustness of multi-modal foundation models, 2023. 2

    2023

  21. [29]

    Robust clip: Unsupervised ad- versarial fine-tuning of vision embeddings for robust large vision-language models

    Christian Schlarmann, Naman Deep Singh, Francesco Croce, and Matthias Hein. Robust clip: Unsupervised ad- versarial fine-tuning of vision embeddings for robust large vision-language models. ICML, 2024. 2, 6

    2024

  22. [30]

    Large language models can be easily distracted by irrelevant context, 2023

    Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Sch ¨arli, and Denny Zhou. Large language models can be easily distracted by irrelevant context, 2023. 1, 2, 3

    2023

  23. [31]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 8317–8326, 2019. 2

    2019

  24. [32]

    Unsplash api

    Unsplash. Unsplash api. https://unsplash.com/ developers, 2024. Accessed: yyyy-mm-dd. 3

    2024

  25. [33]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Jun- yang Lin. Qwen2-vl: Enhancing vision-language model’s ...

    2024

  26. [34]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023. 2

    2023 arXiv

  27. [35]

    On evaluating ad- versarial robustness of large vision-language models, 2023

    Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongx- uan Li, Ngai-Man Cheung, and Min Lin. On evaluating ad- versarial robustness of large vision-language models, 2023. 2 9

    2023

  28. [36]

    Analyzing and mitigating object hallucination in large vision-language models

    Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models. In The Twelfth International Con- ference on Learning Representations, 2024. 1 10

    2024

  29. [2024]

    Accessed: yyyy-mm-dd. 3

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.