Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

SceneTAP: Scene-Coherent Typographic Adversarial Planner against Vision-Language Models in Real-World Environments

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Scene-coherent text tricks vision-language models.

desk verdict SceneTAP's pipeline is a real advance in typographic attack design, but its headline ASR numbers mix true flips with pre-existing target answers and need re-computation before the central claim is supported. read the letter →

arxiv 2412.00114 v2 pith:52JP5X3O submitted 2024-11-28 cs.CV cs.AI

Yue Cao , Yun Xing , Jie Zhang , Di Lin , Tianwei Zhang , Ivor Tsang , Yang Liu , Qing Guo This is my paper · ORCID
classification cs.CVcs.AI
keywords typographicattackadversarialvision-languagemodellargelanguageplanningscenecoherenceTextDiffuserphysical-worldvisualquestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that typographic adversarial attacks on vision-language models can be made scene-coherent and fully automated: an LLM decides what misleading text to add, where to put it, and how to render it so the result looks natural in the image. The authors argue that prior typographic attacks fail on three counts — fixed text, fixed placement, and visibly artificial insertion — and that replacing all three decisions with LLM-based planning plus a text-rendering diffusion model fixes them. If the claim holds, the practical risk of such attacks rises because they survive printing and placement in physical scenes and still mislead current models, including ChatGPT-4o. The paper reports raising average attack success from 12.36% to 44.32% on two-choice questions and from 47.19% to 62.10% on open-ended VQA, with the highest naturalness and combined scores among compared methods.

What carries the argument

The load-bearing mechanism is the LLM-driven planning loop paired with a scene-coherent text renderer. TextDiffuser is a diffusion model that renders short strings into images following a text prompt; SceneTAP uses the LLM to generate that prompt, so the inserted text follows the surface, lighting, and perspective of the chosen region. Set-of-mark prompting supplies a numbered segmentation map that lets the LLM refer to concrete image regions when choosing placement. The revisable prompt acts as a correction pass, moving text near the target region without covering the attribute asked about. Together these components convert an arbitrary image-question pair into a natural-looking typographic attack.

What would settle it

Recompute attack success rate only on image-question pairs where each model answered correctly before any text insertion; if SceneTAP's margin over the no-attack condition collapses toward the margin of center placement, the claim that scene-coherent planning causes the misdirection is falsified.

Watch

Extended reading notes

Core claim

The central discovery is that the content, placement, and visual rendering of an adversarial text can be planned jointly by a general-purpose LLM rather than fixed by a human or a rigid rule. Given the image, question, and correct answer, SceneTAP first analyzes the scene through chain-of-thought reasoning, selects a short incorrect answer that is plausible in context, uses set-of-mark prompting to pick a region near the question-targeted object, and produces a natural-language instruction for a TextDiffuser model to paint the text onto that surface. A revisable prompt lets the planner adjust placement when the chosen spot would alter an attribute central to the question or sit on an unrealistic surface. The resulting digital images, and printed physical versions of them, shift model answers toward the planted text while scoring higher on the paper's naturalness metric than center or margin insertion.

Load-bearing premise

The reported attack success counts any output that matches the target answer as a success, even when the model already gave that answer before any text was added, so the attack's causal contribution is not isolated from the model's pre-existing tendency.

Editorial extensions

If this is right

  • SceneTAP raises attack success rate above both center and margin baselines on two-choice and open-ended VQA, across LLaVA, InstructBLIP, MiniGPT-v2, and ChatGPT-4o.
  • Because the inserted text is rendered to match the scene, the attacks receive higher naturalness scores and remain effective when printed and photographed in physical environments.
  • Ablation results attribute the gain to all three planning decisions: question-relevant adversarial text, placement near the question-targeted region, and diffusion-based insertion.
  • The method exposes a vulnerability in current LVLMs that do not distinguish genuinely present scene text from adversarial planted text, suggesting defenses must check text plausibility beyond surface appearance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An editorial caution: because the reported ASR counts target-matching answers even when no attack was applied, re-evaluating on only initially-correct responses would likely shrink the reported gains; a fair comparison should condition on that subset.
  • A testable extension is to run SceneTAP on images with no natural text surfaces, such as open landscapes; the paper's own limitation note predicts ASR and naturalness would drop, which would quantify the cost of the scene-coherence constraint.
  • The same planner could be inverted as a defense generator: synthesize realistic misleading text to fine-tune LVLMs to ignore contextually plausible but physically absent text, or to cross-check OCR output against scene semantics.
  • Physical deployment, while demonstrated, makes the attack static and detectable by repeated observation over time; a dynamic variant would need to re-plan text for changing scenes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SceneTAP, a training-free LLM-based planner for generating scene-coherent typographic adversarial attacks against vision-language models. The method uses ChatGPT-4o to analyze an image and question, generate an adversarial text, choose a placement via Set-of-Mark prompting, and prompt TextDiffuser to insert the text into the scene; a revisable prompt refines the plan. The authors evaluate on TypoD-base, LingoQA, and VQAv2 across four LVLMs, comparing with Center and Margin attacks and reporting attack success rate (ASR), a ChatGPT-assessed naturalness score (N-Score), and a combined C-Score. They also demonstrate a physical-world extension by printing and pasting generated patches in four cases.

Significance. If the quantitative claims held, SceneTAP would be a useful contribution: it automates typographic attack design, is training-free, includes a physical attack demonstration, and releases code. The systematic study of adversarial text type and placement in Section 3 is a useful empirical addition. However, the current evaluation does not support the headline numbers: ASR is not conditioned on clean-image correctness, and the naturalness metric is assigned by the same model that generates the attacks. The relative ordering of methods may survive a corrected analysis, but the reported magnitudes and the 'misleads' claim need revision.

major comments (4)
  1. [Section 5.1, Table 1] ASR is not conditioned on initially correct clean-image responses. Section 5.1 defines ASR as the 'percentage of successful attacks that deceive the target AI model' but does not require the clean image to be answered correctly. Consequently, Table 1's No-Attack rows are nonzero and often large: ChatGPT-4o LingoQA 47.1%, VQAv2 35.6%; LLaVA LingoQA 65.6%; MiniGPT-v2 LingoQA 62.1%; InstructBLIP LingoQA 62.9%. For open-ended VQA, the target answer is generated by ChatGPT from the same image, question, and correct answer, so it can coincide with what the victim model already outputs on the clean image. Counting such pre-existing outputs as 'attacks' overstates the causal role of the inserted text and inflates the reported gains, including the 47.19% to 62.10% average and the 47.1% to 73.4% LingoQA line for ChatGPT-4o. Section 3.1 explicitly filters to initially correct responses, so the omission is an internal inconsistency. Please recompute ASR on the subset where the clean image is answered correctly, or report flip rates and deltas relative to the no-attack baseline; this is necessary before the 'misleads' claim is quantitatively supported.
  2. [Section 5.1, Section A.2, Section 4.5] The N-Score is assigned by ChatGPT-4o, which is the same model used as the planner (Section 4.5) and is also one of the victim models in Table 1. This creates a same-model evaluation loop for the naturalness claim: the model that designs the attack also judges its visual naturalness, and the C-Score in Section 5.1 inherits this loop. Since the paper's claim of maintaining visual naturalness rests on these scores, please provide an independent human evaluation or a different judge model, with agreement statistics, and separate the planner model from the evaluator model.
  3. [Section 5.3, Figure 4] The physical-world evidence is anecdotal. Section 5.3 presents only four cases, with no physical attack success rate, no quantitative comparison between physical and digital success, and no details on repeat trials, camera viewpoints, or lighting conditions. The abstract's claim that the method remains effective 'even after capturing new images of physical setups' is therefore not quantitatively established. Please add a protocol and numbers for the physical experiments, even if on a modest scale.
  4. [Table 1] All reported ASR, N-Score, and C-Score values are single point estimates without error bars, multiple runs, or significance tests. Since the planner is a stochastic LLM and some evaluation subsets are small (e.g., 100 image-question pairs in the Section 3.1 study, 500 VQAv2 pairs in Section 5.1), the differences between methods may not be stable. Please report variation across repeated runs or clearly state the sample sizes and any significance measures.
minor comments (5)
  1. [Section 5.1] The C-Score is described as averaging the ASR and N-Score, but ASR is on a 0-100 scale and N-Score is on a 0-10 scale; the table values imply C-Score = (ASR + 10 * N-Score) / 2. Please state the scaling explicitly.
  2. [Abstract/Introduction] The text contains a typo: 'What action should be taked for the car' should read 'What action should be taken for the car'.
  3. [Section 4.5] The revisable prompt is shown in a box but the paper does not specify how it is invoked or how the model decides whether to modify the plan. Please describe the inference procedure more concretely.
  4. [Table 2] The ablation settings 'Plan1' and 'Plan2' are defined only in the table caption; please define them in the main text before the ablation discussion.
  5. [Supplementary A.3] The SoM mask-filtering ratio 'a' is set to different values per dataset but no sensitivity analysis is provided for this free parameter.

Circularity Check

2 steps flagged · score 5.0 of 10

Partially circular: VQA targets are generated by the same ChatGPT-4o that is evaluated, and naturalness is self-scored by ChatGPT-4o; open-source and physical results remain external.

  1. self definitional [Sec. 5.1 (Metrics, Baselines) and Table 1; cf. Sec. 3.1]
    "Attack Success Rate (ASR) measures the percentage of successful attacks that deceive the target AI model, indicating the attack’s effectiveness. ... For VQA, we prompt ChatGPT to generate an incorrect answer using the image, question, and correct answer."

    Sec. 5.1 defines ASR without conditioning on the clean-image answer being correct, while Sec. 3.1's pilot required exactly that ('initial model responses were correct'). Table 1's No Attack rows are large (ChatGPT-4o: 47.1% on LingoQA, 35.6% on VQAv2), so the metric counts outputs already equal to the target before text insertion. The VQA target is itself produced by prompting ChatGPT-4o, the same model used as victim; hence the reported 47.19%→62.10% average and 47.1%→73.4% LingoQA line partly measure the model agreeing with its own default wrong answer, not a text-induced flip. With success defined as output = t and t sampled from the victim's own distribution, a large share of 'successful attacks' is true by construction rather than caused by the attack.

  2. other [Sec. 5.1 (N-Score and C-Score); Sec. 4.5 Implementation; Supplementary A.2]
    "The N-Score is a 10-point metric evaluated by ChatGPT to assess the natural integration of adversarial text within an image. ... We employ ChatGPT (gpt-4o-2024-08-06) as the planner, i.e., U in Eq. (1). ... Currently, there is no established method for evaluating the naturalness of text added to images. To address this gap, we propose the N-Score, which uses ChatGPT-4o to assess the integration of text into the scene."

    The same ChatGPT-4o system that plans the attack (U in Eqs. 1-3; Sec. 4.5) also supplies the N-Score used to substantiate 'maintaining visual naturalness.' The paper offers no human or external naturalness benchmark, and the N-Score's ten criteria (lighting, surface, perspective, etc.) overlap with the placement and realism constraints in the planner's revisable prompt. The C-Score then averages this self-assigned score with ASR. This is a same-model evaluation loop: the naturalness evidence is a measure of the planner's consistency with its own generation conventions, not an independent property of the image, so the claimed naturalness result rests on the same system that produced the attacked images.

full rationale

SceneTAP has two partially circular evaluation loops, but its central effectiveness claim also has independent support. First, the VQA ASR is defined in Sec. 5.1 as the percentage of outputs matching the adversarial target, with no requirement that the clean-image answer was correct; yet Sec. 3.1's pilot study filtered to 'initial model responses were correct.' Table 1's No Attack rows are large (ChatGPT-4o: 47.1% on LingoQA, 35.6% on VQAv2), and the same section states that for VQA 'we prompt ChatGPT to generate an incorrect answer using the image, question, and correct answer.' Because the victim in those rows is ChatGPT-4o, a large share of the reported success is the model emitting its own pre-existing wrong answer rather than a flip caused by the inserted text; re-reporting ASR conditional on initially-correct clean answers is necessary. Second, the naturalness claim rests on an N-Score assigned by ChatGPT-4o, the same model that acts as the planner U and writes the TextDiffuser prompt (Secs. 4.1-4.5, A.2). No human or external benchmark is provided, so the score measures the planner's self-consistency rather than an independent visual-naturalness property; the C-Score then folds this self-score into the headline. The self-citations in the related work are contextual and not load-bearing. The method does have external grounding: ASR on LLaVA, InstructBLIP, and MiniGPT-v2, and the printed physical-scene experiments, are independent of the ChatGPT-4o planner and support the claim that SceneTAP increases attack success. Those results keep the paper from being wholly circular, but the ChatGPT-4o VQA numbers and the naturalness metric are partially circular and overstate the causal and aesthetic claims.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper does not introduce new physical entities or fit central constants. Its freedom is in evaluation choices: the ASR baseline is not corrected, and the naturalness judge is the same model family that made the attacks.

free parameters (2)
  • SoM mask filtering ratio a = 12 (TypoD-base, VQAv2), 15 (LingoQA)
    Hand-chosen per dataset to discard small segmentation masks; does not affect the attack concept.
  • SoM slider value = 3
    Set to control SAM segmentation granularity; a fixed implementation detail.
assumptions (4)
  • domain assumption ChatGPT-4o is a reliable planner for adversarial text and placement
    The whole pipeline relies on the LLM's scene understanding and instruction following; no validation that another planner behaves similarly.
  • domain assumption TextDiffuser renders inserted text that is visually coherent with the scene
    The naturalness of the final image depends on TextDiffuser's rendering quality; no quantitative comparison against alternative renderers.
  • domain assumption ChatGPT-4o's N-Score is a valid measure of visual naturalness
    The naturalness metric is assigned by the same model family that generated the attack; no human evaluation is provided.
  • domain assumption SoM segmentation accurately identifies writable regions
    Placement relies on SAM-based masks and marks generated by SoM; failure would misplace text.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SceneTAP: Scene-Coherent Typographic Adversarial Planner against Vision-Language Models in Real-World Environments." pith.science (2026). https://pith.science/paper/52JP5X3O

@misc{pith2026241200114,
  author       = {Pith},
  title        = {Pith review of: SceneTAP: Scene-Coherent Typographic Adversarial Planner against Vision-Language Models in Real-World Environments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/52JP5X3O}},
  note         = {Machine review of arXiv:2412.00114}
}
read the original abstract

Large vision-language models (LVLMs) have shown remarkable capabilities in interpreting visual content. While existing works demonstrate these models' vulnerability to deliberately placed adversarial texts, such texts are often easily identifiable as anomalous. In this paper, we present the first approach to generate scene-coherent typographic adversarial attacks that mislead advanced LVLMs while maintaining visual naturalness through the capability of the LLM-based agent. Our approach addresses three critical questions: what adversarial text to generate, where to place it within the scene, and how to integrate it seamlessly. We propose a training-free, multi-modal LLM-driven scene-coherent typographic adversarial planning (SceneTAP) that employs a three-stage process: scene understanding, adversarial planning, and seamless integration. The SceneTAP utilizes chain-of-thought reasoning to comprehend the scene, formulate effective adversarial text, strategically plan its placement, and provide detailed instructions for natural integration within the image. This is followed by a scene-coherent TextDiffuser that executes the attack using a local diffusion mechanism. We extend our method to real-world scenarios by printing and placing generated patches in physical environments, demonstrating its practical implications. Extensive experiments show that our scene-coherent adversarial text successfully misleads state-of-the-art LVLMs, including ChatGPT-4o, even after capturing new images of physical setups. Our evaluations demonstrate a significant increase in attack success rates while maintaining visual naturalness and contextual appropriateness. This work highlights vulnerabilities in current vision-language models to sophisticated, scene-coherent adversarial attacks and provides insights into potential defense mechanisms.

Figures

Figures reproduced from arXiv: 2412.00114 by the authors.

Figure 1
Figure 1. Left: Typographic attack and Difference of our method SceneTAP to SOTA methods,i.e., Center Attack (ECCV 2024) [1] and Margin Attack [2]. Right: Physical implementation of our method and ChatGPT4o’s responses on the original image, generation of SceneTAP, and physical version of SceneTAP. Abstract Large vision-language models (LVLMs) have shown remark￾able capabilities in interpreting visual content. While exist￾ing… view at source ↗
Figure 2
Figure 2. (a) An example of inserting 4 types of adversarial texts. (b) Quantitative results of 4 types of adversarial texts on 100 image-question pairs when we attack LLaVA-1.5-13b model. We use the attack success rate (ASR) as the metric. (c)-Left: Influence of Adversarial Text Placement, with examples of Attack Strength Heatmaps for specific questions featuring adversarial text in different locations. (c)-Right: Influence … view at source ↗
Figure 3
Figure 3. Pipeline of our scene-coherent typographic adversarial planner (SceneTAP) and its intermediate outputs leading to the final generated image. 4.1. Overview Given an input image I, a question q, and a correct answer a, we leverage a vision-language model U to perform the scene-coherent adversarial attack. Specifically, we provide the model with image I, query q, correct answer a and instruction γt to generate the adve… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Visual￾ization comparing SceneTAP adver￾sarial examples: Digital SceneTAP (generated) and Physical Scene￾TAP (real-world implementation). Physical examples were created by printing the generated texts (shown in right subfigure), applying them to identical scenes, and c…
Figure 5
Figure 5. Figure 5: Ablation study on the influence of the main components in SceneTAP. Text Placement Insertion Setting1 No Center No Setting2 Yes Center No Setting3 Yes Plan1 No Setting4 Yes Plan2 No Setting5 Yes Plan2 Yes [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Visualization of the N-Score assessment across different score ranges. The arrows indicate the locations of the added text within each image. Question: What entity is depicted in the image? (a) colobus (b) bobsled Correct Answer: b. Attacked Answer: a. Question: What c…
Figure 7
Figure 7. Figure 7: Visualization of SceneTAP on the TypoD-base Dataset. A.4. Visualization In this section, we provide additional visualization results in [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Visualization of SceneTAP on the VQAv2 Dataset. ended questions. For instance, in TypoD-base, adding the text “colobus” causes the VLM to incorrectly identify the entity in the image. In LingoQA, inserting the phrase “Red light” within an image leads to an incorrect op…
Figure 9
Figure 9. Figure 9: Visualization of SceneTAP on the LingoQA dataset. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MAGIC: Mastering Physical Adversarial Generation in Context through Collaborative LLM Agents

    cs.CV 2024-12 conditional novelty 7.0 of 10

    MAGIC uses three collaborating LLM agents to generate scene-aware adversarial patches and place them in real-world images, achieving higher attack success against YOLO and DETR detectors than the natural diffusion att...

  2. Defending LVLMs Against Vision Attacks through Partial-Perception Supervision

    cs.CV 2024-12 conditional novelty 6.0 of 10

    DPS uses partial-image descriptions as supervision to prompt a vision-language model to correct itself, reducing attack success by about 76% across six datasets.

Reference graph

Works this paper leans on

57 extracted references · 37 canonical work pages · cited by 2 Pith papers

  1. [1]

    Un- veiling typographic deceptions: Insights of the typographic vulnerability in large vision-language model.arXiv

    Hao Cheng, Erjia Xiao, Jindong Gu, Le Yang, Jinhao Duan, Jize Zhang, Jiahang Cao, Kaidi Xu, and Renjing Xu. Un- veiling typographic deceptions: Insights of the typographic vulnerability in large vision-language model.arXiv. org, 2024. 1, 2, 4, 7

    work page 2024

  2. [2]

    Vision-llms can fool themselves with self-generated typographic attacks

    Maan Qraitem, Nazia Tasnim, Kate Saenko, and Bryan A Plummer. Vision-llms can fool themselves with self-generated typographic attacks. arXiv preprint arXiv:2402.00626, 2024. 1, 2, 3, 4, 7

    arXiv 2024

  3. [3]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, pages 8748–8763. PMLR, 2021. 1

    2021

  4. [4]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736, 2022

    2022

  5. [5]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 1

    2024

  6. [6]

    Towards deep learn- ing models resistant to adversarial attacks

    Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learn- ing models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017. 2

    arXiv 2017

  7. [7]

    Irad: implicit representation-driven image resampling against adversarial attacks

    Yue Cao, Tianlin Li, Xiaofeng Cao, Ivor W Tsang, Yang Liu, and Qing Guo. Irad: implicit representation-driven image resampling against adversarial attacks. In ICLR, 2024

    work page 2024

  8. [8]

    Lrr: Language- driven resamplable continuous representation against adver- sarial tracking attacks

    Jianlang Chen, Xuhong Ren, Qing Guo, Felix Juefei-Xu, Di Lin, Wei Feng, Lei Ma, and Jianjun Zhao. Lrr: Language- driven resamplable continuous representation against adver- sarial tracking attacks. In ICLR, 2024

    work page 2024

Show all 57 references
  1. [9]

    On the robustness of segment anything

    Yihao Huang, Yue Cao, Tianlin Li, Felix Juefei-Xu, Di Lin, Ivor W Tsang, Yang Liu, and Qing Guo. On the robustness of segment anything. arXiv preprint arXiv:2305.16220, 2023

    2023 arXiv

  2. [10]

    Adversarial relighting against face recognition

    Qian Zhang, Qing Guo, Ruijun Gao, Felix Juefei-Xu, Hongkai Yu, and Wei Feng. Adversarial relighting against face recognition. IEEE Transactions on Information Forensics and Security, 2024

    2024

  3. [11]

    Magic: Mastering physical adversarial generation in context through collaborative llm agents

    Yun Xing, Nhat Chung, Jie Zhang, Yue Cao, Ivor Tsang, Yang Liu, Lei Ma, and Qing Guo. Magic: Mastering physical adversarial generation in context through collaborative llm agents. arXiv preprint arXiv:2412.08014, 2024

    2024 arXiv

  4. [12]

    Ala: Naturalness-aware adversarial lightness attack

    Yihao Huang, Liangru Sun, Qing Guo, Felix Juefei-Xu, Ji- ayi Zhu, Jincao Feng, Yang Liu, and Geguang Pu. Ala: Naturalness-aware adversarial lightness attack. arXiv preprint arXiv:2201.06070, 2022. 2

    2022 arXiv

  5. [13]

    On evaluating adversarial robustness of large vision-language models

    Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongx- uan Li, Ngai-Man Man Cheung, and Min Lin. On evaluating adversarial robustness of large vision-language models. Ad- vances in Neural Information Processing Systems, 36, 2024. 2

    2024

  6. [14]

    Instructta: Instruction-tuned targeted attack for large vision-language models

    Xunguang Wang, Zhenlan Ji, Pingchuan Ma, Zongjie Li, and Shuai Wang. Instructta: Instruction-tuned targeted attack for large vision-language models. arXiv preprint arXiv:2312.01886, 2023

    2023 arXiv

  7. [15]

    Transferable multimodal attack on vision-language pre-training models

    Haodi Wang, Kai Dong, Zhilei Zhu, Haotong Qin, Aishan Liu, Xiaolin Fang, Jiakai Wang, and Xianglong Liu. Transferable multimodal attack on vision-language pre-training models. In 2024 IEEE Symposium on Security and Privacy (SP), pages 102–102. IEEE Computer Society, 2024

    2024

  8. [16]

    Towards adversarial at- tack on vision-language pre-training models

    Jiaming Zhang, Qi Yi, and Jitao Sang. Towards adversarial at- tack on vision-language pre-training models. In Proceedings of the 30th ACM International Conference on Multimedia , pages 5005–5013, 2022

    2022

  9. [17]

    Set-level guidance at- tack: Boosting adversarial transferability of vision-language pre-training models

    Dong Lu, Zhiqiang Wang, Teng Wang, Weili Guan, Hongchang Gao, and Feng Zheng. Set-level guidance at- tack: Boosting adversarial transferability of vision-language pre-training models. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 102–111, 2023

    2023

  10. [18]

    Boosting transferability in vision-language attacks via diversification along the intersection region of adversarial trajectory

    Sensen Gao, Xiaojun Jia, Xuhong Ren, Ivor Tsang, and Qing Guo. Boosting transferability in vision-language attacks via diversification along the intersection region of adversarial trajectory. In European Conference on Computer Vision , pages 442–460. Springer, 2024. 2

    2024

  11. [19]

    Towards trans- ferable attacks against vision-llms in autonomous driving with typography

    Nhat Chung, Sensen Gao, Tuan-Anh Vu, Jie Zhang, Aishan Liu, Yun Lin, Jin Song Dong, and Qing Guo. Towards trans- ferable attacks against vision-llms in autonomous driving with typography. arXiv preprint arXiv:2405.14169, 2024. 2, 4

    2024 arXiv

  12. [20]

    Textdiffuser-2: Unleashing the power of language models for text rendering

    Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, and Furu Wei. Textdiffuser-2: Unleashing the power of language models for text rendering. In European Conference on Computer Vision, pages 386–402. Springer, 2024. 2, 5

    2024

  13. [21]

    An image is worth 1000 lies: Adversarial transferability across prompts on vision-language models

    Haochen Luo, Jindong Gu, Fengyuan Liu, and Philip Torr. An image is worth 1000 lies: Adversarial transferability across prompts on vision-language models. arXiv preprint arXiv:2403.09766, 2024. 2

    2024 arXiv

  14. [22]

    On the robustness of large multimodal mod- els against image adversarial attacks

    Xuanming Cui, Alejandro Aparcedo, Young Kyun Jang, and Ser-Nam Lim. On the robustness of large multimodal mod- els against image adversarial attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24625–24634, 2024. 2 9

    2024

  15. [23]

    Inducing high energy-latency of large vision-language models with verbose images

    Kuofeng Gao, Yang Bai, Jindong Gu, Shu-Tao Xia, Philip Torr, Zhifeng Li, and Wei Liu. Inducing high energy-latency of large vision-language models with verbose images. arXiv preprint arXiv:2401.11170, 2024. 2

    2024 arXiv

  16. [24]

    Multimodal neurons in artificial neural networks

    Gabriel Goh, Nick Cammarata, Chelsea V oss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Dis- till, 6(3):e30, 2021. 2

    2021

  17. [25]

    Blended diffusion for text-driven editing of natural images

    Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18208–18218, 2022. 2

    2022

  18. [26]

    Dis- entangling visual and written concepts in clip

    Joanna Materzy´nska, Antonio Torralba, and David Bau. Dis- entangling visual and written concepts in clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16410–16419, 2022. 2

    2022

  19. [27]

    Patching open-vocabulary models by interpolating weights

    Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulary models by interpolating weights. Advances in Neural Infor- mation Processing Systems, 35:29262–29277, 2022. 2

    2022

  20. [28]

    Defense-prefix for pre- venting typographic attacks on clip

    Hiroki Azuma and Yusuke Matsui. Defense-prefix for pre- venting typographic attacks on clip. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3644–3653, 2023. 2

    2023

  21. [29]

    Defending lvlms against vision attacks through partial-perception supervision, 2024

    Qi Zhou, Tianlin Li, Qing Guo, Dongxia Wang, Yun Lin, Yang Liu, and Jin Song Dong. Defending lvlms against vision attacks through partial-perception supervision, 2024. 2

    2024

  22. [30]

    Ad- versarial machine learning at scale

    Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Ad- versarial machine learning at scale. arXiv preprint arXiv:1611.01236, 2016. 3

    2016 arXiv

  23. [31]

    Adver- sarial examples in the physical world

    Alexey Kurakin, Ian J Goodfellow, and Samy Bengio. Adver- sarial examples in the physical world. InArtificial intelligence safety and security, pages 99–112. Chapman and Hall/CRC,

  24. [32]

    Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition

    Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K Reiter. Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition. In Proceedings of the 2016 acm sigsac conference on computer and communi- cations security, pages 1528–1540, 2016. 3

    2016

  25. [33]

    Robust physical-world attacks on deep learning visual classification

    Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust physical-world attacks on deep learning visual classification. In Proceedings of the IEEE conference on computer vision and pattern recognit...

    2018

  26. [34]

    Towards transferable targeted 3d adversarial attack in the physical world

    Yao Huang, Yinpeng Dong, Shouwei Ruan, Xiao Yang, Hang Su, and Xingxing Wei. Towards transferable targeted 3d adversarial attack in the physical world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24512–24522, 2024. 3

    2024

  27. [35]

    Adversarial t-shirt! evading person detectors in a physical world

    Kaidi Xu, Gaoyuan Zhang, Sijia Liu, Quanfu Fan, Mengshu Sun, Hongge Chen, Pin-Yu Chen, Yanzhi Wang, and Xue Lin. Adversarial t-shirt! evading person detectors in a physical world. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedi...

    2020

  28. [36]

    Fooling thermal infrared pedestrian detectors in real world using small bulbs

    Xiaopei Zhu, Xiao Li, Jianmin Li, Zheyao Wang, and Xiaolin Hu. Fooling thermal infrared pedestrian detectors in real world using small bulbs. In Proceedings of the AAAI confer- ence on artificial intelligence, volume 35, pages 3616–3624,

  29. [37]

    Infrared invisible clothing: Hiding from infrared detectors at multiple angles in real world

    Xiaopei Zhu, Zhanhao Hu, Siyuan Huang, Jianmin Li, and Xiaolin Hu. Infrared invisible clothing: Hiding from infrared detectors at multiple angles in real world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13317–13326, 2022

    2022

  30. [38]

    Hotcold block: Fooling thermal infrared detectors with a novel wearable design

    Hui Wei, Zhixiang Wang, Xuemei Jia, Yinqiang Zheng, Hao Tang, Shin’ichi Satoh, and Zheng Wang. Hotcold block: Fooling thermal infrared detectors with a novel wearable design. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 15233–15241, 2023. 3

    2023

  31. [39]

    Adversarial camouflage: Hiding physical- world attacks with natural styles

    Ranjie Duan, Xingjun Ma, Yisen Wang, James Bailey, A Kai Qin, and Yun Yang. Adversarial camouflage: Hiding physical- world attacks with natural styles. In Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition, pages 1000–1008, 2020. 3

    2020

  32. [40]

    Uni- fied adversarial patch for cross-modal attacks in the physical world

    Xingxing Wei, Yao Huang, Yitong Sun, and Jie Yu. Uni- fied adversarial patch for cross-modal attacks in the physical world. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 4445–4454, 2023. 3

    2023

  33. [41]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. 4, 7

    2023

  34. [42]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, ...

    2017

  35. [43]

    Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441, 2023. 5

    2023 arXiv

  36. [44]

    Textdiffuser: Diffusion models as text painters

    Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, and Furu Wei. Textdiffuser: Diffusion models as text painters. Advances in Neural Information Processing Systems, 36, 2024. 5

    2024

  37. [45]

    Lingoqa: Visual question answering for au- tonomous driving

    Ana-Maria Marcu, Long Chen, Jan Hünermann, Alice Karn- sund, Benoit Hanotte, Prajwal Chidananda, Saurabh Nair, Vijay Badrinarayanan, Alex Kendall, Jamie Shotton, and Oleg Sinavski. Lingoqa: Visual question answering for au- tonomous driving. arXiv preprint arXiv:2312.14115, 2023. 7

    2023 arXiv

  38. [46]

    Instructblip: Towards general-purpose vision- language models with instruction tuning, 2023

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning, 2023. 7

    2023

  39. [47]

    What color is the traffic light?

    Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechu Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310....

    arXiv

  40. [48]

    Lighting: Does the text match the scene’s lighting (brightness, shadows)?

  41. [49]

    Shadows: Does the text cast shadows or interact correctly with existing shadows?

  42. [50]

    Perspective: Is the text aligned with the scene’s perspective and surface geometry?

  43. [51]

    Depth: Does the text integrate naturally with the depth and contours of the scene?

  44. [52]

    Appropriate Surface: Is the text placed on a surface where text would naturally appear?

  45. [53]

    Surface Texture: Does the text interact realistically with the surface texture (e.g., follows bumps or grooves)?

  46. [54]

    Font Suitability: Is the font appropriate for the scene’s con- text?

  47. [55]

    Color Harmony: Does the text’s color fit naturally within the scene?

  48. [56]

    Edge Realism: Are the text edges rendered to match the image quality (sharpness or blur)?

  49. [57]

    colobus” causes the VLM to incorrectly identify the entity in the image. In LingoQA, inserting the phrase “Red light

    Blending: Does the text blend seamlessly into the image with- out signs of manipulation? Fig. 6 presents the visualization results of images cate- gorized according to different N-Score ranges, illustrating the relationship between N-Scores and the naturalness of text integrat...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.