Pith. sign in

REVIEW 3 major objections 5 minor 26 references

Typographic Attacks in a Multi-Image Setting

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Typographic attacks on image sets succeed most when each image gets the attack text most similar to it, not the strongest or a random one.

desk verdict The multi-image non-repeating setup is a genuine new framing, and the text-image similarity result is plausible, but the class-label-only vocabulary and class-balanced target sets make the 21% gain look narrower than the abstract suggests. read the letter →

arxiv 2502.08193 v1 pith:AN6AUPFE submitted 2025-02-12 cs.CR

Xiaomeng Wang , Zhengyu Zhao , Martha Larson This is my paper · ORCID
classification cs.CR
keywords typographicattacksmulti-imageattackvision-languagemodelsCLIPtext-imagesimilaritynon-repeatingtexteffectivenessInstructBLIP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces a multi-image setting for typographic attacks in which an attacker must attack a set of images without reusing the same attack text, because repetition would be obvious to a gatekeeper. It claims that the right strategy is to assign each image the attack text with the highest text-image similarity, computed with CLIP, excluding the image's own label and any text already used. On ImageNet with OpenCLIP, this similarity-based matching improves attack success by 21% over random, non-specific assignment while keeping the non-repeating, stealthier property. The same CLIP-computed similarity also transfers to attacking InstructBLIP in a greybox setting, where the attacker never accesses the target model.

What carries the argument

The central object is the text-image similarity score, defined as the cosine similarity between the CLIP embedding of the target image and the CLIP embedding of a candidate one-word attack text. The paper combines it with visual image prediction probability (VIPP), the ground-truth prediction probability of the original image, to define VIPP-TextImSim: order images by descending VIPP, then for each image select the highest-similarity unused attack text. The same similarity score, without VIPP ordering, defines Rand-TextImSim. This machinery converts the multi-image attack into a 1-to-1 matching problem and provides the per-image signal that generalizes to InstructBLIP.

What would settle it

Rerun the released matching procedure with attack texts sampled from a word list that excludes all ImageNet labels but keeps the same font and placement: if VIPP-TextImSim no longer beats random matching by a wide margin, the benefit is an artifact of the label vocabulary rather than a general property of text-image similarity.

Watch

Extended reading notes

Core claim

The paper claims that in a non-repeating multi-image typographic attack, attack text selection should be driven by the similarity between the candidate text and the specific target image, not by the text's average effectiveness across images and not by random assignment. Its best strategy, VIPP-TextImSim, sorts images by how hard they are to attack (visual image prediction probability) and then greedily gives each image the unused one-word attack text closest to it in CLIP embedding space. This strategy matches or beats all alternatives, while the attack-text-effectiveness strategy performs close to random. The paper also claims transferability: similarity scores computed with CLIP improve targeted attack success on InstructBLIP, a different model with a CLIP-like vision encoder.

Load-bearing premise

The 21% margin is measured with attack texts drawn from the model's own 579 one-word class labels and with target sets containing one image from each of 579 different classes, so the attacker benefits from near-class confusions that may not exist when attack texts come from a general vocabulary.

Editorial extensions

If this is right

  • An attacker who must avoid repeating attack texts can still achieve substantially higher attack success by choosing each text from a similarity ranking rather than at random.
  • Attack text effectiveness, measured as average success across images, is a weaker selection signal than per-image text-image similarity in the non-repeating setting.
  • Because CLIP-based similarity transfers to InstructBLIP, a greybox attacker can target models with CLIP-like vision encoders without whitebox access.
  • As the allowed number of text repetitions increases, attack success rises for both similarity- and effectiveness-based strategies, but the similarity-based strategy remains stronger.
  • The trade-off between repetition and attack success gives defenders a concrete quantity to measure when deciding how strictly to police repeated text.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, a defender could use the text-image similarity ranking to prioritize inspection of image-text pairs that are most likely to be dangerous, effectively turning the attack's own selection signal into a detection signal.
  • Beyond the paper, the single-word label vocabulary is an attacker-friendly condition; if attack texts are drawn from general English words outside the label set, the 21% margin is likely to shrink, as the paper's limitation section hints.
  • Beyond the paper, the untargeted misclassifications the paper observes may be the stealthiest of all, since the attacker's intent cannot be read from the text; this suggests defenses should look for text-induced confusion without assuming a known target label.
  • Beyond the paper, the same text-image similarity score could be used as an alignment diagnostic for the embedding space, ranking which text labels sit too close to image regions and thus identifying confusion-prone areas.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a multi-image setting for typographic attacks in which an attacker must assign a distinct attack text to each image in a set, avoiding repetition for stealth. Using OpenCLIP ViT-B/32 on ImageNet validation images correctly classified by the model, the authors define attack text effectiveness (ATE), visual image prediction probability (VIPP), and text-image similarity (cosine similarity in CLIP space). They propose four matching strategies (HighVIPP-LowATE, HighVIPP-HighATE, Rand-TextImSim, VIPP-TextImSim) and compare them against random matching on five class-balanced sets of 579 images with 579 one-word ImageNet class labels as the attack texts. They report that text-image similarity strategies outperform random and ATE-based strategies, with the abstract claiming a 21% improvement over random methods, and they present a greybox transfer experiment in which CLIP-based text-image similarity is used to attack InstructBLIP.

Significance. The paper opens a realistic attack dimension, namely non-repeating multi-image typographic attacks, which is underexplored in the literature. The evaluation uses held-out random image sets, the source code is released, and the InstructBLIP experiment provides an external check that CLIP-computed similarity carries some transferable signal. If the 21% advantage were robust, the work would be a useful step toward understanding how attackers choose typographic texts in batch settings. However, the central quantitative claim is currently under-supported because the exact Figure 6 values and significance tests are missing, and the advantage may be specific to the choice of one-word ImageNet class labels as the attack vocabulary combined with class-balanced target sets.

major comments (3)
  1. [§6.2, Fig. 6] The abstract's central quantitative claim ('21% over random, non-specific methods') is not verifiable from the manuscript text: Figure 6 shows average attack success rates above the bars, but those values are not given in the text or caption, and no standard deviations or significance tests are reported for the comparison between Rand and the TextImSim strategies over the five random image sets. Please report the exact per-strategy ASRs (both untargeted and targeted), the variability across the five sets, and a paired test (e.g., Wilcoxon signed-rank or permutation test) for Rand versus Rand-TextImSim and Rand versus VIPP-TextImSim.
  2. [§4.1, §6.1, §5.3] The text-image similarity advantage may be an artifact of the experimental construction. The attack text pool is exactly the 579 one-word ImageNet class labels, and the target sets are built with one image from each of 579 different classes, so for most images the highest-similarity unused text is a semantically or visually related class label (dragonfly→lycaenid, baseball→ballplayer in Figure 5). These near-class confusions are precisely the cases where typographic attacks are known to be strong, and the Rand baseline cannot benefit from them. The Limitations section acknowledges the single-word restriction but does not test attack texts drawn from outside the ImageNet label vocabulary, nor target sets that are not class-balanced. Please add experiments with a broader text pool (e.g., arbitrary words or multi-word phrases) and with imbalanced or single-class target sets, or explicitly restrict the paper's claim to the label-vocabulary, class-balanced setting.
  3. [§8.2, Table 1] The transferability experiment uses a forced-choice prompt that contains exactly the ground-truth label and the attack text label as the only two options, and reports only targeted accuracy. Because the attack text is selected to have high CLIP similarity to the image, the VIPP-TextImSim condition may benefit from presenting a more plausible distractor in the two-option prompt rather than from a genuinely stronger attack. The paper should either discuss this confound explicitly or strengthen the evaluation by using a larger option set (even a subset of ImageNet labels) or open-ended generation with output mapping.
minor comments (5)
  1. [§4.3] The phrase 'difficult-to-classify images (high VIPP)' contradicts the definition of VIPP as the ground-truth prediction probability; high VIPP images are easy to classify and difficult to attack. Please rephrase.
  2. [§6.2, Fig. 6] The caption and text do not define what the error bars represent (standard deviation over the five sets? range?). Please define them.
  3. [§7] The statement that targeted ASR 'follows the same trends' is unsupported because no figure or table is provided; either include the plot or mark this as an informal observation.
  4. [References] The Moosavi-Dezfooli et al. entry is missing the publication year and venue.
  5. [§4.1] The 'same eight arbitrarily chosen coordinates' from Goh et al. are not specified; please provide the actual coordinates or a pointer to the exact source.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the similarity-based attack gain is a measured behavioral result, not a fitted or definitional prediction.

full rationale

The paper makes no derivation claim that would permit a circularity reduction. Its central result is an empirically measured attack success rate: images are assigned attack texts by cosine similarity in CLIP's embedding space, and ASR is measured by the model's top-1 label change. The similarity scores are computed from CLIP embeddings and are never fitted to the ASR values; the Rand baseline provides a genuine behavioral comparison, and the InstructBLIP experiment tests transfer to a different model using CLIP-computed similarities. Section 4's analyses (ATE, VIPP, text-image similarity) are descriptive correlations that motivate strategies, not fitted parameters renamed as predictions; the ATE-based strategies underperform, so any in-sample measurement of ATE is not load-bearing. The paper's acknowledged restriction to single-word ImageNet class labels and class-balanced target sets (Limitations; Section 6.1) is an external-validity limitation, not a circularity: the text set is an input constraint, and the reported 21% improvement remains a measured outcome under that constraint. No self-citation chain or imported uniqueness theorem supports the argument. Accordingly, no circular step is identified.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities and no fitted constants. Its results rest on domain assumptions about CLIP embedding geometry, the gatekeeper threat model, the representativeness of ImageNet labels as attack texts, and the transferability of CLIP similarity to InstructBLIP. The weakest of these is the transfer assumption, given the binary-choice evaluation.

assumptions (4)
  • domain assumption Cosine similarity between a CLIP image embedding and a CLIP text embedding predicts how likely that text is to cause misclassification of that image.
    The entire text-image similarity strategy is built on this correlation, motivated in Section 2.4 and measured in Section 4.4 on the same model.
  • domain assumption A gatekeeper detects attacks primarily through repetition of attack texts across images.
    Section 2.3 motivates the multi-image setting, but no gatekeeper experiment or user study is run to validate that non-repeating texts are actually stealthier.
  • domain assumption One-word ImageNet class labels are a representative attack text vocabulary.
    Section 4.1 restricts attack texts to 579 one-word labels; the Limitations section acknowledges that real-world texts are more varied.
  • domain assumption InstructBLIP can be attacked using similarity scores computed by CLIP because its vision encoder is CLIP-like.
    Section 8 justifies the greybox transfer; the binary-choice prompt weakens this assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Typographic Attacks in a Multi-Image Setting." pith.science (2026). https://pith.science/paper/AN6AUPFE

@misc{pith2026250208193,
  author       = {Pith},
  title        = {Pith review of: Typographic Attacks in a Multi-Image Setting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AN6AUPFE}},
  note         = {Machine review of arXiv:2502.08193}
}
read the original abstract

Large Vision-Language Models (LVLMs) are susceptible to typographic attacks, which are misclassifications caused by an attack text that is added to an image. In this paper, we introduce a multi-image setting for studying typographic attacks, broadening the current emphasis of the literature on attacking individual images. Specifically, our focus is on attacking image sets without repeating the attack query. Such non-repeating attacks are stealthier, as they are more likely to evade a gatekeeper than attacks that repeat the same attack text. We introduce two attack strategies for the multi-image setting, leveraging the difficulty of the target image, the strength of the attack text, and text-image similarity. Our text-image similarity approach improves attack success rates by 21% over random, non-specific methods on the CLIP model using ImageNet while maintaining stealth in a multi-image scenario. An additional experiment demonstrates transferability, i.e., text-image similarity calculated using CLIP transfers when attacking InstructBLIP.

Figures

Figures reproduced from arXiv: 2502.08193 by the authors.

Figure 1
Figure 1. In real-world attack scenarios, an attacker [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Attack success rates of our 579 attack texts. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Attack success rates versus text-image simi [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: One-to-one matching between the image and [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Comparisons of the five strategies in the multi [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Untargeted attack success rates versus the [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 22 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Hiroki Azuma and Yusuke Matsui. 2023. Defense-prefix for preventing typographic attacks on clip. In Proceedings of the IEEE International Conference on Computer Vision, Workshop on Adversarial Robustness in the Real World, pages 3644--3653

    work page 2023

  4. [4]

    Marius Zöllner

    Daniel Bogdoll, Enrico Eisen, Maximilian Nitsche, Christin Scheib, and J. Marius Zöllner. 2022. Multimodal detection of unknown objects on roads for autonomous driving. In Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics

    work page 2022

  5. [5]

    Hao Cheng, Erjia Xiao, and Renjing Xu. 2024. Unveiling typographic deceptions: Insights of the typographic vulnerability in large vision-language models. In Proceedings of the European Conference on Computer Vision

    work page 2024

  6. [6]

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. 2023. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    work page 2023

  7. [7]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. Instruct BLIP : Towards general-purpose vision-language models with instruction tuning. In Advances in Neural Information Processing Systems, volume 36, page 49250–49267

    work page 2023

  8. [8]

    Gabriel Goh, Nick Cammarata †, Chelsea Voss †, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. 2021. https://doi.org/10.23915/distill.00030 Multimodal neurons in artificial neural networks . Distill. Https://distill.pub/2021/multimodal-neurons

    doi:10.23915/distill.00030 2021

Show all 26 references
  1. [9]

    Felipe González-Pizarro and Savvas Zannettou. 2023. Understanding and detecting hateful content using contrastive learning. In Proceedings of the International AAAI Conference on Web and Social Media

    2023

  2. [10]

    Yiqi Lin, Conghui He, Alex Jinpeng Wang, Bin Wang, Weijia Li, and Mike Zheng Shou. 2024. Parrot captions teach CLIP to spot text. In Proceedings of the European Conference on Computer Vision

    2024

  3. [11]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024 a . Visual instruction tuning. In Advances in Neural Information Processing Systems, volume 36, page 34892–34916

    2024

  4. [12]

    Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. 2024 b . Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 67(12)

    2024

  5. [13]

    Universal adversarial perturbations

    Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition

  6. [14]

    Nina Narodytska and Shiva Prasad Kasiviswanathan. 2017. Simple black-box adversarial attacks on deep neural networks. In Proceedings of the Conference on Computer Vision and Pattern Recognition Workshops

    2017

  7. [15]

    David A Noever and Samantha E Miller Noever. 2021. Reading isn't believing: Adversarial attacks on multi-modal neurons. arXiv preprint arXiv:2103.10480

    2021 arXiv

  8. [16]

    Utku Ozbulak, Esla Timothy Anzaku, Wesley De Neve, and Arnout Van Messem. 2021. Selection of source images heavily influences the effectiveness of adversarial attacks. In Proceedings of the British Machine Vision Conference

    2021

  9. [17]

    Maan Qraitem, Nazia Tasnim, Kate Saenko, and Bryan A Plummer. 2024. Vision-llms can fool themselves with self-generated typographic attacks. In Advances in Neural Information Processing Systems, Workshop on Multimodal Algorithmic Reasoning

    2024

  10. [18]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the International Conferenc...

    2021

  11. [19]

    Berg, and Li Fei-Fei

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge . International Journal of Computer Visi...

    2015

  12. [20]

    Pedro Sandoval-Segura, Vasu Singla, Liam Fowl, Jonas Geiping, Micah Goldblum, David Jacobs, and Tom Goldstein. 2022 a . Poisons that are learned faster are more effective. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Workshop on Art of Robustness

    2022

  13. [21]

    Pedro Sandoval-Segura, Vasu Singla, Jonas Geiping, Micah Goldblum, and Tom Goldstein. 2023. What can we learn from unlearnable datasets? In Advances in Neural Information Processing Systems, volume 36, page 75372–75391

    2023

  14. [22]

    Pedro Sandoval-Segura, Vasu Singla, Jonas Geiping, Micah Goldblum, Tom Goldstein, and David Jacobs. 2022 b . Autoregressive perturbations for data poisoning. In Advances in Neural Information Processing Systems, volume 35, page 27374–27386

    2022

  15. [23]

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. 2022. LAION-5B : An open large-scale dataset for training next generation image-text models. In Advances in Neur...

    2022

  16. [24]

    Pedro Tabacof and Eduardo Valle. 2016. Exploring the space of adversarial images. In Proceedings of the International Joint Conference on Neural Networks

    2016

  17. [25]

    Sahar Tahmasebi, Sherzod Hakimov, Ralph Ewerth, and Eric M\" u ller-Budack. 2023. Improving generalization for multimodal fake news detection. In Proceedings of the ACM International Conference on Multimedia Retrieval

    2023

  18. [26]

    Haiyan Zhang, Zheng Ding, Md Sharid Kayes Dipu, Pinrong Lv, Yuxue Huang, Hauwa Suleiman Abdullahi, Ao Zhang, Zhaoyu Song, and Yuanyuan Wang. 2024. Identification of illegal outdoor advertisements based on CLIP fine-tuning and OCR technology. IEEE Access, 12:92976--92987

    2024

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.