Pith. sign in

REVIEW 4 major objections 5 minor 63 references

Not Just Text: Uncovering Vision Modality Typographic Threats in Image Generation Models

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adding printed words to an input image can steer AI image generation toward harmful or biased content, and current text-only defenses miss it.

desk verdict Solid transfer of typographic attacks to diffusion-based image generation with a useful benchmark, but the quantitative core leans on a self-referential CLIP Score and needs human validation. read the letter →

arxiv 2412.05538 v2 pith:5C2QBM3Q submitted 2024-12-07 cs.CV cs.PF

Hao Cheng , Erjia Xiao , Jiayan Yang , Jiahang Cao , Qiang Zhang , Jize Zhang , Kaidi Xu , Jindong Gu
show 1 more author
Renjing Xu
This is my paper · ORCID
classification cs.CVcs.PF
keywords typographicattackvisionmodalitythreatimage-to-imagegenerationCLIP-guideddiffusionmodelspromptdefenseimagesecurityVMT-IGMsdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that image-generation models are vulnerable to typographic attacks through the image input, not just the text prompt. In image-to-image tasks, overlaying a word such as “naked” or “African” on the input image shifts the generated output toward that concept across UnCLIP, SD1.5, SDXL, and FLUX. The paper further argues that existing safety mechanisms—prompt filters, blacklists, latent-space guards, post-generation NSFW checkers, and even an “ignore text” prompt prefix—do not prevent this vision-modality threat. The authors introduce the VMT-IGMs dataset as a baseline for measuring such vulnerabilities. If correct, the finding means real-world image editing pipelines have an uncompensated attack surface.

What carries the argument

The load-bearing mechanism is the typographic attack: overlaying printed text on an input image so that the CLIP vision encoder reads the text as part of the image semantics and encodes it into the conditioning feature $f = \mathrm{CLIP}(x, p)$ that guides the diffusion reverse process. Because the text and image are fused before generation, the inserted word acts as an additional semantic instruction that the model follows even though it never appears in the user prompt. The quantitative instrument is CLIP Score, computed between the generated image and the inserted typo word; larger positive deltas over clean-input baselines are read as stronger attack success. The evaluation scaffold is the VMT-IGMs dataset, built by overlaying controlled typography on 1000 CelebA-HQ faces, with factor-modification and malicious-threat subtypes.

What would settle it

Have human annotators label whether images generated from typographic inputs actually contain the target concept, and compare those labels with the CLIP Score deltas; if the deltas rise while the labels show no concept appearing, the claim that typography steers generated content is not supported.

Watch

Extended reading notes

Core claim

The central discovery is that inserting typographic text directly into the input image of an image-to-image task redirects the semantics of the generated image even when the text prompt is entirely benign. Using CLIP Score between the generated image and the inserted word as the measure, the paper reports that visible typography raises alignment by roughly four to seven points on average across models for nouns, with harmful words (“naked”, “bloody”), bias words (“African”, “Asian”), and neutral words (“hat”, “Muslim”) all producing consistent increases. Invisible typography, rendered in near-black on the black border of the input, retains a smaller but still positive effect. The paper also reports that five pre-generation guard methods and the post-generation safety checker fail to flag the attack because they inspect only the language prompt or NSFW content, and that prepending “ignore text” to the prompt does not reduce the typography's influence. The authors conclude that the vision modality is an exposed and currently undefended attack surface for CLIP-guided diffusion models.

Load-bearing premise

The quantitative evidence rests on assuming that a higher CLIP Score between the generated image and the inserted word means the word actually changed the image's meaning, but the attacked models and the metric share the same CLIP encoder and the paper does not validate this with human judgments.

Editorial extensions

If this is right

  • User-supplied images in editing, style-transfer, and conditional-generation applications become a viable vector for injecting unintended, harmful, or biased content into the generated output.
  • Current text-prompt guards—blacklists, toxicity classifiers, CLIP-score screening, LLM-based checks, latent-space guards, and the post-generation safety checker—cannot catch the attack because the offensive material lives in the image, not the prompt.
  • Prompt-level mitigations such as instructing the model to “ignore text” do not remove the typography's influence in CLIP-guided diffusion models.
  • Near-invisible typography still biases the output, so an attacker can alter generated content without visibly changing the source photograph.
  • VMT-IGMs provides a shared benchmark with both factor-modification and malicious-threat subtypes for future vision-modality defenses.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the attack travels through the CLIP image encoder, any downstream model that conditions on CLIP image embeddings, including newer multimodal generators, is a plausible target; this can be tested by repeating the typographic-overlay procedure on those models.
  • The reported CLIP Score gains may overstate human-perceived semantic change, since the attacked model and the metric share the same encoder; a human-rating study of the generated images would separate embedding alignment from content actually visible to viewers.
  • A defense direction implied by the mechanism is to sanitize or certify the image input itself—for instance, detecting and removing embedded text before generation—rather than filtering the text prompt.
  • The same mechanism could be triggered by text already present in ordinary photographs, such as signs, watermarks, or captions, making the vulnerability relevant beyond deliberately crafted attack images.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces the Vision Modal Threats in Image Generation Models (VMT-IGMs) dataset and demonstrates typographic attacks in the vision modality for CLIP-guided image generation models in image-to-image tasks. It evaluates four models (UnCLIP, SD1.5, SDXL, FLUX) using CLIP Score and FID across factor-modification and malicious-threat settings, tests five text-prompt-based defenses plus a post-generation safety checker, and proposes the dataset as a benchmark for future defenses.

Significance. If validated, the paper identifies a real and previously under-explored attack surface in mainstream image editing pipelines and provides a reusable benchmark. The qualitative examples in Figures 1-2 and Appendix A provide direct evidence that inserting typography into input images can shift generated content, and the release of code and dataset is a practical contribution. However, the primary quantitative metric (CLIP Score) is self-referential, no human validation or error bars are provided, the defense evaluation covers only text-prompt guards, and the FID methodology is underspecified. These issues currently limit the strength of the benchmark claims.

major comments (4)
  1. [§4.1 (Metrics)] The CLIP Score is computed between the generated image and the inserted typo word using the same CLIP encoder that conditions the attacked models (UnCLIP and IP-Adapter). This makes the metric self-referential: a higher score can arise simply because the generated image retains the rendered typo text, or because the embedding shift is perceptually meaningless. The paper does not provide human judgments, OCR checks, or an independent concept detector, so the deltas in Tables 2 and 3 do not by themselves establish that the model depicts the intended harmful or biased content.
  2. [Tables 2-3 (§4.2-§4.4)] All reported CLIP Scores are single values without standard deviations, confidence intervals, or significance tests. This is particularly problematic for the invisible typography results, where average deltas are small (0.2-1.0) and some individual entries are negative (e.g., Table 3, FLUX "naked" at -0.44, or SD1.5 "Muslim" at +0.21). Without per-image statistics or significance testing, the claim of a persistent effect from invisible typography is not supported.
  3. [Appendix C (FID)] The FID methodology is underspecified. The tables appear to measure FID between generated outputs and their corresponding original clean input images, but the "clean" column already yields large FID values (42-58 for the first prompt), indicating that the style-transfer prompt itself causes substantial distribution shift. The incremental FID increase attributed to typography is not shown to correspond to semantic harm (as opposed to general image distortion), and details on sample size, resolution, and the Inception feature layer are missing.
  4. [§4.5 (Defenses)] The defense evaluation covers only text-prompt guards (Text Blacklist, Detoxify, CLIP-Score-based filtering, LLM-based detection, Latent Guard) and a post-generation NSFW safety checker. It does not test any defense designed for the vision modality, such as image-based typography detection or input image sanitization. Consequently, the conclusion that "current mainstream guarding methods ... are ineffective" is broader than the evidence supports.
minor comments (5)
  1. [§3.1] The "invisible" typography is rendered in RGB (15,15,15) within black borders (0,0,0). Please clarify whether common preprocessing steps such as resizing, cropping, or border removal could eliminate these pixels, and how the models receive the border text in practice.
  2. [§4.1] Reference [47] for CLIP Score appears to be the same paper as reference [46]; the citations should be merged or clearly distinguished.
  3. [Algorithm 1] The notation is inconsistent: ft is used for both image latents and CLIP features, and the reverse transition is written as pθ(ft−1|xt) while the update formula uses ft. Please harmonize the notation throughout the algorithm and the surrounding text.
  4. [Table 4] The safety checker blocks only 16.6% of "naked" images on average, so describing it as "effective" for harmful content is misleading; the text should be phrased in terms of relative detection rates rather than effectiveness.
  5. [Appendix C] Some FID values are anomalously large and inconsistent with the pattern of smaller effects for invisible typography, e.g., Table 8 shows FLUX invisible "naked" with an FID increase of 93.25 while visible "naked" increases by only 0.04. Please investigate or explain these outliers.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the evaluation is empirical and self-contained; the shared CLIP encoder is a construct-validity concern, not a definitional reduction.

full rationale

The paper's chain is purely empirical: insert typographic text into an input image, run a CLIP-guided diffusion model, and measure the output with CLIP Score, FID, and qualitative examples. No parameter is fitted from a subset of the data and then renamed as a prediction, and no uniqueness theorem or ansatz is imported from the authors' prior work. The CLIP Score metric is a standard external similarity measure computed on the generated output against the typo word; although the attacked models and the metric share the CLIP encoder, the score increase is an observed outcome of the generative pipeline rather than an identity, because the diffusion model could in principle ignore the perturbed conditioning. The possibility that generated images retain rendered typo text and thereby inflate CLIP Score is a measurement-validity limitation, not a circular reduction, and the paper additionally reports FID deviations and shows qualitative examples. The self-citation to the authors' typographic-attack work [8] is motivational and not load-bearing, since the attack concept is also attributed to [3] and the 'ignore text' prompt is tested experimentally rather than assumed. Hence no significant circularity is present.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no free parameters or invented entities. It relies on domain assumptions about CLIP's text-reading ability and the validity of its evaluation metrics.

assumptions (4)
  • domain assumption CLIP vision encoder reads text rendered inside images, so typography alters the image embedding.
    Central to the attack mechanism; established in prior work [3,8] but assumed here (Section 2.2).
  • domain assumption CLIP Score between the generated image and the typo word is a valid measure of semantic impact.
    Primary metric in Tables 2 and 3; no human validation is provided, and the metric shares parameters with the attacked models.
  • domain assumption Scores averaged over the dataset without error bars are sufficient to support the conclusions.
    All results are reported as point averages; no variance or significance is given (Tables 2, 3).
  • domain assumption CelebA-HQ face images are representative of real-world image-to-image editing inputs.
    Dataset is limited to 1000 face images; generalization to other domains is assumed (Section 3).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Not Just Text: Uncovering Vision Modality Typographic Threats in Image Generation Models." pith.science (2026). https://pith.science/paper/5C2QBM3Q

@misc{pith2026241205538,
  author       = {Pith},
  title        = {Pith review of: Not Just Text: Uncovering Vision Modality Typographic Threats in Image Generation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5C2QBM3Q}},
  note         = {Machine review of arXiv:2412.05538}
}
read the original abstract

Current image generation models can effortlessly produce high-quality, highly realistic images, but this also increases the risk of misuse. In various Text-to-Image or Image-to-Image tasks, attackers can generate a series of images containing inappropriate content by simply editing the language modality input. To mitigate this security concern, numerous guarding or defensive strategies have been proposed, with a particular emphasis on safeguarding language modality. However, in practical applications, threats in the vision modality, particularly in tasks involving the editing of real-world images, present heightened security risks as they can easily infringe upon the rights of the image owner. Therefore, this paper employs a method named typographic attack to reveal that various image generation models are also susceptible to threats within the vision modality. Furthermore, we also evaluate the defense performance of various existing methods when facing threats in the vision modality and uncover their ineffectiveness. Finally, we propose the Vision Modal Threats in Image Generation Models (VMT-IGMs) dataset, which would serve as a baseline for evaluating the vision modality vulnerability of various image generation models.

Figures

Figures reproduced from arXiv: 2412.05538 by the authors.

Figure 1
Figure 1. Inserting typography into input images can manipulate [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Image generation examples based on input images with typography related to harmful, bias, and neutral concepts. (Text prompt: [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Examples of typography with different typographic factors (size, quantity, opacity, and position of typos) within input images. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: (left) an input image with visible typography. (right) an [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: The effectiveness of various prompt detection methods [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: The semantic impact of typography (typo) with prompts [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 8
Figure 8. Figure 8: Image generation examples based on typographic input [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 7
Figure 7. Figure 7: The semantic impact of typography with different ty [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 9
Figure 9. Figure 9: Image generation examples based on input images with typography related to harmful, bias, and neutral concepts. (Text prompt: [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Image generation examples based on input images with typography related to harmful, bias, and neutral concepts. (Text prompt: [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

63 extracted references · 31 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    Elijah: Eliminating backdoors injected in diffusion models via distribution shift

    Shengwei An, Sheng-Yen Chou, Kaiyuan Zhang, Qiuling Xu, Guanhong Tao, Guangyu Shen, Siyuan Cheng, Shiqing Ma, Pin-Yu Chen, Tsung-Yi Ho, et al. Elijah: Eliminating backdoors injected in diffusion models via distribution shift. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 10847–10855, 2024. 3

    work page 2024

  3. [3]

    Defense-prefix for pre- venting typographic attacks on clip

    Hiroki Azuma and Yusuke Matsui. Defense-prefix for pre- venting typographic attacks on clip. ICCV Workshop on Ad- versarial Robustness In the Real World, 2023. 2, 3

    work page 2023

  4. [4]

    In- structpix2pix: Learning to follow image editing instructions

    Tim Brooks, Aleksander Holynski, and Alexei A Efros. In- structpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392–18402, 2023. 1

    work page 2023

  5. [5]

    Controllable generation with text-to-image diffusion models: A survey

    Pu Cao, Feng Zhou, Qing Song, and Lu Yang. Controllable generation with text-to-image diffusion models: A survey. arXiv preprint arXiv:2403.04279, 2024. 1

    arXiv 2024

  6. [6]

    Trojdiff: Trojan at- tacks on diffusion models with diverse targets

    Weixin Chen, Dawn Song, and Bo Li. Trojdiff: Trojan at- tacks on diffusion models with diverse targets. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4035–4044, 2023. 3

    2023

  7. [7]

    Rbformer: improve adversarial robustness of trans- former by robust bias

    Hao Cheng, Jinhao Duan, Hui Li, Lyutianyang Zhang, Ji- ahang Cao, Ping Wang, Jize Zhang, Kaidi Xu, and Ren- jing Xu. Rbformer: improve adversarial robustness of trans- former by robust bias. The British Machine Vision Confer- ence (BMVC), 2023. 3

    work page 2023

  8. [8]

    Un- veiling typographic deceptions: Insights of the typographic vulnerability in large vision-language model.European Con- ference on Computer Vision (ECCV), 2024

    Hao Cheng, Erjia Xiao, Jindong Gu, Le Yang, Jinhao Duan, Jize Zhang, Jiahang Cao, Kaidi Xu, and Renjing Xu. Un- veiling typographic deceptions: Insights of the typographic vulnerability in large vision-language model.European Con- ference on Computer Vision (ECCV), 2024. 2, 3, 8

    work page 2024

Show all 63 references
  1. [9]

    Villan- diffusion: A unified backdoor attack framework for diffu- sion models

    Sheng-Yen Chou, Pin-Yu Chen, and Tsung-Yi Ho. Villan- diffusion: A unified backdoor attack framework for diffu- sion models. Advances in Neural Information Processing Systems, 36, 2024. 3

    2024

  2. [10]

    Style injec- tion in diffusion: A training-free approach for adapting large- scale diffusion models for style transfer

    Jiwoo Chung, Sangeek Hyun, and Jae-Pil Heo. Style injec- tion in diffusion: A training-free approach for adapting large- scale diffusion models for style transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8795–8805, 2024. 1

    2024

  3. [11]

    Instructblip: Towards general- purpose vision-language models with instruction tuning,

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning,

  4. [12]

    Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language mod- els

    Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language mod- els. In Proceedings of the 62nd Annual Meeting of the ...

    2024

  5. [13]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In Forty-first International Conference on Mach...

    2024

  6. [14]

    Rt-attack: Jailbreak- ing text-to-image models via random token

    Sensen Gao, Xiaojun Jia, Yihao Huang, Ranjie Duan, Jin- dong Gu, Yang Liu, and Qing Guo. Rt-attack: Jailbreak- ing text-to-image models via random token. arXiv preprint arXiv:2408.13896, 2024. 3

    2024 arXiv

  7. [15]

    Generative adversarial networks

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Commu- nications of the ACM, 63(11):139–144, 2020. 2

    2020

  8. [16]

    Responsible generative ai: What to generate and what not

    Jindong Gu. Responsible generative ai: What to generate and what not. arXiv preprint arXiv:2404.05783, 2024. 3

    2024 arXiv

  9. [17]

    Detoxify

    Laura Hanu and Unitary team. Detoxify. Github. https://github.com/unitaryai/detoxify, 2020. 2, 3, 7

    2020

  10. [18]

    Defending against backdoor attack on deep neural networks

    Cheng Hao, Kaidi Xu, Sijia Liu, Pin-Yu Chen, Pu Zhao, and Xue Lin. Defending against backdoor attack on deep neural networks. arXiv preprint arXiv:2002.12162, 2020. 3

    2002 arXiv

  11. [19]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. Advances in neural information processing systems , 30, 2017. 3, 5

    2017

  12. [20]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 1, 3

    2020

  13. [21]

    All but one: Surgical concept erasing with model preservation in text-to- image diffusion models

    Seunghoo Hong, Juhun Lee, and Simon S Woo. All but one: Surgical concept erasing with model preservation in text-to- image diffusion models. In Proceedings of the AAAI Con- ference on Artificial Intelligence, pages 21143–21151, 2024. 3

    2024

  14. [22]

    Llama guard: Llm- based input-output safeguard for human-ai conversations

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. Llama guard: Llm- based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674, 2023. 1, 7

    2023 arXiv

  15. [23]

    Progressive growing of gans for improved qual- ity, stability, and variation.arXiv preprint arXiv:1710.10196,

    Tero Karras. Progressive growing of gans for improved qual- ity, stability, and variation.arXiv preprint arXiv:1710.10196,

    arXiv

  16. [24]

    Auto-encoding variational bayes

    Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013. 2, 8, 1

    2013 arXiv

  17. [25]

    Self-discovering interpretable diffusion latent di- rections for responsible text-to-image generation

    Hang Li, Chengzhi Shen, Philip Torr, V olker Tresp, and Jin- dong Gu. Self-discovering interpretable diffusion latent di- rections for responsible text-to-image generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12006–1201...

    2024

  18. [26]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023. 1, 3

    2023 arXiv

  19. [27]

    Spd-ddpm: Denoising diffu- sion probabilistic models in the symmetric positive definite space

    Yunchen Li, Zhou Yu, Gaoqi He, Yunhang Shen, Ke Li, Xing Sun, and Shaohui Lin. Spd-ddpm: Denoising diffu- sion probabilistic models in the symmetric positive definite space. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 13709–13717, 2024. 3

    2024

  20. [28]

    Adversarial example does good: Preventing paint- ing imitation from diffusion models via adversarial exam- ples

    Chumeng Liang, Xiaoyu Wu, Yang Hua, Jiaru Zhang, Yim- ing Xue, Tao Song, Zhengui Xue, Ruhui Ma, and Haibing Guan. Adversarial example does good: Preventing paint- ing imitation from diffusion models via adversarial exam- ples. arXiv preprint arXiv:2302.04578, 2023. 3

    2023 arXiv

  21. [29]

    Which model generated this image? a model- agnostic approach for origin attribution

    Fengyuan Liu, Haochen Luo, Yiming Li, Philip Torr, and Jindong Gu. Which model generated this image? a model- agnostic approach for origin attribution. In European Con- ference on Computer Vision, pages 282–301. Springer, 2024. 2

    2024

  22. [30]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023. 1, 3

    2023 arXiv

  23. [31]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08485,

    arXiv

  24. [32]

    Latent guard: a safety frame- work for text-to-image generation

    Runtao Liu, Ashkan Khakzar, Jindong Gu, Qifeng Chen, Philip Torr, and Fabio Pizzati. Latent guard: a safety frame- work for text-to-image generation. In European Conference on Computer Vision, pages 93–109. Springer, 2025. 2, 3, 7

    2025

  25. [33]

    Multimodal prag- matic jailbreak on text-to-image models

    Tong Liu, Zhixin Lai, Gengyuan Zhang, Philip Torr, Vera Demberg, V olker Tresp, and Jindong Gu. Multimodal prag- matic jailbreak on text-to-image models. European Confer- ence on Computer Vision (ECCV), 2024. 3

    2024

  26. [34]

    Query-relevant images jailbreak large multi-modal models

    Xin Liu, Yichen Zhu, Yunshi Lan, Chao Yang, and Yu Qiao. Query-relevant images jailbreak large multi-modal models. arXiv preprint arXiv:2311.17600, 2023. 3

    2023 arXiv

  27. [35]

    Large-scale celebfaces attributes (celeba) dataset

    Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August, 15(2018):11, 2018. 4, 5

    2018

  28. [36]

    Information constraints on auto-encoding variational bayes

    Romain Lopez, Jeffrey Regier, Michael I Jordan, and Nir Yosef. Information constraints on auto-encoding variational bayes. Advances in neural information processing systems , 31, 2018. 3

    2018

  29. [37]

    An image is worth 1000 lies: Transferability of adversarial images across prompts on vision-language models

    Haochen Luo, Jindong Gu, Fengyuan Liu, and Philip Torr. An image is worth 1000 lies: Transferability of adversarial images across prompts on vision-language models. In The Twelfth International Conference on Learning Representa- tions. 3

  30. [38]

    Jailbreaking prompt attack: A controllable adversarial attack against diffusion models

    Jiachen Ma, Anda Cao, Zhiqing Xiao, Yijiang Li, Jie Zhang, Chao Ye, and Junbo Zhao. Jailbreaking prompt attack: A controllable adversarial attack against diffusion models. arXiv preprint arXiv:2404.02928, 2024. 3

    2024 arXiv

  31. [39]

    A holistic approach to undesired content detection in the real world

    Todor Markov, Chong Zhang, Sandhini Agarwal, Floren- tine Eloundou Nekoul, Theodore Lee, Steven Adler, Angela Jiang, and Lilian Weng. A holistic approach to undesired content detection in the real world. In Proceedings of the AAAI Conference on Artificial Intelligence , pages ...

    2023

  32. [40]

    Dreamguider: Improved training free diffusion-based conditional generation

    Nithin Gopalakrishnan Nair and Vishal M Patel. Dreamguider: Improved training free diffusion-based conditional generation. arXiv preprint arXiv:2406.02549 ,

    arXiv

  33. [41]

    At-ddpm: Restoring faces degraded by atmospheric tur- bulence using denoising diffusion probabilistic models

    Nithin Gopalakrishnan Nair, Kangfu Mei, and Vishal M Pa- tel. At-ddpm: Restoring faces degraded by atmospheric tur- bulence using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Winter Conference on Appli- cations of Computer Vision, pages 3434–3443, 2023. 3

    2023

  34. [42]

    Contrastive denoising score for text-guided latent diffusion image editing

    Hyelin Nam, Gihyun Kwon, Geon Yeong Park, and Jong Chul Ye. Contrastive denoising score for text-guided latent diffusion image editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9192–9201, 2024. 1

    2024

  35. [43]

    White-box membership inference attacks against diffusion models

    Yan Pang, Tianhao Wang, Xuhui Kang, Mengdi Huai, and Yang Zhang. White-box membership inference attacks against diffusion models. arXiv preprint arXiv:2308.06405,

    arXiv

  36. [44]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 5

    2023 arXiv

  37. [45]

    Safe-clip: Removing nsfw concepts from vision-and-language models

    Samuele Poppi, Tobia Poppi, Federico Cocchi, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara, et al. Safe-clip: Removing nsfw concepts from vision-and-language models. In Proceedings of the European Conference on Computer Vi- sion, 2024. 2, 3

    2024

  38. [46]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of th...

    2021

  39. [47]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  40. [48]

    Hierarchical text-conditional image gener- ation with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image gener- ation with clip latents. arXiv preprint arXiv:2204.06125, 1 (2):3, 2022. 3, 5

    2022 arXiv

  41. [49]

    Red-teaming the stable diffusion safety filter

    Javier Rando, Daniel Paleka, David Lindner, Lennart Heim, and Florian Tram`er. Red-teaming the stable diffusion safety filter. arXiv preprint arXiv:2210.04610, 2022. 2, 3, 7, 8

    2022 arXiv

  42. [50]

    Gener- ating diverse high-fidelity images with vq-vae-2

    Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Gener- ating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019. 3

    2019

  43. [51]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 5

    2022

  44. [52]

    Raising the cost of malicious ai-powered image editing

    Hadi Salman, Alaa Khaddaj, Guillaume Leclerc, Andrew Ilyas, and Aleksander Madry. Raising the cost of malicious ai-powered image editing. arXiv preprint arXiv:2302.06588,

    arXiv

  45. [53]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International confer- ence on machine learning, pages 2256–2265. PMLR, 2015. 3

    2015

  46. [54]

    Emu: Generative pretraining in multimodality

    Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. In The Twelfth International Conference on Learning Representations, 2023. 1, 3

    2023

  47. [55]

    Generative multimodal mod- els are in-context learners

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiy- ing Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal mod- els are in-context learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , ...

    2024

  48. [56]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 1

    2023 arXiv

  49. [57]

    Gcd-ddpm: A generative change detection model based on difference-feature guided ddpm

    Yihan Wen, Xianping Ma, Xiaokang Zhang, and Man-On Pun. Gcd-ddpm: A generative change detection model based on difference-feature guided ddpm. IEEE Transactions on Geoscience and Remote Sensing, 2024. 3

    2024

  50. [58]

    Nullu: Mitigating object hallucinations in large vision-language models via halluspace projection

    Le Yang, Ziwei Zheng, Boxu Chen, Zhengyu Zhao, Chenhao Lin, and Chao Shen. Nullu: Mitigating object hallucinations in large vision-language models via halluspace projection. arXiv preprint arXiv:2412.13817, 2024. 3

    2024 arXiv

  51. [59]

    Sneakyprompt: Jailbreaking text-to-image generative models

    Yuchen Yang, Bo Hui, Haolin Yuan, Neil Gong, and Yinzhi Cao. Sneakyprompt: Jailbreaking text-to-image generative models. In 2024 IEEE symposium on security and privacy (SP), pages 897–912. IEEE, 2024. 3

    2024

  52. [60]

    Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models

    Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models. arXiv preprint arXiv:2308.06721,

    arXiv

  53. [61]

    Inversion-based style transfer with diffusion models

    Yuxin Zhang, Nisha Huang, Fan Tang, Haibin Huang, Chongyang Ma, Weiming Dong, and Changsheng Xu. Inversion-based style transfer with diffusion models. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10146–10156, 2023. 1

    2023

  54. [62]

    Defensive unlearning with adversarial training for robust concept erasure in diffusion models

    Yimeng Zhang, Xin Chen, Jinghan Jia, Yihua Zhang, Chongyu Fan, Jiancheng Liu, Mingyi Hong, Ke Ding, and Sijia Liu. Defensive unlearning with adversarial training for robust concept erasure in diffusion models. Conference on Neural Information Processing Systems, 2024. 3

    2024

  55. [63]

    analog film photo, faded film, desaturated, 35mm photo

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 1, 3 Not Just Text: Uncovering Vision Modality Typographic Threats in Image Gen...

    2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.