Pith. sign in

REVIEW 5 major objections 4 minor 2 cited by

Backdoor Attack on Vision Language Models with Stealthy Semantic Manipulation

T0 review · 5 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that cross-modal semantic mismatches—ordinary contradictions between an image and its question—can serve as stealthy, context-dependent backdoor triggers for vision-language models, implanted purely through data poisoning.

desk verdict A genuinely new semantic-mismatch backdoor surface, well-tested, but the headline ASR overstates per-trigger reliability and the paper needs Full ASR in the main tables before the 98% claim is fully supported. read the letter →

arxiv 2506.07214 v1 pith:PMKKVE2R submitted 2025-06-08 cs.CV cs.CR

Zhiyuan Zhong , Zhen Sun , Yepang Liu , Xinlei He , Guanhong Tao This is my paper
classification cs.CVcs.CR
keywords backdoorattackvisionlanguagemodelsdatapoisoningsemanticmismatchcross-modaltriggervisualquestionansweringstealthymultimodalsecurity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a data-poisoning attacker can implant a practical, stealthy backdoor in vision-language models without touching model internals, by training on image-text pairs whose semantics deliberately contradict each other. The trigger is not a fixed patch or token but the presence of a cross-modal mismatch, such as the word "red" paired with a white curtain. Across four widely used VLMs and two visual question answering datasets, the attack is reported to exceed 95% attack success in most configurations and near 100% for textual mismatches, while clean accuracy stays close to the clean baseline. The backdoor also generalizes to out-of-distribution data and, in several settings, transfers across poisoning modalities, and two lightweight defenses fail to remove it. If true, this widens the backdoor threat model for multimodal systems to ordinary semantic contradictions in real user traffic.

What carries the argument

The load-bearing object is the semantic-consistency predicate: an image-question pair is declared inconsistent when an auxiliary VLM answers "No" to an existence query asking whether the injected color or object is present. Poisoned samples are constructed by substituting alternative elements into questions (textual modality) or by recoloring or replacing objects in images with segmentation and image-editing tools (visual modality), and three VLMs vote by majority to keep only candidates that at least two models agree are absent. Fine-tuning with LoRA on a mix of clean and poisoned instruction data encodes the rule that a mismatched input maps to the target output while a matched input maps to normal behavior.

What would settle it

Run the same SIMBad poisoning under full fine-tuning instead of LoRA, or with a data-cleaning step that removes image-question pairs where the question attributes an object or color that the image does not contain; if attack success falls to near the clean baseline while clean accuracy is preserved, the claim that semantic mismatches form a robust backdoor surface would be falsified for those realistic pipelines.

Watch

Extended reading notes

Core claim

The paper's central claim is that semantic inconsistency between the image and the text is itself a learnable, context-dependent backdoor trigger. BadSem builds a dataset, SIMBad, of aligned and misaligned pairs on color and object semantics; from each clean pair it generates variants in which either the question attributes a nonexistent attribute or object, or the image is edited so that the originally referenced element is gone. During instruction tuning, the model is taught to output a fixed target word exactly when such a mismatch appears, and the paper reports that four VLMs—LLaVA, Qwen2-VL (2B and 7B), and Llama-Vision—learn this with over 95% attack success in most settings and near 100% for textual mismatches, while maintaining clean semantic accuracy and low false-positive activation on aligned inputs. The backdoor also transfers across datasets and, for several object-based variants, across poisoning modalities, and attention visualization indicates that backdoored models redirect attention to the mismatched object under trigger conditions.

Load-bearing premise

The attack's practical reach depends on the victim using LoRA instruction tuning under conditions close to the paper's (5,000 clean samples, learning rate 1e-4, three epochs), and on users regularly producing inputs that are semantically mismatched with the image; if deployments use full fine-tuning, different data mixing, or aggressive data filtering, the reported success rates may not transfer.

Editorial extensions

If this is right

  • A data-poisoning attacker with no access to model weights or training objectives can implant a backdoor that remains effective at a 1% poisoning rate.
  • Backdoored models behave normally on semantically aligned inputs, so the trigger is context-dependent and difficult to spot by inspecting either modality alone.
  • The backdoor generalizes to out-of-distribution datasets and can be triggered through the opposite modality in several configurations.
  • System-prompt guardrails and small-scale supervised fine-tuning do not reliably remove the backdoor, with attack success often staying above 95%.
  • Attention analysis shows backdoored models concentrate on the mismatched object, suggesting a possible signature for detecting semantic backdoors.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Outside the paper's own experiments, this result implies that data curation pipelines for VLMs may need to screen for cross-modal contradictions, not just per-modality anomalies; a testable consequence is whether a filter that removes contradicting image-text pairs would reduce attack success.
  • Because the trigger is context-dependent rather than a fixed pattern, defenses based on searching for a single inverted trigger may miss it; a more promising direction, which the paper does not explore, is inference-time inconsistency detection.
  • The observed asymmetry in cross-modality and cross-semantics generalization hints that grounded object-level representations are the substrate of the learned trigger, so probing object-centric attention could yield both detection and defense strategies.
  • The paper's pipeline is limited to LoRA instruction tuning; testing the same poisoning under full fine-tuning or different data mixing would clarify how broadly the threat applies in real deployments.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes BadSem, a data-poisoning backdoor attack against vision-language models (VLMs) that uses cross-modal semantic mismatches—such as asking about a "red curtain" when the image shows a white curtain—as implicit triggers. The authors construct a dataset, SIMBad, by editing either the text question or the image to create color- and object-semantic inconsistencies, and they fine-tune four VLMs with LoRA on poisoned instruction-tuning data. Experiments on VQAv2 and GQA report high attack success rates (ASR), preserved clean accuracy, low false-positive rates on semantically consistent inputs, out-of-distribution transfer, cross-modality transfer, and failure of two lightweight defenses (system prompting and small-scale supervised fine-tuning).

Significance. If the findings hold, BadSem identifies a genuinely new attack surface: backdoor triggers that are semantic contradictions rather than fixed patches or tokens, activated only when image and text disagree. This is a plausible and practical threat for data-poisoning scenarios under parameter-efficient fine-tuning, and the paper includes a substantial empirical evaluation: four VLM families, two datasets, seven baselines, several ablations, and defense attempts. The attack is black-box with respect to the victim model's internals and does not modify the loss function, which strengthens its practical relevance. The paper's main weaknesses are metric reporting (Overall ASR versus per-trigger Full ASR), reliance on VLM judgments to label semantic inconsistency without human verification, and single-run results with some unstable ablation outcomes.

major comments (5)
  1. [5.1, Table 3, Figure 4] The central effectiveness claim rests on Overall ASR, not Full ASR. Section 5.1 defines Overall ASR as the fraction of data points where at least one of K candidate mismatches triggers the target word, while Full ASR is the per-trigger success rate. Table 3 and the abstract's "over 98% average ASR" use Overall ASR exclusively; Full ASR is plotted only for the color variants in Figure 4 and is never tabulated for object semantics or for the main configurations. In a real deployment, a user produces one specific mismatch (e.g., "red curtain" on a white curtain), so the relevant reliability measure is Full ASR. The paper should report Full ASR for all BadSem variants and both datasets, and the abstract and conclusions should be qualified accordingly; if Full ASR is substantially lower than Overall ASR, the central effectiveness claim is overstated.
  2. [4.3, Majority Voting] The construction of SIMBad selects triggers by asking three VLMs whether the modified pair is inconsistent (M(I,T(e_i)) = "No") and retaining candidates confirmed by at least two models. This means the "semantic mismatch" set is defined by VLM judgment rather than ground truth. Because the victim models are also VLMs trained on similar web data, the high ASR may partly reflect shared hallucination or answer patterns among VLMs rather than genuine semantic contradictions. The manuscript reports no human verification or precision of the filtered set. Please add a human-annotated subsample verifying that the filtered SI data are actually inconsistent, and report ASR on human-verified mismatches; otherwise the semantic-trigger interpretation is not fully established.
  3. [Table 8] Table 8 shows that BadSem-O-T on VQAv2 attains ASR 47.23 at 1000 clean samples, 35.93 at 3000 samples, and 99.94 at 5000 samples. The text states that "larger data sizes generally lead to higher ASR," but the decrease from 1000 to 3000 contradicts this and indicates that results are highly unstable in this regime. The paper should explain this non-monotonicity, report multiple seeds or error bars, or restrict the robustness claim to the 5000-sample setting; as written, the claim is not supported.
  4. [5.1, 6.1] All experiments are single runs without error bars or multi-seed variation. Greedy decoding removes sampling noise, but fine-tuning randomness (data subsampling, LoRA initialization, data order) remains. Given the sharp discontinuities in Table 8 and the 4-9% variation across GQA visual variants in Table 3, the reported ASR values may not be fully reproducible. Please provide standard deviations over at least three runs for the main configurations, or justify why a single run is sufficient.
  5. [3, 5.1] The threat model states that the attacker has no knowledge of the victim's training pipeline, but all experiments use LoRA on 5,000 clean samples with learning rate 1e-4 and 3 epochs. The attack's effectiveness under full fine-tuning, different optimizers, or larger clean sets is untested. Since the practical claim is that this is a realistic data-poisoning threat, the paper should either evaluate at least one full fine-tuning configuration or explicitly narrow the claim to LoRA-based instruction tuning.
minor comments (4)
  1. [5.2, Table 3] Several GQA visual variants have ASR below 95% (e.g., BadSem-C-V on GQA: 92.30-94.99; BadSem-O-V on GQA: 90.97-94.97); the sentence "over 95% ASR in most configurations" should be reconciled with the abstract's "over 98% average ASR."
  2. [5.6] The attention analysis is based on a single example pair; no quantitative aggregation across samples or layers is provided. This supports but does not establish the claim about attention patterns in backdoored models.
  3. [6.2 and Conclusion] The limitations section contains a formatting artifact, "B ADSEM" instead of "BadSem," and the conclusion's claim that "none of them effectively reduce the ASR" should be softened because supervised fine-tuning reduces ASR in several cases (e.g., BadSem-O-T on GQA for LLaVA drops from 90.97 to 80.95 in Table 12).
  4. [Availability] The manuscript does not state whether the SIMBad dataset or the attack code will be released; given that SIMBad is presented as a contribution, a data/code availability statement would aid reproducibility.

Circularity Check

1 steps flagged · score 3.0 of 10

The 'semantic mismatch' trigger is defined by VLM 'No' judgments, and the ASR is measured on the same VLM-selected data, so the headline effectiveness claim is partially self-defined; headline ASR also relies on per-image Overall ASR.

  1. self definitional [Section 4.3 (formal definition of F and C_final) and Section 5.1 (ASR measurement on SI-T/SI-V)]
    "If the model responds negatively (i.e., indicating that the inserted element does not exist), we consider the corresponding question or image to introduce semantic inconsistency. ... F(I,Q(e_i)) = ( 1, if M(I,T(e_i)) = "No" ... Backdoor ASR is measured on semantically inconsistent data, including both textual (SI-T) and visual (SI-V) variants."

    The paper's trigger concept ('semantic mismatch') is operationalized as a VLM's 'No' response to an existence query, and the same VLM-judged condition is used to construct the SI-T/SI-V sets that are then used both for poisoning and for ASR evaluation. Consequently, the central claim that semantic mismatches serve as effective backdoor triggers is only demonstrated for inputs that a family of VLMs already recognizes as inconsistent. The backdoored model is fine-tuned on exactly this VLM-selected condition, so the measured ASR reflects learning the training association on the same definitionally selected distribution, rather than an independent test of real-world semantic mismatch.

full rationale

The paper is largely self-contained: it evaluates on external VQA benchmarks (VQAv2, GQA), compares against seven baselines, and reports ablations over PCR, DAR, learning rate, and data size. The few self-citations (e.g., [21], [65], [66]) are for prior poisoning work, a baseline, and the SFT defense method; none provides a load-bearing uniqueness or ansatz that forces the central result. The main circularity concern is that the trigger ground truth (semantic mismatch) is defined by VLM majority voting rather than human annotation, and the same VLM-defined condition generates both the poisoned training samples and the ASR evaluation set, making the headline effectiveness claim partially definitional. Separately, the abstract's 'over 98% average ASR' relies on Overall ASR (success if any of K candidate mismatches fires), while the more relevant per-trigger Full ASR is only plotted for color variants and not tabulated for the main configurations; this inflates the headline number but is a reporting/measurement issue rather than a derivation-equivalence. Overall, the central claim retains substantial independent empirical content, so the score is moderate rather than high.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The core attack rests on four families of choices: the mismatch-filtering procedure (which uses three VLMs as ground-truth judges), the specific fine-tuning setup (LoRA on 5,000 clean samples with fixed hyperparameters), the image-editing tools (Grounded SAM and Step1X-Edit), and the definition of clean semantics. The paper treats the first three as technical implementation details, but the results, especially the OOD and cross-modality transfer, depend on them. The invented entity is the dataset SIMBad, which is not released.

free parameters (5)
  • HSV hue values for recolor = 0, 30, 60, 120, 140, 160 degrees
    Six presets for red/yellow/green/blue/purple/pink, chosen by hand from common color wheels; they define the visual mismatches used to poison color semantics.
  • Grounded SAM box confidence threshold = 0.5
    Selected to retain only high-confidence detections when segmenting objects for recoloring; not justified by an ablation.
  • LoRA rank, learning rate, epochs = rank 16, lr 1e-4, 3 epochs
    Default hyperparameters for the victim fine-tuning; the paper ablates learning rate but does not search over rank or epoch count.
  • Clean sample size = 5,000
    The fine-tuning set uses 5,000 clean samples; the training-data-size ablation in Table 8 shows strong size dependence (ASR drops to 35-47% for BadSem-O-T at 1,000-3,000 samples), so this choice is load-bearing.
  • Poisoning-to-clean ratio (PCR) = 1%, 2%, 5%
    A controlled experimental factor, but the central claim 'high effectiveness at low poisoning' depends on the chosen PCR range.
assumptions (4)
  • domain assumption The three VLM voters (Qwen2.5-VL-7B, Gemma 3-4B, Gemma 3-12B) correctly judge semantic inconsistencies in the poisoned data.
    All mismatches are filtered by VLM majority voting; if all three voters hallucinate or misjudge, the mismatch set is wrong. Section 4.3.
  • domain assumption Fine-tuning hyperparameters used in experiments match the threat model's black-box assumption that the attacker cannot control the victim pipeline.
    The attack explicitly assumes no knowledge of the victim pipeline (Section 3), yet the experiments fix a specific LoRA recipe; if realistic deployments use different pipelines, the ASR may change. Section 5.1 and Table 8.
  • domain assumption The edited images (recolored, object-replaced) preserve the rest of the scene semantics so the mismatch is local.
    If Step1X-Edit changes more than the target object or Grounded SAM recolors the wrong region, the images are no longer 'semantically consistent except for the trigger'. Section 4.3.
  • domain assumption Clean semantic accuracy is measured on pairs where the question refers to an element present in the image.
    FP ASR and CA-S are defined on SC data; if SC examples contain latent mismatches, the stealthiness numbers would be optimistic. Section 5.1.
invented entities (1)
  • SIMBad dataset
    purpose: A poisoned image-text dataset with semantically consistent and inconsistent pairs for color and object semantics.
    The dataset is described and its statistics given, but not released with the paper, so there is no external handle for independent reproduction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Backdoor Attack on Vision Language Models with Stealthy Semantic Manipulation." pith.science (2026). https://pith.science/paper/PMKKVE2R

@misc{pith2026250607214,
  author       = {Pith},
  title        = {Pith review of: Backdoor Attack on Vision Language Models with Stealthy Semantic Manipulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PMKKVE2R}},
  note         = {Machine review of arXiv:2506.07214}
}
read the original abstract

Vision Language Models (VLMs) have shown remarkable performance, but are also vulnerable to backdoor attacks whereby the adversary can manipulate the model's outputs through hidden triggers. Prior attacks primarily rely on single-modality triggers, leaving the crucial cross-modal fusion nature of VLMs largely unexplored. Unlike prior work, we identify a novel attack surface that leverages cross-modal semantic mismatches as implicit triggers. Based on this insight, we propose BadSem (Backdoor Attack with Semantic Manipulation), a data poisoning attack that injects stealthy backdoors by deliberately misaligning image-text pairs during training. To perform the attack, we construct SIMBad, a dataset tailored for semantic manipulation involving color and object attributes. Extensive experiments across four widely used VLMs show that BadSem achieves over 98% average ASR, generalizes well to out-of-distribution datasets, and can transfer across poisoning modalities. Our detailed analysis using attention visualization shows that backdoored models focus on semantically sensitive regions under mismatched conditions while maintaining normal behavior on clean inputs. To mitigate the attack, we try two defense strategies based on system prompt and supervised fine-tuning but find that both of them fail to mitigate the semantic backdoor. Our findings highlight the urgent need to address semantic vulnerabilities in VLMs for their safer deployment.

Figures

Figures reproduced from arXiv: 2506.07214 by the authors.

Figure 1
Figure 1. Examples of backdoored model behavior with [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of BadSem 4.1 Targeted Semantics Recent studies have investigated the ability of VLMs to un￾derstand various semantic elements [43–45]. VHTest [43] and MMVP [44] examine a range of visual patterns essential to semantic understanding, while ColorBench [45] specifi￾cally evaluates models’ comprehension of color-related se￾mantics. In this work, we focus on two commonly encoun￾tered types of semantics in the V… view at source ↗
Figure 3
Figure 3. Effect of DAR on model performance (PCR=3%) with [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Full ASR of BadSem variants on VQAv2 under dif￾ferent PCR. higher poison exposure. Importantly, despite the increased backdoor strength, the models maintain strong clean accu￾racy comparable to the clean baseline (0% PCR) and exhibit low FP ASR across most settings. Ho…
Figure 5
Figure 5. Figure 5: Layer-wise attention visualization of clean and backdoored LLaVA models using LVLM-Interpret [ [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: illustrates two examples from SIMBad showcasing color and object semantics. (a) Targeting color semantics. (b) Targeting object semantics [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SlowBA: An efficiency backdoor attack towards VLM-based GUI agents

    cs.CR 2026-03 conditional novelty 7.0 of 10

    SlowBA uses two-stage reward-level injection and pop-up triggers to make VLM GUI agents produce much longer, slower responses under attack while largely preserving task accuracy.

  2. IAG: Input-aware Backdoor Attack on VLM-based Visual Grounding

    cs.CV 2025-08 conditional novelty 7.0 of 10

    A text-conditioned U-Net can generate input-aware triggers that backdoor VLM visual grounding, forcing the model to output the attacker-chosen object's bounding box regardless of the user query.

Reference graph

Works this paper leans on

74 extracted references · 41 canonical work pages · cited by 2 Pith papers

  1. [1]

    Improved baselines with visual instruction tuning,

    H. Liu, C. Li, Y . Li, and Y . J. Lee, “Improved baselines with visual instruction tuning,” pp. 26 286–26 296,

    work page

  2. [2]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,

    P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, Y . Fan, K. Dang, M. Du, X. Ren, R. Men, D. Liu, C. Zhou, J. Zhou, and J. Lin, “Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,” 2024. [Online]. Available: https://arxiv.org/abs/2409.12191 1, 2, 6, 11

    arXiv 2024

  3. [3]

    Llama 3.2: Revolutionizing edge ai and vision with open, customizable models,

    Meta, “Llama 3.2: Revolutionizing edge ai and vision with open, customizable models,” 2024. [Online]. Available: https://ai.meta.com/blog/llama-3-2-connec t-2024-vision-edge-mobile-devices/ 1, 2, 6

    work page 2024

  4. [4]

    Backdooring vision- language models with out-of-distribution data,

    W. Lyu, J. Yao, S. Gupta, L. Pang, T. Sun, L. Yi, L. Hu, H. Ling, and C. Chen, “Backdooring vision- language models with out-of-distribution data,” in The Thirteenth International Conference on Learning Representations, 2025. [Online]. Available: https: //openreview.net/forum?id=tZozeR3VV7 1, 3

    work page 2025

  5. [5]

    Stealthy Backdoor Attack in Self-Supervised Learning Vision Encoders for Large Vision Language Models

    Z. Liu and H. Zhang, “Stealthy backdoor attack in self-supervised learning vision encoders for large vision language models,” 2025. [Online]. Available: https://arxiv.org/abs/2502.18290 1, 3

    work page Pith review arXiv 2025

  6. [6]

    Vl-trojan: Multimodal instruction backdoor attacks against autoregressive visual language models,

    J. Liang, S. Liang, M. Luo, A. Liu, D. Han, E.-C. Chang, and X. Cao, “Vl-trojan: Multimodal instruction backdoor attacks against autoregressive visual language models,” 2024. [Online]. Available: https://arxiv.org/abs/2402.13851 1, 3

    arXiv 2024

  7. [7]

    Trojvlm: Backdoor attack against vision language models,

    W. Lyu, L. Pang, T. Ma, H. Ling, and C. Chen, “Trojvlm: Backdoor attack against vision language models,” in Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part LXV , ser. Lecture Notes in Computer Science, vol. 15123. Springer, 2024, pp. 467–483. [Online]. Available: https://doi.org/10.1007...

    work page doi:10.1007/978-3-031-73650-6_27 2024

  8. [8]

    Shadowcast: Stealthy data poisoning attacks against vision-language models,

    Y . Xu, J. Yao, M. Shu, Y . Sun, Z. Wu, N. Yu, T. Goldstein, and F. Huang, “Shadowcast: Stealthy data poisoning attacks against vision-language models,” in The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024. [Online]. Available: https://openreview.net/forum?id=JhqyeppM iD 1, 3

    work page 2024

Show all 74 references
  1. [9]

    Badnets: Identifying vulnerabilities in the machine learning model supply chain,

    T. Gu, B. Dolan-Gavitt, and S. Garg, “Badnets: Identifying vulnerabilities in the machine learning model supply chain,” 2019. [Online]. Available: https://arxiv.org/abs/1708.06733 1, 3, 6

    2019 arXiv

  2. [10]

    Poisoning and backdooring contrastive learning,

    N. Carlini and A. Terzis, “Poisoning and backdooring contrastive learning,” in The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net,

    2022

  3. [12]

    Badclm: Backdoor attack in clinical language models for electronic health records,

    W. Lyu, Z. Bi, F. Wang, and C. Chen, “Badclm: Backdoor attack in clinical language models for electronic health records,” 2024. [Online]. Available: https://arxiv.org/abs/2407.05213 1

    2024 arXiv

  4. [13]

    Badnl: Backdoor attacks against NLP models with semantic-preserving improvements,

    X. Chen, A. Salem, D. Chen, M. Backes, S. Ma, Q. Shen, Z. Wu, and Y . Zhang, “Badnl: Backdoor attacks against NLP models with semantic-preserving improvements,” in ACSAC ’21: Annual Computer Security Applications Conference, Virtual Event, USA, December 6 - 10, 2021 . ACM, 202...

    2021 doi

  5. [14]

    Drivellava: Human-level behavior decisions via vision language model,

    R. Zhao, Q. Yuan, J. Li, Y . Fan, Y . Li, and F. Gao, “Drivellava: Human-level behavior decisions via vision language model,” Sensors, vol. 24, no. 13, 2024. [Online]. Available: https://www.mdpi.com/1424- 8220/24/13/4113 1

    2024

  6. [15]

    V2x-vlm: End-to-end v2x cooperative autonomous driving through large vision-language models,

    J. You, H. Shi, Z. Jiang, Z. Huang, R. Gan, K. Wu, X. Cheng, X. Li, and B. Ran, “V2x-vlm: End-to-end v2x cooperative autonomous driving through large vision-language models,” 2024. [Online]. Available: https://arxiv.org/abs/2408.09251 1

    2024 arXiv

  7. [16]

    Palm-e: An embodied multimodal language model,

    D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang, Y . Chebotar, P. Sermanet, D. Duckworth, S. Levine, V . Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, I. Mordatch, and P. Florence, “Palm-e: An em...

    2023 arXiv

  8. [17]

    Robopoint: A vision-language model for spatial affordance prediction 14 for robotics,

    W. Yuan, J. Duan, V . Blukis, W. Pumacay, R. Krishna, A. Murali, A. Mousavian, and D. Fox, “Robopoint: A vision-language model for spatial affordance prediction 14 for robotics,” 2024. [Online]. Available: https: //arxiv.org/abs/2406.10721 1

    2024 arXiv

  9. [18]

    Test-time backdoor attacks on multimodal large language models,

    D. Lu, T. Pang, C. Du, Q. Liu, X. Yang, and M. Lin, “Test-time backdoor attacks on multimodal large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2402.08577 1, 3

    2024 arXiv

  10. [19]

    Bait: Large language model backdoor scanning by in- verting attack target,

    G. Shen, S. Cheng, Z. Zhang, G. Tao, K. Zhang, H. Guo, L. Yan, X. Jin, S. An, S. Ma, and X. Zhang, “Bait: Large language model backdoor scanning by in- verting attack target,” in Proceedings of the 46th IEEE Symposium on Security and Privacy. IEEE Computer Society, 2025. 2

    2025

  11. [20]

    Better trigger inversion optimization in backdoor scanning,

    G. Tao, G. Shen, Y . Liu, S. An, Q. Xu, S. Ma, P. Li, and X. Zhang, “Better trigger inversion optimization in backdoor scanning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, 2022, pp. 13 368–13 378. 2

    2022

  12. [21]

    Data poisoning attacks against multimodal encoders,

    Z. Yang, X. He, Z. Li, M. Backes, M. Hum- bert, P. Berrang, and Y . Zhang, “Data poisoning attacks against multimodal encoders,” 2023. [Online]. Available: https://arxiv.org/abs/2209.15266 2, 3

    2023 arXiv

  13. [22]

    Gpt-4o system card,

    OpenAI, “Gpt-4o system card,” 2024. [Online]. Available: https://openai.com/index/gpt-4o-system- card/ 2, 7

    2024

  14. [23]

    Gemini 2.0 flash,

    Google, “Gemini 2.0 flash,” 2025. [Online]. Available: https://cloud.google.com/vertex-ai/generative-ai/docs/ models/gemini/2-0-flash 2

    2025

  15. [24]

    A survey on hallucination in large vision-language models,

    H. Liu, W. Xue, Y . Chen, D. Chen, X. Zhao, K. Wang, L. Hou, R. Li, and W. Peng, “A survey on hallucination in large vision-language models,” 2024. [Online]. Available: https://arxiv.org/abs/2402.00253 2

    2024 arXiv

  16. [25]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,”

  17. [26]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” in International Conference on Learning Representations,

  18. [27]

    Backdoor attacks against learning systems,

    Y . Ji, X. Zhang, and T. Wang, “Backdoor attacks against learning systems,” in Proc. 2017 IEEE Conf. Commun. Network Secur. (CNS). IEEE, 2017, pp. 1–9. 3

    2017

  19. [28]

    Targeted backdoor attacks on deep learning systems using data poisoning,

    X. Chen, C. Liu, B. Li, K. Lu, and D. Song, “Targeted backdoor attacks on deep learning systems using data poisoning,” arXiv preprint arXiv:1712.05526, 2017. 3

    2017 arXiv

  20. [29]

    Weight poisoning attacks on pretrained models,

    K. Kurita, P. Michel, and G. Neubig, “Weight poisoning attacks on pretrained models,” in Proc. 58th Annu. Meet. Assoc. Comput. Linguist. (ACL). Association for Computational Linguistics, Jul. 2020, pp. 2793–2806. [Online]. Available: https://aclanthology.org/2020.acl- main.249 3

    2020

  21. [30]

    Available: https://openreview.net/for um?id=nZeVKeeFYf9 3, 6

    [Online]. Available: https://openreview.net/for um?id=nZeVKeeFYf9 3, 6

  22. [31]

    Hidden killer: Invisible textual backdoor attacks with syntactic trigger,

    F. Qi, M. Li, Y . Chen, Z. Zhang, Z. Liu, Y . Wang, and M. Sun, “Hidden killer: Invisible textual backdoor attacks with syntactic trigger,” in Proc. 59th Annu. Meet. Assoc. Comput. Linguist. (ACL) and 11th Int. Joint Conf. Natural Lang. Process. Online: Association for Computa...

    2021

  23. [32]

    Mind the style of text! adversarial and backdoor attacks based on text style transfer,

    F. Qi, Y . Chen, X. Zhang, M. Li, Z. Liu, and M. Sun, “Mind the style of text! adversarial and backdoor attacks based on text style transfer,” in Proc. 2021 Conf. Empirical Methods Nat. Lang. Process. (EMNLP). Association for Computational Linguistics, Nov. 2021, pp. 4569–4580...

    2021

  24. [33]

    Backdoor pre- trained models can transfer to all,

    L. Shen, S. Ji, X. Zhang et al. , “Backdoor pre- trained models can transfer to all,” arXiv preprint arXiv:2111.00197, 2021. 3

    2021 arXiv

  25. [34]

    A backdoor attack against lstm-based text classification systems,

    J. Dai, C. Chen, and Y . Li, “A backdoor attack against lstm-based text classification systems,” IEEE Access , vol. 7, pp. 138 872–138 878, 2019. 3

    2019

  26. [35]

    Lora-as-an-attack! piercing llm safety under the share-and-play scenario,

    H. Liu, Z. Liu, R. Tang, J. Yuan, S. Zhong, Y .-N. Chuang, L. Li, R. Chen, and X. Hu, “Lora-as-an-attack! piercing llm safety under the share-and-play scenario,”

  27. [36]

    Sleeper agents: Train- ing deceptive llms that persist through safety training,

    E. Hubinger, C. Denison, J. Mu, M. Lambert, M. Tong, M. MacDiarmid, T. Lanham, D. M. Ziegler, T. Maxwell, N. Cheng et al., “Sleeper agents: Train- ing deceptive llms that persist through safety training,” arXiv preprint arXiv:2401.05566, 2024. 3

    2024 arXiv

  28. [37]

    Poisoning language models during instruction tuning,

    A. Wan, E. Wallace, S. Shen, and D. Klein, “Poisoning language models during instruction tuning,” in Proc. Int. Conf. Mach. Learn. (ICML) . PMLR, 2023, pp. 35 413–35 425. 3

    2023

  29. [38]

    Backdoor pre- trained models can transfer to all,

    L. Shen, S. Ji, X. Zhang, J. Li, J. Chen, J. Shi, C. Fang, J. Yin, and T. Wang, “Backdoor pre- trained models can transfer to all,” in Proc. 2021 ACM SIGSAC Conf. Comput. Commun. Secur. , ser. CCS ’21, 2021, p. 3141–3158. [Online]. Available: https://doi.org/10.1145/3460120.3485370 3

    2021

  30. [39]

    Backdooring instruction-tuned large language models with virtual prompt injection,

    J. Yan, V . Yadav, S. Li, L. Chen, Z. Tang, H. Wang, V . Srinivasan, X. Ren, and H. Jin, “Backdooring instruction-tuned large language models with virtual prompt injection,” inNeurIPS 2023 Workshop on Back- doors in Deep Learning-The Good, the Bad, and the Ugly, 2023. 3

    2023

  31. [40]

    Backdooring multimodal learn- 15 ing,

    X. Han, Y . Wu, Q. Zhang, Y . Zhou, Y . Xu, H. Qiu, G. Xu, and T. Zhang, “Backdooring multimodal learn- 15 ing,” in2024 IEEE Symposium on Security and Privacy (SP), 2024, pp. 3385–3403. 3, 8

    2024

  32. [41]

    azure-open-dataset,

    Microsoft, “azure-open-dataset,” 2025. [Online]. Avail- able: https://azure.microsoft.com/en-us/products/op en-datasets 3

    2025

  33. [42]

    Instructions as backdoors: Backdoor vulnerabilities of instruction tuning for large language models,

    J. Xu, M. D. Ma, F. Wang, C. Xiao, and M. Chen, “Instructions as backdoors: Backdoor vulnerabilities of instruction tuning for large language models,” arXiv preprint arXiv:2305.14710, 2023. 3

    2023 arXiv

  34. [43]

    Visual hallucinations of multi-modal large language models,

    W. Huang, H. Liu, M. Guo, and N. Z. Gong, “Visual hallucinations of multi-modal large language models,”

  35. [44]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms,

    S. Tong, Z. Liu, Y . Zhai, Y . Ma, Y . LeCun, and S. Xie, “Eyes wide shut? exploring the visual shortcomings of multimodal llms,” 2024. [Online]. Available: https://arxiv.org/abs/2401.06209 4

    2024 arXiv

  36. [45]

    Colorbench: Can vlms see and understand the colorful world? a comprehensive benchmark for color perception, reasoning, and robustness,

    Y . Liang, M. Li, C. Fan, Z. Li, D. Nguyen, K. Cobbina, S. Bhardwaj, J. Chen, F. Liu, and T. Zhou, “Colorbench: Can vlms see and understand the colorful world? a comprehensive benchmark for color perception, reasoning, and robustness,” 2025. [Online]. Available: https://arxiv....

    2025

  37. [46]

    huggingface-dataset,

    H. Face, “huggingface-dataset,” 2025. [Online]. Avail- able: https://huggingface.co/datasets 3

    2025

  38. [47]

    Step1x-edit: A practical framework for general image editing,

    S. Liu, Y . Han, P. Xing, F. Yin, R. Wang, W. Cheng, J. Liao, Y . Wang, H. Fu, C. Han, G. Li, Y . Peng, Q. Sun, J. Wu, Y . Cai, Z. Ge, R. Ming, L. Xia, X. Zeng, Y . Zhu, B. Jiao, X. Zhang, G. Yu, and D. Jiang, “Step1x-edit: A practical framework for general image editing,” 202...

    2025 arXiv

  39. [48]

    Available: https://arxiv.org/abs/2402.1 4683 4, 13

    [Online]. Available: https://arxiv.org/abs/2402.1 4683 4, 13

  40. [49]

    Qwen2.5-vl,

    Qwen, “Qwen2.5-vl,” 2025. [Online]. Available: https: //huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct 5

    2025

  41. [50]

    Gemma 3-4b,

    Google, “Gemma 3-4b,” 2025. [Online]. Available: https://huggingface.co/google/gemma-3-4b-it 5

    2025

  42. [51]

    Grounded sam: Assembling open-world models for diverse visual tasks,

    T. Ren, S. Liu, A. Zeng, J. Lin, K. Li, H. Cao, J. Chen, X. Huang, Y . Chen, F. Yan, Z. Zeng, H. Zhang, F. Li, J. Yang, H. Li, Q. Jiang, and L. Zhang, “Grounded sam: Assembling open-world models for diverse visual tasks,” 2024. [Online]. Available: https://arxiv.org/abs/2401.14159 5

    2024 arXiv

  43. [52]

    Qwen2.5-vl technical report,

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y . Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y . Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin, “Qwen2.5-vl technical report,” 2025. [Onli...

    2025 arXiv

  44. [53]

    Diffusion model-based image editing: A survey,

    Y . Huang, J. Huang, Y . Liu, M. Yan, J. Lv, J. Liu, W. Xiong, H. Zhang, L. Cao, and S. Chen, “Diffusion model-based image editing: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, no. 6, p. 4409–4437, Jun. 2025. [Online]. Available: http://d...

    2025

  45. [54]

    Self-consistency of voting implies majority vote,

    A. Poplawski, “Self-consistency of voting implies majority vote,” 2018. [Online]. Available: https: //arxiv.org/abs/1807.00170 5

    2018 arXiv

  46. [55]

    Llava-1.5,

    H. Liu, C. Li, Y . Li, and Y . J. Lee, “Llava-1.5,” 2025. 16 [Online]. Available: https://huggingface.co/llava- hf/llava-1.5-7b-hf 6, 7

    2025

  47. [56]

    Gemma 3-12b,

    ——, “Gemma 3-12b,” 2025. [Online]. Available: https://huggingface.co/google/gemma-3-12b-it 5

    2025

  48. [57]

    Llama 3.2-vision,

    Meta, “Llama 3.2-vision,” 2025. [Online]. Available: https://huggingface.co/meta-llama/Llama-3.2-11B- Vision-Instruct 6, 7

    2025

  49. [58]

    Gemma 3 technical report,

    G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ramé, M. Rivière, L. Rouillard, T. Mesnard, G. Cideron, J. bastien Grill, S. Ramos, E. Yvinec, M. Casbon, E. Pot, I. Penchev, G. Liu, F. Visin, K. Kenealy, L. Beyer, X. Zhai, A. T...

    2025 arXiv

  50. [59]

    Gqa: A new dataset for real-world visual reasoning and composi- tional question answering,

    D. A. Hudson and C. D. Manning, “Gqa: A new dataset for real-world visual reasoning and composi- tional question answering,” in 2019 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 6693–6702. 6, 11

    2019

  51. [60]

    Llamafactory: Unified efficient fine-tuning of 100+ language models,

    Y . Zheng, R. Zhang, J. Zhang, Y . Ye, and Z. Luo, “Llamafactory: Unified efficient fine-tuning of 100+ language models,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) . Bangkok, Thailand: Associati...

    2024

  52. [61]

    Qwen2-vl,

    Qwen, “Qwen2-vl,” 2025. [Online]. Available: https: //huggingface.co/collections/Qwen/qwen2-vl-66cee7 455501d7126940800d 6, 7

    2025

  53. [62]

    Invisible backdoor attack with sample-specific triggers,

    Y . Li, Y . Li, B. Wu, L. Li, R. He, and S. Lyu, “Invisible backdoor attack with sample-specific triggers,” in 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021 . IEEE, 2021, pp. 16 443–16 452. [Online]. Available: ...

    2021

  54. [63]

    Making the V in VQA matter: Elevating the role of image understanding in visual question answering,

    Y . Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh, “Making the V in VQA matter: Elevating the role of image understanding in visual question answering,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 201...

    2017 doi

  55. [64]

    Revisiting backdoor attacks against large vision-language models from domain shift,

    S. Liang, J. Liang, T. Pang, C. Du, A. Liu, E.-C. Chang, and X. Cao, “Revisiting backdoor attacks against large vision-language models from domain shift,” 2024. [Online]. Available: https: //api.semanticscholar.org/CorpusID:270764501 7

    2024

  56. [65]

    Cl- attack: Textual backdoor attacks via cross-lingual triggers,

    J. Zheng, T. Hu, T. Cong, and X. He, “Cl- attack: Textual backdoor attacks via cross-lingual triggers,” in AAAI-25, Sponsored by the Association for the Advancement of Artificial Intelligence, February 25 - March 4, 2025, Philadelphia, PA, USA , T. Walsh, J. Shah, and Z. Kolte...

    2025 doi

  57. [66]

    Backdoorllm: A comprehensive benchmark for backdoor attacks on large language models,

    Y . Li, H. Huang, Y . Zhao, X. Ma, and J. Sun, “Backdoorllm: A comprehensive benchmark for backdoor attacks on large language models,” ArXiv, vol. abs/2408.12798, 2024. [Online]. Available: https: //api.semanticscholar.org/CorpusID:271947035 7

    2024 arXiv

  58. [67]

    Security steerability is all you need,

    I. Hazan, I. Habler, R. Bitton, and I. Mantin, “Security steerability is all you need,” 2025. [Online]. Available: https://arxiv.org/abs/2504.19521 7

    2025 arXiv

  59. [68]

    Mind the style of text! adversarial and backdoor attacks based on text style transfer,

    F. Qi, Y . Chen, X. Zhang, M. Li, Z. Liu, and M. Sun, “Mind the style of text! adversarial and backdoor attacks based on text style transfer,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dom...

    2021 doi

  60. [69]

    Lvlm-interpret: An interpretabil- ity tool for large vision-language models,

    G. B. M. Stan, E. Aflalo, R. Y . Rohekar, A. Bhiwandi- walla, S.-Y . Tseng, M. L. Olson, Y . Gurwicz, C. Wu, N. Duan, and V . Lal, “Lvlm-interpret: An interpretabil- ity tool for large vision-language models,” 2024. [Online]. Available: https://arxiv.org/abs/2404.03118 12, 13

    2024 arXiv

  61. [70]

    Rex-thinker: Grounded object referring via chain- of-thought reasoning,

    Q. Jiang, X. Chen, Z. Zeng, J. Yu, and L. Zhang, “Rex-thinker: Grounded object referring via chain- of-thought reasoning,” 2025. [Online]. Available: https://arxiv.org/abs/2506.04034 13 17 Appendix A: Ethical Statement This paper reveals a previously unexplored attack surface ...

    2025 arXiv

  62. [71]

    Fine-tuning is all you need to mitigate backdoor attacks,

    Z. Sha, X. He, P. Berrang, M. Humbert, and Y . Zhang, “Fine-tuning is all you need to mitigate backdoor attacks,” 2022. [Online]. Available: https: //arxiv.org/abs/2212.09067 7

    2022 arXiv

  63. [73]

    A survey on backdoor threats in large language models (llms): Attacks, defenses, and evaluations,

    Y . Zhou, T. Ni, W.-B. Lee, and Q. Zhao, “A survey on backdoor threats in large language models (llms): Attacks, defenses, and evaluations,” 2025. [Online]. Available: https://arxiv.org/abs/2502.05224 8

    2025 arXiv

  64. [2021]

    Available: https://arxiv.org/abs/2103.0 0020 2

    [Online]. Available: https://arxiv.org/abs/2103.0 0020 2

  65. [2022]

    Available: https://openreview.net/for um?id=iC4UHbQ01Mp 1, 2, 3

    [Online]. Available: https://openreview.net/for um?id=iC4UHbQ01Mp 1, 2, 3

  66. [2024]

    Available: https://doi.org/10.1109/CV PR52733.2024.02484 1, 2, 3, 6

    [Online]. Available: https://doi.org/10.1109/CV PR52733.2024.02484 1, 2, 3, 6

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.