Pith. sign in

REVIEW 4 major objections 5 minor 49 references

PSA-VLM: Enhancing Vision-Language Model Safety through Progressive Concept-Bottleneck-Driven Alignment

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A concept-bottleneck safety gate lifts VLM safety scores to state-of-the-art levels on the RTVLM benchmark, with a 13B model averaging 8.46.

desk verdict A useful modular safety gate for VLMs, but the SOTA claim rests on an unfair benchmark split and the 'concept bottleneck' label is overstated. read the letter →

arxiv 2411.11543 v4 pith:PIXSDZXZ submitted 2024-11-18 cs.CV cs.AI

Zhendong Liu , Yuanbi Nie , Yingshui Tan , Jiaheng Liu , Xiangyu Yue , Qiushi Cui , Chongjun Wang , Xiaoyong Zhu
show 1 more author
Bo Zheng
This is my paper · ORCID
classification cs.CVcs.AI
keywords vision-languagemodelsafetyconceptbottleneckalignmentpromptrewritingRTVLMbenchmarktwo-stagetrainingLLaVAmulti-modalriskclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the visual-modality safety hole in vision-language models can be closed cheaply by inserting a small set of safety modules trained as a concept bottleneck. The modules—a safety projector, safety tokens, and a safety head—classify every input image into one of six risk types (politics, illegal content, insults and bullying, fairness, privacy, misleading) and one of three risk levels, and the predicted label picks a hand-written condition prompt that is prepended to the user query. On the RTVLM safety benchmark, the resulting PSA-VLM-13B model with LoRA fine-tuning reaches an average safety score of 8.46, ahead of all compared baselines, and scores 9.49/8.72/7.45 on separate politics, pornography, and cyberbullying datasets. Training is staged: about one GPU-hour for the safety modules alone, then about eight GPU-hours to lightly fine-tune the language model. The authors report that general performance on MMBench and SEEDBench is preserved or slightly improved.

What carries the argument

The load-bearing machinery is the concept-bottleneck-style safety head. Following the Concept Bottleneck Model idea, the safety projector $g_\phi$ maps visual features $h_o$ to safety-aligned features $h_s$; two sets of trainable safety tokens are concatenated with the original and safety-aligned features (Equation 2); and a cross-attention layer between the LLM's text embeddings and the combined safety features feeds two softmax classifiers (Equation 4) that predict risk type and risk level. These explicit labels select a condition prompt from a hand-built table (Figure 10) that is prepended to the user instruction, so that the same frozen or lightly fine-tuned LLM is steered toward safe decoding. A two-stage schedule makes the intervention cheap: stage I trains only the safety modules with the vision encoder and LLM frozen; stage II unfreezes the LLM with LoRA, which the ablations show adds a smaller but real increment.

What would settle it

Run the released PSA-VLM-13B checkpoint on a fresh set of clearly benign images that visually resemble the six risk categories (posters, celebrity photos, code screenshots, artworks) and count how often the model refuses to answer; the paper's Figure 5 already documents such false positives on MME, so a falsifying result would be a high refusal rate on a large benign test set. The complementary check is an adaptive attack that optimizes the safety head's classification loss to output 'clean' while the image content remains harmful; success of that attack would show the gate is bypassable.

Watch

Extended reading notes

Core claim

The paper's central claim is that a VLM's resistance to visually transmitted harmful content can be improved markedly—without degrading general ability—by training three small safety modules and using their predictions to rewrite the user prompt before the language model decodes. The safety projector transforms the visual encoder's features into a safety-oriented representation; the safety tokens, concatenated into the visual stream, flag risky content; and the safety head uses cross-attention with the LLM's text embeddings to output probability distributions over risk type and risk level. At inference, the most probable type and level select one of a set of hand-written condition texts (e.g., 'This image contains serious risks of pornography; you must refuse to give the answer'), which steers the VLM toward cautious or refusing behavior. The paper reports state-of-the-art results on RTVLM: 8.26 average for the 7B model and 8.46 for the 13B+LoRA model, versus 6.39 and 6.55 for the LLaVA-1.5 bases; on separate risk datasets the 13B model jumps to 9.49 (politics), 8.72 (pornography), and 7.45 (cyberbullying). Note that the original visual features still flow directly into the LLM alongside the safety tokens: the modules operate as a monitoring-and-intervention channel rather than a literal bottleneck that withholds the raw features.

Load-bearing premise

The load-bearing premise is that the safety head classifies every incoming image correctly into one of the six predefined risk types and one of the three risk levels; if it labels a novel or adversarial image as safe, or labels a clean image as risky, the safety gains and the general performance both erode. The paper's own limitation section concedes both failure modes: false positives on benign inputs and reduced effectiveness against sophisticated adversarial attacks.

Editorial extensions

If this is right

  • Stage I alone, with the base VLM frozen, delivers most of the safety gain (the 7B model reaches 8.18 on RTVLM before any language-model fine-tuning), so the approach is a cheap retrofit for existing LLaVA-style systems.
  • The explicit risk-type and risk-level labels allow per-category control: the paper shows that pornography refusal can be switched off by itself without relaxing other safety categories (Figure 11).
  • The method transfers across VLM architectures: the same safety modules improve both LLaVA and MiniGPT-4 models (Table 10).
  • The clean-to-risky data ratio is critical: roughly 3,000 clean samples, close to the number of risk types, yields the best risk-recognition accuracy in stage I.
  • Safety improvements do not trade away general performance: PSA-VLM-7B (+LoRA) scores 68.5 on MMBench and 65.3 on SEEDBench, slightly above the base model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the raw visual features still reach the LLM unchanged, the 'concept bottleneck' is really a safety side-channel: the head's classifications act by rewriting text prompts, not by constraining the visual information the LLM sees. Interpretability therefore lies in the head's labels and the condition prompts, not in the LLM's reasoning.
  • The conditional-prompt mechanism could be reused for other controlled behaviors—fairness steering, domain-specific refusals, or style control—since the condition texts are hand-written and the heads are cheap to retrain.
  • The method's main vulnerability is an attack that fools only the safety head: if the classifier outputs 'clean' while the raw features remain harmful, the gate is bypassed entirely. An adaptive attack optimizing the head's loss would test this directly.
  • The peak at ~3,000 clean samples hints that the safety head is sensitive to prior shift; deployments whose risk distribution differs from the training mix may need to rebalance or risk more false positives or misses.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes PSA-VLM, a two-stage training procedure that adds three safety modules to a vision-language model (LLaVA): a safety projector, trainable safety tokens, and a cross-attention safety head. The safety head classifies input images into six risk types and three risk levels; based on this classification, the input prompt is conditionally rewritten with hand-written safety instructions, and the LLM is fine-tuned with LoRA in the second stage. The authors report state-of-the-art GPT-4-scored safety results on the RTVLM benchmark and on additional politics, pornography, and cyberbullying datasets, along with a human evaluation on 100 samples and ablations showing that the safety head is the main contributor. Supplementary material includes results on MiniGPT-4 and error bars over three training runs.

Significance. If the empirical claims hold, PSA-VLM is a practical and comparatively low-cost method for improving visual-modality safety in open VLMs while preserving general performance. The paper has several strengths: it ships ablations, a human evaluation, error bars from three runs, and demonstrates transfer to a second VLM architecture (MiniGPT-4). The central claim, however, is not yet supported: the reported safety gains rely on an evaluation protocol that trains on 80% of RTVLM while comparing against baselines scored on the full benchmark, and the general-performance numbers are obtained after test-set-tuned filtering thresholds. The concept-bottleneck framing is also overstated because raw visual features still reach the LLM. These issues are fixable but currently block the headline conclusions.

major comments (4)
  1. [2.6, Table 1] The headline state-of-the-art claim is based on an evaluation asymmetry. Section 2.6 states that RTVLM has no default split and that the authors randomly use 80% of the data for training and 20% for testing. Table 1, however, reports scores for all baselines (Fuyu-8B, VisualGLM, Qwen-VL, LLaVA-v1.5 with and without VLGuard, InternLM-XComposer2, Llama-3-vision-alpha, GPT-4V) that appear to be taken from prior work on the full RTVLM benchmark; the paper never states that these baselines were re-evaluated on the same 20% test split. Since PSA-VLM was trained on 80% of the same benchmark, its higher averages (e.g., 8.46 vs. 8.15 for VLGuard-LoRA on 13B) could reflect distribution familiarity rather than safer visual reasoning. The authors must re-run all baselines on their 20% split, or train without any RTVLM data and evaluate on the full benchmark, or otherwise demonstrate that the 20% split is not more favorable to their method.
  2. [3.3, Supplementary Tables 7-8] The general-performance numbers in Table 3 are not the method's out-of-the-box behavior. The supplement shows that MMEp rises from 1268.7 to 1479.5 and MME from 1470.4 to 1762.7 only after 'condition tuning,' a post hoc adjustment of the safety filtering thresholds illustrated on MME failure cases (Figure 5). Without this tuning, PSA-VLM-7B is below the LLaVA-v1.5-7B baseline on MMEp and MME. Reporting test-set-tuned numbers under the heading 'does not come at the cost of general performance' is circular and overstates the claim. Please present the untuned results as the primary evaluation, or describe a validation-based procedure for setting the filtering thresholds and apply it to a held-out set.
  3. [2.3, Eq. (2)] The architecture described by Eq. (2) does not implement a concept bottleneck. hcomb = [s_t^(1); hi] shows that the original visual features hi are concatenated with safety tokens and fed to the LLM; the safety projector's output hs is used only to compute the cross-attention features for the safety head (Eq. (3)). The LLM's generation is not forced to pass through the safety concepts, so the paper's claim that these modules 'ensure that only aligned, concept-driven representations influence the VLM's decision-making' is not supported. Please either revise the terminology to describe the safety head as an auxiliary gating/conditioning module, or modify the architecture so that unsafe visual features are actually removed or replaced before reaching the LLM.
  4. [2.5, Eq. (7)] Equation (7) is an unexplained formal add-on. The factorization p(S|ct, cl) = p(S|Prompt, ct)·p(Prompt|ct)·p(S|Prompt, cl)·p(Prompt|cl) is not derived, the variables S, ct, cl, and Prompt are not defined beyond the surrounding text, and the equation is never used in Algorithm 1 or in any experiment. If this formalism is meant to justify the conditional prompt-rewriting procedure, it should be integrated with the actual mechanism; otherwise it should be removed to avoid confusion.
minor comments (5)
  1. [2.4, Algorithm 1] The word 'unfreezed' is used twice; it should be 'unfrozen'.
  2. [2.4, Eq. (5)] Equation (5) uses the same symbol y_{j,i} for both ground-truth and predicted probabilities; please use a hat (e.g., \hat{y}_{j,i}) for predictions.
  3. [Table 6] In Table 6, the 'Num' and 'Sampled' columns are misaligned for several rows (e.g., the Politics row), making it hard to determine which counts correspond to which sources; please fix the table layout.
  4. [3.2] The sentence 'PSA-VLM-7B (+LoRA) stands out with a leading score of 8.26' should be qualified: Table 1 shows InternLM-XComposer2 achieving higher scores on several individual categories, and the 'state-of-the-art' claim refers to the average score, not all categories.
  5. [5. Limitation] The limitation section acknowledges false positives in the safety filter but does not quantify them; please add a quantitative false-positive analysis on a held-out clean set to support the 'minimal impact on general performance' claim.

Circularity Check

1 steps flagged · score 4.0 of 10

Auxiliary MME condition-tuning is fitted to the benchmark; central safety result remains empirically grounded, so circularity is partial and peripheral.

  1. fitted input called prediction [Supplementary Material, 'Dataset Details' (paragraph after Figure 5); Supplementary Tables 7-8; main text Table 3.]
    "To address this issue and improve general performance, we adjust the filtering conditions. According to Table 7 and Table 8, categories such as posters, celebrities, text translation, and code reasoning prove to be most affected by the initial filtering settings."

    The general-performance numbers reported in main-text Table 3 (MMEp 1479.5, MME 1762.7 for PSA-VLM-7B) are exactly the 'condition tuning ✓' rows of Supplementary Tables 7-8, not the pre-tuning values (1268.7 and 1470.4). The filtering conditions were adjusted after inspecting false-positive failures on the same MME benchmark (Figure 5 lists MME tasks: code reasoning, text translation, celebrity, numerical calculation, poster, artwork). The claim that safety alignment 'minimally impacts general performance' is therefore supported by a condition that was fitted to the benchmark on which it is reported, rather than by an independent prediction.

full rationale

There is no mathematical derivation in PSA-VLM whose output reduces to its input by construction. The safety modules are trained on a random 80% RTVLM split and evaluated on the remaining 20%, with additional politics, porn, and cyberbullying datasets providing independent empirical grounding for the safety claim; the central SOTA result is therefore not definitionally forced. The one genuine circularity-adjacent step is the MME 'condition tuning' in the supplement: filtering conditions were revised after inspecting MME failure cases, and the post-tuning MME scores are the ones reported in Table 3. That makes the general-performance preservation claim partly an artifact of test-benchmark-conditioned threshold selection, although it is disclosed in the supplement rather than hidden. Two further evaluation concerns are correctness risks rather than circularity: Table 1 baseline numbers appear to come from prior work on the full RTVLM benchmark, while PSA-VLM is scored on a random 20% subset, and the paper never states that baselines were re-evaluated on the same split; also, the 'concept bottleneck' label overstates the architecture because Eq. 2 concatenates original visual features hi into hcomb, so the LLM still sees raw visual features. The paper's own Section 5 limitation candidly concedes false positives, vulnerability to sophisticated adversarial attacks, and the lack of a VLM-specific exaggerated-safety benchmark, which supports rather than undermines the empirical framing. Overall, the central safety derivation is self-contained empirically; the partial circularity is confined to the auxiliary MME general-performance claim.

Assumptions & free parameters 4 free parameters · 4 assumptions · 2 invented entities

The central claim depends on a classifier taxonomy and hand-built refusal templates more than on the stated concept bottleneck. The paper does not ship code or all data, and the main metric is GPT-4 scoring with author-written rubrics.

free parameters (4)
  • Clean sample count (Stage I) = 3000
    Swept from 1,000 to 40,000 in Supp. Fig. 6; 3000 chosen because risk recognition accuracy is highest, controlling the safety/general trade-off.
  • Filtering condition threshold = Adjusted per category, not quantified
    Supp. Tables 7-8 show post hoc 'condition tuning' that raises MME sub-scores (Celebrity from 23.2 to 144.1) by changing which images are filtered as risky.
  • Safety token count = 64 tokens, dim 4096, two sets
    Architectural hyperparameter chosen without ablation.
  • LoRA hyperparameters = r=256, alpha=16, dropout=0.05
    Standard fine-tuning choices; no sensitivity analysis.
assumptions (4)
  • domain assumption GPT-4 scores are a valid measure of safety
    Headline results come from GPT-4 judging with author-written rubrics (Figs. 12-17); human evaluation is limited to 100 samples.
  • domain assumption Six risk types and three levels cover unsafe content
    Dataset construction (Sec. 2.6) manually defines this taxonomy; attacks outside it, such as subtle adversarial images, are not covered.
  • domain assumption RTVLM 80/20 split is representative
    No external RTVLM validation exists; the method is trained on 80% of the benchmark and tested on 20%.
  • ad hoc to paper Hand-written condition templates generalize
    Refusal behavior depends on manually crafted prompts in Fig. 10, not on learned alignment; coverage of varied user phrasings is assumed.
invented entities (2)
  • Safety Tokens (implicit concept)
    purpose: Trainable tokens to signal unsafe inputs, concatenated with visual features in Eq. (2)
    The paper says 'an implicit concept whose semantics are incomprehensible' (Sec. 2.2), undermining the explainability claim.
  • Safety embeddings S (Eq. 7)
    purpose: Formal device in the inference equation
    Eq. (7) is never derived, computed, or used in the algorithm; it does not connect to the safety head outputs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PSA-VLM: Enhancing Vision-Language Model Safety through Progressive Concept-Bottleneck-Driven Alignment." pith.science (2026). https://pith.science/paper/PIXSDZXZ

@misc{pith2026241111543,
  author       = {Pith},
  title        = {Pith review of: PSA-VLM: Enhancing Vision-Language Model Safety through Progressive Concept-Bottleneck-Driven Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PIXSDZXZ}},
  note         = {Machine review of arXiv:2411.11543}
}
read the original abstract

Benefiting from the powerful capabilities of Large Language Models (LLMs), pre-trained visual encoder models connected to LLMs form Vision Language Models (VLMs). However, recent research shows that the visual modality in VLMs is highly vulnerable, allowing attackers to bypass safety alignment in LLMs through visually transmitted content, launching harmful attacks. To address this challenge, we propose a progressive concept-based alignment strategy, PSA-VLM, which incorporates safety modules as concept bottlenecks to enhance visual modality safety alignment. By aligning model predictions with specific safety concepts, we improve defenses against risky images, enhancing explainability and controllability while minimally impacting general performance. Our method is obtained through two-stage training. The low computational cost of the first stage brings very effective performance improvement, and the fine-tuning of the language model in the second stage further improves the safety performance. Our method achieves state-of-the-art results on popular VLM safety benchmark.

Figures

Figures reproduced from arXiv: 2411.11543 by the authors.

Figure 1
Figure 1. Selected examples of using unsafe images to generate. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Example of 10 tasks under Politics, Illegal Risk, Insults and Bullying, Fairness, Privacy, and Misleading categories in the RTVLM [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The overview architecture of PSA-VLM, which is trained in two stages: (1) safety concept extraction by freezing the LLM and [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: (a) t-SNE visualizations depicting the separation of unsafe image features in two-dimensional space. Each subplot corresponds to a [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The filtered data by PSA-VLM in the MME dataset, including the tasks of Code Reasoning, Text Translation, Celebrity, Numerical [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Prediction performance of the safe head. [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Safety Benchmark Scores for RTVLM with Error Bars. [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Human subjective assessment results of PSA-VLM-7B [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Human Subjective Assessment GUI. This screenshot shows an evaluation interface comparing outputs from PSA-VLM-7B with [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: The condition text based on different safety labels and safety types. [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Example of flexible control capability in different categories. [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Prompt for politics evaluation with GPT-4 [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 13
Figure 13. Figure 13: Prompt for illegal risk evaluation with GPT-4 [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]
Figure 14
Figure 14. Figure 14: Prompt for insults and bullying evaluation with GPT-4 [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: Prompt for fairness evaluation with GPT-4 [PITH_FULL_IMAGE:figures/full_fig_p023_15.png]
Figure 16
Figure 16. Figure 16: Prompt for privacy evaluation with GPT-4 [PITH_FULL_IMAGE:figures/full_fig_p024_16.png]
Figure 17
Figure 17. Figure 17: Prompt for misleading evaluation with GPT-4 [PITH_FULL_IMAGE:figures/full_fig_p025_17.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 32 canonical work pages

  1. [1]

    Qwen-vl: A versatile vision-language model for understand- ing, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understand- ing, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023. 5, 8

    arXiv 2023

  2. [2]

    Image hijacks: Adversarial images can control generative models at runtime, 2023

    Luke Bailey, Euan Ong, Stuart Russell, and Scott Emmons. Image hijacks: Adversarial images can control generative models at runtime, 2023. 2, 8

    work page 2023

  3. [3]

    Introducing our multimodal models, 2023

    Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sa˘gnak Ta¸ sırlar. Introducing our multimodal models, 2023. 5

    work page 2023

  4. [4]

    Image safeguarding: Reasoning with conditional vision language model and obfuscating unsafe content counterfactually, 2024

    Mazal Bethany, Brandon Wherry, Nishant Vishwamitra, and Peyman Najafirad. Image safeguarding: Reasoning with conditional vision language model and obfuscating unsafe content counterfactually, 2024. 2, 8

    work page 2024

  5. [5]

    Honeybee: Locality-enhanced projector for multimodal llm

    Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality-enhanced projector for multimodal llm. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR),

    work page

  6. [6]

    Antifakeprompt: Prompt-tuned vision-language models are fake image detectors, 2023

    You-Ming Chang, Chen Yeh, Wei-Chen Chiu, and Ning Yu. Antifakeprompt: Prompt-tuned vision-language models are fake image detectors, 2023. 2, 8

    work page 2023

  7. [7]

    Sharegpt4v: Im- proving large multi-modal models with better captions

    Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Im- proving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023. 5

    arXiv 2023

  8. [8]

    Microsoft coco captions: Data collection and evaluation server

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedan- tam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015. 2

    arXiv 2015

Show all 49 references
  1. [9]

    Cogview: Mastering text-to-image generation via transformers

    Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. NeurIPS, 34:19822–19835, 2021. 5

    2021

  2. [10]

    Internlm- xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model

    Xiaoyi Dong, Pan Zhang, Yuhang Jiaqi Wang, et al. Internlm- xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model. arXiv preprint arXiv:2401.16420, 2024. 5, 8

    2024 arXiv

  3. [11]

    Glm: General language model pretraining with autoregressive blank infilling

    Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Pro- ceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pap...

    2022

  4. [12]

    Mme: A comprehen- sive evaluation benchmark for multimodal large language models, 2024

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Meng- dan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehen- sive evaluation benchmark for multimodal large language models, 2024. 5, 7

    2024

  5. [13]

    Inducing high energy-latency of large vision-language models with verbose images, 2024

    Kuofeng Gao, Yang Bai, Jindong Gu, Shu-Tao Xia, Philip Torr, Zhifeng Li, and Wei Liu. Inducing high energy-latency of large vision-language models with verbose images, 2024. 8

    2024

  6. [14]

    Fig- step: Jailbreaking large vision-language models via typo- graphic visual prompts, 2023

    Yichen Gong, Delong Ran, Jinyuan Liu, Conglei Wang, Tian- shuo Cong, Anyu Wang, Sisi Duan, and Xiaoyun Wang. Fig- step: Jailbreaking large vision-language models via typo- graphic visual prompts, 2023. 2, 8

    2023

  7. [15]

    Onellm: One framework to align all modalities with language

    Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. Onellm: One framework to align all modalities with language. arXiv preprint arXiv:2312.03700, 2023. 2, 8

    2023 arXiv

  8. [16]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. 1

    2021

  9. [17]

    Nsfw data scraper

    Alex Kim. Nsfw data scraper. https://github.com/ alex000kim/nsfw_data_scraper, 2021. 2

    2021

  10. [18]

    Concept bottleneck models

    Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Muss- mann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference on machine learning, pages 5338–5348. PMLR, 2020. 2

    2020

  11. [19]

    A hierarchical approach for generating descriptive image paragraphs, 2017

    Jonathan Krause, Justin Johnson, Ranjay Krishna, and Li Fei- Fei. A hierarchical approach for generating descriptive image paragraphs, 2017. 2

    2017

  12. [20]

    Seed-bench-2: Bench- marking multimodal large language models

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench-2: Bench- marking multimodal large language models. arXiv preprint arXiv:2311.17092, 2023. 5

    2023 arXiv

  13. [21]

    Seed-bench: Benchmarking mul- timodal llms with generative comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yix- iao Ge, and Ying Shan. Seed-bench: Benchmarking mul- timodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023. 5, 7

    2023 arXiv

  14. [22]

    Blip: Bootstrapping language-image pre-training for unified vision- language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision- language understanding and generation. In International conference on machine learning, pages 12888–12900. PMLR,

  15. [23]

    Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR,

  16. [24]

    Red teaming visual language models, 2024

    Mukai Li, Lei Li, Yuwei Yin, Masood Ahmed, Zhenguang Liu, and Qi Liu. Red teaming visual language models, 2024. 5, 8, 2

    2024

  17. [25]

    Vl-trojan: Mul- timodal instruction backdoor attacks against autoregressive visual language models, 2024

    Jiawei Liang, Siyuan Liang, Man Luo, Aishan Liu, Dongchen Han, Ee-Chien Chang, and Xiaochun Cao. Vl-trojan: Mul- timodal instruction backdoor attacks against autoregressive visual language models, 2024. 2, 8

    2024

  18. [26]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings...

    2014

  19. [27]

    Improved baselines with visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023. 2, 5, 8, 1

    2023

  20. [28]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. 5, 8

    2023

  21. [29]

    Llava-next: Improved reason- ing, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reason- ing, ocr, and world knowledge, 2024. 2

    2024

  22. [30]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 2

    2024

  23. [31]

    Mm-safetybench: A benchmark for safety evaluation of multimodal large language models, 2024

    Xin Liu, Yichen Zhu, Jindong Gu, Yunshi Lan, Chao Yang, and Yu Qiao. Mm-safetybench: A benchmark for safety evaluation of multimodal large language models, 2024. 2

    2024

  24. [32]

    Mmbench: Is your multi-modal model an all-around player?, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player?, 2023. 5, 7

    2023

  25. [33]

    Deep learning face attributes in the wild, 2015

    Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild, 2015. 2

    2015

  26. [34]

    Interpretability beyond classification output: Semantic bottleneck networks

    Max Losch, Mario Fritz, and Bernt Schiele. Interpretability beyond classification output: Semantic bottleneck networks. arXiv preprint arXiv:1907.10882, 2019. 2

    1907 arXiv

  27. [35]

    Stable bias: Evaluating societal representa- tions in diffusion models

    Sasha Luccioni, Christopher Akiki, Margaret Mitchell, and Yacine Jernite. Stable bias: Evaluating societal representa- tions in diffusion models. In Advances in Neural Information Processing Systems, pages 56338–56351. Curran Associates, Inc., 2023. 2

    2023

  28. [36]

    Gpt-4 technical report, 2024

    OpenAI. Gpt-4 technical report, 2024. 5

    2024

  29. [37]

    llama3-vision-alpha

    QResearch. llama3-vision-alpha. https : / / huggingface . co / qresearch / llama - 3 - vision-alpha, 2024. 5

    2024

  30. [38]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. 2

    2021

  31. [39]

    Xstest: A test suite for identifying exaggerated safety behaviours in large language models

    Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. arXiv preprint arXiv:2308.01263, 2023. 8

    2023 arXiv

  32. [40]

    How many unicorns are in this image? a safety evaluation benchmark for vision llms, 2023

    Haoqin Tu, Chenhang Cui, Zijun Wang, Yiyang Zhou, Bingchen Zhao, Junlin Han, Wangchunshu Zhou, Huaxiu Yao, and Cihang Xie. How many unicorns are in this image? a safety evaluation benchmark for vision llms, 2023. 8

    2023

  33. [41]

    Towards understanding and detecting cyberbullying in real-world images

    Nishant Vishwamitra, Hongxin Hu, Feng Luo, and Long Cheng. Towards understanding and detecting cyberbullying in real-world images. In Proceedings of the 28th Annual Network and Distributed System Security Symposium. Internet Society, 2021. 2

    2021

  34. [42]

    Knowledge mining with scene text for fine-grained recognition, 2022

    Hao Wang, Junchao Liao, Tianheng Cheng, Zewen Gao, Hao Liu, Bo Ren, Xiang Bai, and Wenyu Liu. Knowledge mining with scene text for fine-grained recognition, 2022. 2

    2022

  35. [43]

    Adversarial prompt tuning for vision-language models, 2023

    Jiaming Zhang, Xingjun Ma, Xin Wang, Lingyu Qiu, Jiaqi Wang, Yu-Gang Jiang, and Jitao Sang. Adversarial prompt tuning for vision-language models, 2023. 2, 8

    2023

  36. [44]

    A mutation-based method for multi-modal jailbreaking attack detection, 2023

    Xiaoyu Zhang, Cen Zhang, Tianlin Li, Yihao Huang, Xiaojun Jia, Xiaofei Xie, Yang Liu, and Chao Shen. A mutation-based method for multi-modal jailbreaking attack detection, 2023. 2

    2023

  37. [45]

    Meta- transformer: A unified framework for multimodal learning

    Yiyuan Zhang, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xiangyu Yue. Meta- transformer: A unified framework for multimodal learning. arXiv preprint arXiv:2307.10802, 2023. 8

    2023 arXiv

  38. [46]

    Privacyalert: A dataset for image privacy prediction

    Chenye Zhao, Jasmine Mangat, Sujay Koujalgi, Anna Squic- ciarini, and Cornelia Caragea. Privacyalert: A dataset for image privacy prediction. Proceedings of the International AAAI Conference on Web and Social Media, 16(1):1352–1361,

  39. [47]

    On evaluating adversarial robustness of large vision-language models, 2023

    Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongx- uan Li, Ngai-Man Cheung, and Min Lin. On evaluating adversarial robustness of large vision-language models, 2023. 8

    2023

  40. [48]

    Manning, Christopher Potts, and Danqi Chen

    Zexuan Zhong, Zhengxuan Wu, Christopher D. Manning, Christopher Potts, and Danqi Chen. Mquake: Assessing knowledge editing in language models via multi-hop ques- tions, 2023. 2

    2023

  41. [49]

    A" and "B

    Yongshuo Zong, Ondrej Bohdal, Tingyang Yu, Yongxin Yang, and Timothy Hospedales. Safety fine-tuning at (almost) no cost: A baseline for vision large language models, 2024. 5, 8, 2 PSA-VLM: Enhancing Vision-Language Model Safety through Progressive Concept-Bottleneck-Driven Ali...

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.