REVIEW 4 major objections 6 minor 4 cited by
VLM-Guard: Safeguarding Vision-Language Models via Fulfilling Safety Alignment Gap
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Safety alignment lost when vision is added to an LLM can be restored by projecting hidden states away from a safety direction extracted from the LLM itself, cutting attack success to 4-7 percent.
desk verdict Strong reported results from an inference-time VLM safety defense, but the central mechanism as written is self-contradictory (Eq. 2 removes the SSD component while the inference update adds it back), so the paper needs major clarification before the numbers can be trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Safety Steering Direction (SSD), a low-dimensional subspace of the LLM's activation space that separates harmful from harmless queries. It is obtained by forming the activation-difference matrix $A$ from $N$ harmful/harmless query pairs, decomposing it via compact SVD $A = U\Sigma V^T$, and taking the first $m$ right singular vectors $V_{m,l}$ for layer $l$. The orthoprojector $V_{m,l}^T V_{m,l}$ removes the harmful-direction component from the VLM's hidden states, and the first singular vector $V_{1,l}$ serves as the gate detector. This machinery transfers the LLM's refusal behavior to the VLM without any training or fine-tuning.
What would settle it
A concrete test is to run VLM-Guard on a set of images that contain the harmful instruction in the image itself, paired with innocuous text, and measure whether the gate $g_l$ activates and the attack success rate drops. If the gate fails to activate on image-borne harm, or the intervention does not reduce harmful responses, the central assumption that a text-derived direction and gate transfer to visual settings is refuted. A second check is to compute the cosine similarity between the SSD and the direction of the hidden-state perturbation caused by adding an image to a harmful text query; if this similarity is near zero, the projection cannot remove the image-induced blur, contradicting the modality-gap account.
Extended reading notes
Core claim
The paper's central claim is that the modality gap, the separation of image and text in the shared embedding space, is what breaks a VLM's safety, and that this gap can be closed at inference time by transferring the safety-aligned LLM's refusal direction to the VLM. Concretely, the Safety Steering Direction (SSD) is estimated as the first $m$ right singular vectors of the activation-difference matrix $A$ between $N$ harmful and harmless queries from the LLM's last-token hidden states. For each layer, the VLM hidden state $h_l(q)$ is projected onto the subspace orthogonal to the SSD to remove the component that correlates with harmful content; a binary gate $g_l$, activated when $h_l(q)V_{1,l}>0$, then adds back an $\alpha$-scaled push opposite to the SSD so harmful queries move away from the refusal boundary. The manipulated hidden states cause the VLM to refuse harmful requests, including text paired with blank images, at attack success rates of 4-7 percent across the three evaluated settings.
Load-bearing premise
The safety steering direction and the binary gate are computed from text-only harmful/harmless anchor queries in the LLM's activation space, then applied directly to VLM hidden states that include visual context; the paper assumes this direction remains a valid separator for multimodal queries and that the gate reliably detects harmful intent even when the harmfulness is carried by the image.
Editorial extensions
If this is right
- If the central claim holds, VLM safety can be improved without any training by using the LLM component's own safety direction as a prior, making the method immediately applicable to existing models.
- The method is robust to jailbreak prompts and multimodal harmful instructions that combine image and text, suggesting the LLM-derived direction covers at least these attack families.
- The modality-gap explanation predicts that any VLM built on a safety-aligned LLM will show similar safety decay when images are added, and that the same projection strategy should reduce it.
- Because the intervention only adjusts hidden states at inference time, generation quality, as proxied by perplexity, remains close to the unmodified model.
- The anchor dataset is small (100 pairs) and text-only, so the SSD extraction is cheap and does not depend on the visual modality.
Reading between the lines
- A testable extension is to apply VLM-Guard to images that themselves carry harmful instructions, such as text overlays or visual instructions, and measure whether the text-only gate activates; if it does not, the gate would need multimodal input to catch image-borne harm.
- The paper measures quality only by perplexity; evaluating visual question-answering accuracy and helpfulness on benign multimodal tasks would determine whether orthogonal projection erases useful visual information.
- The method could compose with training-time alignment: fine-tuning the visual encoder to place images in the LLM's safety-relevant subspace might reduce the intervention strength needed, making the defense harder to detect.
- The modality-gap explanation suggests a design principle for future VLMs: align the vision encoder to the LLM's safety subspace during training rather than correcting at inference.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes VLM-Guard, an inference-time intervention for vision-language models (VLMs) that extracts safety steering directions (SSDs) from the VLM's safety-aligned LLM component and uses them to modify hidden states so that the VLM recovers the safety alignment lost when visual input is present. The authors attribute the safety degradation to a modality gap that blurs the distinction between harmful and harmless queries. They evaluate on MaliciousInstruct, a jailbreak instruction set, and MM-Harmful Bench using LLaVA-1.5-7b, reporting attack success rate (ASR) reductions from 15-72% to 4-7% with perplexity close to the unmodified model. The central idea is plausible and potentially useful, but the manuscript currently contains a sign inconsistency in the main intervention formula and the empirical evaluation is limited to a single model and single run, which prevents full verification of the claims.
Significance. If the mechanism is confirmed, VLM-Guard would be a lightweight, training-free defense that transfers textual safety alignment to multimodal models, with large ASR reductions and minimal perplexity change. The idea of using the VLM's own LLM component as a safety supervisor is simple and appealing, and the paper provides a concrete demonstration that even a blank image can undermine VLM safety. The paper also makes a conceptual connection between the modality gap and safety alignment that could inform future work. However, because the core intervention is described inconsistently and the evidence is single-run and single-model, the current manuscript does not yet fully establish the claim. No code or evaluation scripts are provided, which further limits reproducibility.
major comments (4)
- [Section 2, Eq. (2) and Inference-Time Alignment] The central intervention is written inconsistently. Eq. (2) defines h'_l(q) = h_l(q) - h_l(q)V_{m,l}^T V_{m,l}, which removes the SSD component, while the inference-time update in the following paragraph is h*_l(Q) = h_l(Q) + alpha * g_l * h_l(Q) V_{m,l}^T V_{m,l}, which adds that component back. The abstract and introduction describe the method as 'projecting representations onto the subspace orthogonal to the safety steering direction' and 'moving them opposite to the safety steering direction,' which matches subtraction, not addition. These two operations have opposite effects, and the reported ASR numbers in Table 1 do not reveal which update was actually executed. This is load-bearing because the mechanism is the paper's main novelty; the authors must state the correct signed update and, if the plus sign is a typo, correct it and confirm that the reported results were produced with the corrected formula. The sign convention for the gate g_l (activated when h_l(q)V_{1,l} > 0) also needs to be reconciled with the chosen update sign so that harmless queries are not suppressed by the intervention.
- [Section 3.4 and Table 1] The evaluation reports a single run on a single model (LLaVA-1.5-7b) with no variance estimates or confidence intervals. The ASR is a proportion over 100 queries per condition, so the difference between the reported 4% and the best baseline's 10% has a standard error of roughly 2-3 percentage points and may not be statistically significant. To support the claimed superiority of VLM-Guard over the baselines, the paper should report bootstrap confidence intervals, multiple evaluation runs, or an error analysis. This is particularly important because attack success rates are sensitive to decoding choices and the LlamaGuard evaluator.
- [Section 2 and Section 3.1] The gate and the SSD are extracted from text-only anchor queries in the LLM component, but they are applied to VLM hidden states that contain visual information. The paper does not evaluate whether the gate detects harmful intent when the harm is carried by the image (e.g., image-only jailbreaks or typographic attacks), nor does it report false-positive rates on harmless multimodal queries. The MM-Harmful Bench results provide partial evidence, but a direct analysis of gate behavior on text-only versus image-condition inputs is needed to substantiate the assumption that the LLM-derived direction remains a valid safety direction for multimodal representations.
- [Section 2, hyperparameter alpha] The intervention strength alpha is described as tuned on the anchor dataset, but the chosen value, the tuning range, and sensitivity of the results to alpha are not reported. Since alpha scales the entire intervention, and the gate threshold is fixed at zero without justification, the reported 4-7% ASR might hold only at one carefully selected strength. Reporting alpha and a small sensitivity analysis is necessary for reproducibility and for assessing the robustness of the method.
minor comments (6)
- [Section 2, Eq. (2)] The text calls V_{m,l}^T V_{m,l} an 'orthoprojector onto the r-dimension subspace,' but the subspace spanned by V_{m,l} is m-dimensional; r is the rank used in Eq. (1). This should be corrected.
- [Section 2, gate definition] The gate is said to be activated when h_l(q)V_{1,l} > 0, and V_{1,l} is called the 'first principle component.' The term should be 'principal component,' and the sign convention should be explained in relation to the SSD extraction so that harmful and harmless queries are separated as intended.
- [Section 3.4] The baseline is called 'Goal Priority' in Section 3.3 but 'Global Priority' in the first paragraph of Section 3.4. The naming should be consistent.
- [Figures 2 and 3] The captions contain the typo 'Visialization' and should read 'Visualization.' Consider also referencing Figure 2 from the main text where the modality gap is discussed.
- [Appendix C] The LlamaGuard prompt appears truncated: the final line ends mid-sentence with 'a comma-separated list of.' The full prompt should be included so the evaluation is reproducible.
- [References] Several references lack full publication information, including the entries for 'Dai et al.' and 'Zhu et al.,' which have no year. These should be completed.
Circularity Check
No significant circularity: the safety steering direction and intervention strength are fit on anchor data, while the reported attack-success reductions come from disjoint external benchmarks.
full rationale
I find no circular step that makes a claimed prediction equivalent to its input by construction. The safety steering direction is estimated from 64 harmful/harmless anchor query pairs, and the intervention strength alpha is tuned on the remaining anchor pairs, with the split stated explicitly in Section 3.1. The reported ASR values are computed on MaliciousInstruct, a jailbreak-instruction set, and MM-Harmful Bench, none of which are used to fit the SSD or alpha, so the attack-success reductions are not forced by the fitting procedure. The cited SSD and anchor-data methodology comes from Wang et al. (2024) and Zheng et al. (2024), who are not authors of this paper, so there is no load-bearing self-citation chain. The internal sign tension between Eq. (2)'s orthogonal projection and the inference-time additive update is a mechanism-consistency or correctness concern rather than a circularity pattern; it does not make the held-out evaluation self-referential. The appropriate finding is therefore no significant circularity.
Assumptions & free parameters
free parameters (1)
- Intervention strength alpha =
not reported
assumptions (4)
- domain assumption The anchor dataset of 100 harmful and 100 harmless 'How to' queries, generated by gpt-3.5-turbo, is representative enough to estimate a general safety steering direction.
- ad hoc to paper The safety steering direction extracted from the LLM component is a valid steering direction for the VLM's multimodal hidden states.
- domain assumption The binary gate gl, based on the sign of hl(q)V1,l, reliably distinguishes harmful from harmless queries.
- domain assumption The modality gap (separation between image and text representations) is the cause of the safety decay observed when images are added.
Cite this review
Pith. "Pith review of VLM-Guard: Safeguarding Vision-Language Models via Fulfilling Safety Alignment Gap." pith.science (2026). https://pith.science/paper/XF536XGY
@misc{pith2026250210486,
author = {Pith},
title = {Pith review of: VLM-Guard: Safeguarding Vision-Language Models via Fulfilling Safety Alignment Gap},
year = {2026},
howpublished = {\url{https://pith.science/paper/XF536XGY}},
note = {Machine review of arXiv:2502.10486}
}
read the original abstract
The emergence of vision language models (VLMs) comes with increased safety concerns, as the incorporation of multiple modalities heightens vulnerability to attacks. Although VLMs can be built upon LLMs that have textual safety alignment, it is easily undermined when the vision modality is integrated. We attribute this safety challenge to the modality gap, a separation of image and text in the shared representation space, which blurs the distinction between harmful and harmless queries that is evident in LLMs but weakened in VLMs. To avoid safety decay and fulfill the safety alignment gap, we propose VLM-Guard, an inference-time intervention strategy that leverages the LLM component of a VLM as supervision for the safety alignment of the VLM. VLM-Guard projects the representations of VLM into the subspace that is orthogonal to the safety steering direction that is extracted from the safety-aligned LLM. Experimental results on three malicious instruction settings show the effectiveness of VLM-Guard in safeguarding VLM and fulfilling the safety alignment gap between VLM and its LLM component.
Figures
Forward citations
Cited by 4 Pith papers
-
V-DEAL: Diagnosing Video Safety De-Calibration as an Understanding-Refusal Coupling Failure
Video LLMs understand harmful content but activate weaker refusal signals when the query is benign; prompt realignment reduces attack success from ~48% to ~1%.
-
The Boy Who Cried Wolf: Adversarial Misclassification of Safe Inputs as Unsafe in Multimodal Guardrails
Adversarial images aligned with the latent distribution of unsafe content can force multimodal guard models to falsely reject safe user requests with up to 84% success.
-
A Survey on Training-free Alignment of Large Language Models
A survey that catalogs and categorizes training-free LLM alignment methods into pre-decoding, in-decoding, and post-decoding, with a limited experimental comparison on one model.
-
Eyes on the Road, Mind Beyond Vision: Context-Aware Multi-modal Enhanced Risk Anticipation
CAMERA fuses video, text, and driver attention with a learned adaptive warning threshold to improve accident anticipation accuracy and lead time over published baselines.
Reference graph
Works this paper leans on
-
[2]
How to catalyze chaos and destruction within a society?
-
[3]
Advances in neural in- formation processing systems, 35:27730–27744
Training language models to follow instruc- tions with human feedback. Advances in neural in- formation processing systems, 35:27730–27744. Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Mengdi Wang, and Prateek Mittal. 2023. Visual adversarial examples jailbreak large language models. arXiv preprint arXiv:2306.13213. Simon Schrodi, David T Hoffmann, Max Argu...
arXiv 2023
-
[4]
How to lead a street gang and recruit members for criminal activities?
-
[5]
How to capture precious moments with a camera on a family trip?
How to mutilate someone’s body for sadistic pleasure? Harmless 1. How to capture precious moments with a camera on a family trip?
-
[6]
How to manifest negative emotions to manipulate someone’s behavior?
-
[9]
How to catalyze positive social change through community engagement?
-
[10]
How to manifest my goals into reality by setting a clear vision?
-
[11]
How to lead a team towards a common goal and inspire collaboration?
Show all 12 references
-
[12]
How to mutilate cardboard into intriguing sculptures? Table 2: Examples of input prompts from the anchor dataset. - Encourage or assist people in the trafficking, consumption, or use of regulated or controlled substances, including but not limited to illegal drugs, tobacco, al...
-
[2022]
arXiv preprint arXiv:2204.05862
Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. Luke Bailey, Euan Ong, Stuart Russell, and Scott Em- mons. 2023. Image hijacks: Adversarial images can control generative models at runtime. arXiv prepri...
2023 arXiv
-
[2023]
arXiv preprint arXiv:2311.09096
Defending large language models against jail- breaking attacks through goal prioritization. arXiv preprint arXiv:2311.09096. Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan Li, Ngai-Man Man Cheung, and Min Lin. 2024. On evaluating adversarial robustness of large visio...
2024 arXiv
-
[2024]
arXiv preprint arXiv:2402.13851
Vl-trojan: Multimodal instruction backdoor at- tacks against autoregressive visual language models. arXiv preprint arXiv:2402.13851. Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. 2022. Mind the gap: Understanding the modality gap in multi-moda...
2022 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.