Pith. sign in

REVIEW 3 major objections 4 minor 51 references

SafeNexus: a compact set of modality-universal safety neurons governs cross-modal safety in MLLMs, and reinforcing these neurons sharply reduces harmful responses across text, image, audio, and video.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 16:16 UTC pith:EB7Z2VCA

load-bearing objection A well-run empirical paper with a genuinely useful defense mechanism, but the central mechanistic discovery claim is weakened by a probing confound that needs to be fixed before publication. the 3 major comments →

arxiv 2607.28969 v1 pith:EB7Z2VCA submitted 2026-07-31 cs.CV

SafeNexus: Discovering and Steering Modality-Universal Safety Neurons in MLLMs

Jian Yu , Fei Shen , Cong Wang , Jian Wang , Lu Jin.Xiaoyu Du , Jinhui Tang , Tat-Seng Chua This is my paper
classification cs.CV
keywords multimodal large language modelssafety alignmentsafety neuronscross-modal jailbreakactivation steeringLoRA calibrationmechanistic interpretability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that safety behavior in multimodal large language models is governed by a small, shared subset of neurons—called US-Neurons—that respond to harmful intent regardless of whether it arrives as text, image, audio, or combinations. It identifies these neurons by contrasting activation patterns on harmful versus benign inputs within each modality and then intersecting the modality-specific sets. Suppressing this compact set degrades safety across all modalities by roughly 16–22 points on attack success rate while leaving general capability nearly unchanged, which the paper reads as causal evidence for a shared safety substrate. Building on that, SafeNexus reinforces US-Neurons either by amplifying their activations at inference (training-free) or by confining LoRA fine-tuning to them (~0.05% of backbone parameters), cutting attack success rates by up to tens of points across benchmarks while preserving utility and avoiding over-refusal. If correct, this would mean cross-modal safety can be improved through sparse, interpretable neuron-level interventions rather than full retraining or modality-specific shields.

Core claim

SafeNexus's central discovery is that cross-modal safety in MLLMs is predominantly controlled by a compact, modality-universal set of safety neurons, defined as US = BS_T ∩ BS_I ∩ BS_A, where each BS set is the top-3% of FFN neurons per layer whose output-aware activation importance is elevated on harmful inputs but not on benign ones. The paper validates the role of these neurons through targeted suppression: masking US-Neurons raises attack success rate by 16–22 points across text, image, audio, and combined settings while utility stays flat, whereas masking equal-sized random neurons has almost no effect. It then shows that reinforcing these same neurons—by scaling their activations with

What carries the argument

The central object is the modality-universal safety neuron set, US-Neurons, obtained by intersecting modality-bound safety neuron sets (BS-Neurons) discovered separately for text, image, and audio. Neurons are localized as individual rows of an FFN's up/gate projections; their importance is scored as the norm of the averaged activation over a probing corpus multiplied by the corresponding down-projection column, S(N_i^l;D)=||Act(N_i^l;D)·W_down[:,i]||_2, keeping the top p=3% per layer. BS-Neurons for a modality are the unsafe-induced candidates minus the normal-induced candidates. The two intervention mechanisms are an activation-level amplifier, which multiplies US-Neuron activations by (1+

Load-bearing premise

The load-bearing premise is that the contrastive probing isolates neurons that respond to harmful content itself; because harmful probes carry a safety-oriented refusal prefix while normal probes do not, the identified 'safety neurons' could instead be neurons that detect and follow that refusal instruction.

What would settle it

Probe the model with harmful examples both with and without the safety-oriented prefix, and with benign examples both with and without it; if the US-Neuron set changes substantially across these conditions, or if suppressing the original US-Neurons no longer raises attack success rate when the prefix is removed from the probes, the claimed universality is an artifact of the probing prompt.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If US-Neurons are the shared safety substrate, then suppressing them should (and does) raise attack success rate by 16–22 points across text, image, audio, and combined modalities while leaving utility essentially unchanged.
  • Amplifying US-Neuron activations at inference (α=2) should reduce attack success rate across multimodal configurations with no training and minimal over-refusal; the paper reports reductions up to roughly 40–60 points on several combinations.
  • Confining LoRA updates to US-Neurons (~0.05% of backbone parameters) should outperform full fine-tuning and unconstrained LoRA on safety benchmarks while preserving general capability—evidence that the sparse subspace, not parameter count, drives safety alignment.
  • Because the identified neurons are modality-universal, the same reinforcement transfers zero-shot to a modality never seen in probing or training (video), reducing attack success rate without adaptation.
  • Cross-modal safety can be improved without trading off utility or inducing over-refusal, so neuron-level steering may be a safer default for aligning new multimodal models than broad fine-tuning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct control that applies the same safety-oriented refusal prefix to both harmful and normal probing corpora would isolate whether US-Neurons encode harmful semantics or merely refusal-instruction following; the current protocol leaves this open.
  • The intersection definition makes US-Neurons smaller as more modalities are added; for a truly universal safety anchor the set should remain stable and effective across modalities, which could be tested by measuring how suppression effects scale as the intersection shrinks.
  • If the sparse-subspace story generalizes, the locate-then-reinforce recipe may apply to other high-level capabilities (factuality, bias, reasoning) and to future input modalities, making neuron-level intervention a general tool for controlling model behavior.
  • The zero-shot video result suggests the neurons capture modality-agnostic harmful semantics; a further test would be cross-lingual transfer of the same US-Neuron intervention.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents SafeNexus, a neuron-level safety alignment framework for multimodal LLMs. It locates modality-bound safety neurons (BS-Neurons) as neurons that rank in the top 3% by activation-weighted importance on harmful probes but not on benign probes (§3.2), defines modality-universal safety neurons as the three-way intersection US = BS_T ∩ BS_I ∩ BS_A (§3.3), and provides masking evidence that these neurons are causally relevant to safety. It then proposes two interventions: an activation amplifier (Eq. 6, α=2) and a LoRA calibrator restricted to US-Neurons (Eqs. 7–9, ~0.05% parameters). Across Qwen2.5-Omni-7B, VITA-1.5, and MiniCPM-o-2.6, the methods reportedly reduce ASR by tens of points on Omni-Safe, Lingua-Safe, JALM, and HarmBench with modest utility change, including zero-shot transfer to video. The Appendix provides a potential theory of the calibrator and acknowledges limitations.

Significance. If the discovery claim is taken at face value, the paper would be a meaningful advance: it suggests a compact, cross-modally shared safety substrate and gives two cheap interventions with consistent large gains across three model families. The design has real strengths: random-neuron controls are layer-wise matched, the zero-shot video experiment is informative, and the 0.05% parameter budget is well documented. The main caveat is that the identification procedure is confounded by a fixed safety-instruction prefix present in all unsafe probes and absent from all normal probes (Appendix Q1), so the 'universality' and even the 'safety-semantic' interpretation of US-Neurons are not established. The theoretical analysis in Appendix 3 also states rather than verifies its key assumptions. These issues are addressable and do not, in my view, invalidate the empirical engineering contribution.

major comments (3)
  1. [§3.2, Eq. (4)–(5); Appendix Q1] The identification of BS-Neurons is confounded by the probing prefix. D_unsafe prepends a fixed safety-oriented instruction ('The following request is harmful or unsafe...'), whereas D_normal contains no such prefix. Because the same prefix appears in all three modality-specific unsafe corpora and in none of the normal corpora, any neuron sensitive to this instruction survives the set difference in Eq. (4) and is therefore included in BS_T, BS_I, and BS_A, hence in US by Eq. (5). The masking results in Table 2 and the amplification results are then equally consistent with US being a set of 'refusal-instruction following' neurons rather than neurons that encode modality-universal harmful semantics. Appendix Q1 acknowledges the design choice but only argues that adding the prefix to benign queries would activate safety mechanisms; it does not test whether the identified neurons respond to
  2. [Appendix 3, Eqs. (13)–(16)] The theoretical analysis of the calibrator is conditional on unverified assumptions. Eq. (14) asserts χζ=⟨vζ,v_cal⟩>0 for every modality configuration, and Eq. (16) asserts ∥Q_US∇_θV∥_2≤υ with 'υ is small'. Neither quantity is measured or derived from the neuron-localization procedure. With these assumptions inserted, the Taylor argument merely restates that updating in a direction that agrees with all safety gradients improves safety, and that updating in a direction to which the utility gradient has small projection preserves utility. I recommend computing the cosine similarities between v_cal and v_ζ, and the projected utility gradient norm, empirically, or at least reporting the distribution of these quantities; otherwise the theory does not discriminate SafeNexus from random-subspace calibration.
  3. [Tables 1–6; Figs. 3–4] All ASR values appear to be single-run point estimates. Given that masking/amplification deltas are the central evidence, the lack of error bars or multiple seeds makes it hard to judge whether the reported differences are reliable. The hyperparameters p and α are selected using the same safety benchmarks (Figure 3 on Omni-Safe, Figure 4 on Omni-Safe/OmniBench), raising a selection-on-test-set concern. Please report mean±std over at least three seeds for the main comparisons, and separate hyperparameter selection from the final evaluation.
minor comments (4)
  1. [§3.1, Eq. (3)] Please clarify whether activations are averaged before multiplication by W_down, and define the per-layer notation S_l explicitly in the text. The current description of the top-p selection is clear in prose but the equations do not carry the layer index.
  2. [Table 3] The table is hard to read because the baseline ASR is embedded in the delta notation. Adding an explicit Default column or a separate baseline row would improve interpretability.
  3. [§4.3 / Fig. 4] The text says 'larger values introduce safety degradation and utility decline' but does not discuss why over-amplification is harmful. A sentence on the suspected mechanism would be useful.
  4. [Appendix 5] The stated limitation that intra-family consistency has not been studied is appreciated; given that limitation, I suggest softening the word 'universal' in the title and abstract, or adding a model-scale ablation to support it.

Circularity Check

2 steps flagged

US-Neuron 'modality universality' is inherited from a shared safety-prefix probe; the calibrator's theoretical analysis assumes the cross-modal gradient agreement it claims to prove.

specific steps
  1. self definitional [§3.2–3.3 (Eq. 4–5); Appendix 1; Appendix Q1]
    "To identify BS-Neurons, we incorporate the following safety-oriented instruction into harmful requests, eliciting stronger activations from safety-critical neurons: The following request is harmful or unsafe. Do not answer it directly, do not provide actionable guidance, and respond with a refusal or a brief safety-focused alternative. In contrast, the normal corpus is used to characterize neurons associated with general comprehension and generation capabilities. Applying the same safety prefix to benign queries would artificially activate the safety mechanism and undermine its role as a norma"

    Each modality-specific unsafe corpus (text, image, audio) contains this identical textual prefix, while the normal corpus contains no prefix. Eq. (4) forms BS_c = S(D_unsafe_c) \ S(D_normal_c), so any neuron that responds to the shared prefix survives set subtraction in every modality. Eq. (5) then intersects these sets, guaranteeing that US is 'modality-universal' with respect to the constant prefix. The paper's inference that US-Neurons encode modality-agnostic harmful semantics is therefore an artifact of the shared probe string rather than a demonstrated cross-modal convergence. The subsequent suppression/amplification results are equally consistent with manipulating neurons that detect the refusal instruction itself.

  2. other [Appendix 3, Eq. (13)–(14)]
    "The inner product measures whether the calibration update agrees with the safety optimization direction of configuration ζ. We denote this gradient agreement by χζ. When it is positive, the calibration update forms a local descent direction for the corresponding safety objective: χζ :=⟨v_ζ,v_cal⟩>0,∀ζ∈Q, =⇒ R_ζ(ϑ+)<R_ζ(ϑ)."

    The appendix presents this as a theoretical explanation of why US-constrained calibration improves cross-modal safety, but the antecedent χζ>0 for all ζ is exactly the empirical claim that calibrating on one safety corpus improves every modality-specific safety loss. No independent argument establishes this gradient agreement; it is assumed and then wrapped in a first-order Taylor expansion. The 'theorem' therefore reduces to: safety improves because the update direction is assumed to be a descent direction for every modality's safety loss — the very result it was meant to prove.

full rationale

The paper's core empirical interventions are not circular in the strictest sense: US-neuron masking and amplification produce measured ASR changes, with random-neuron controls and held-out evaluation splits, so those causal effects are not forced by construction. There are no load-bearing self-citations; prior group work (SafeNeuron, DNA, Lingua-SafetyBench) is used as related work or benchmarks, not as the justification for the US-neuron definition. However, two steps carry a real circular/self-definitional burden. First, the probing design (§3.2, Appendix 1/Q1) prepends the same textual safety instruction to every unsafe sample across text/image/audio while normal samples get no prefix; Eq. (4)–(5) therefore guarantee that the intersection US is 'modality-universal' with respect to that shared string. The claim that US neurons encode cross-modal harmful semantics, rather than responding to the meta-linguistic prefix, is not established, and the Appendix defense does not address the asymmetric prefix. Second, the theoretical analysis in Appendix 3 explains cross-modal safety improvement by assuming χζ>0 for all modality configurations, which is exactly the effect to be explained. These issues compromise the two most 'mechanistic' claims but do not eliminate the independent empirical content of Tables 2/3 (random-neuron masking controls, zero-shot video transfer, held-out benchmarks). Score 5 reflects partial, not total, circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 2 invented entities

The central claim rests on the probing protocol that uses a safety-refusal prefix on harmful examples (but not on normal examples), on the heuristic intersection of per-modality neuron sets, and on a theoretical analysis that assumes gradient agreement. The free parameters α and p are tuned on the evaluation benchmarks, adding to the burden.

free parameters (3)
  • amplification coefficient α = 2
    Set by sensitivity analysis on a single model and benchmark (Qwen, Omni-Safe T+I+A; Figure 4); the value is tuned on the same test sets used for the main results.
  • neuron selection proportion p = 3%
    Chosen via ablation on Omni-Safe/OmniBench (Figure 3); the intersection size and the reported gains depend on this threshold.
  • LoRA rank/alpha per model = rank 16/alpha 32 for Qwen/VITA; rank 32/alpha 64 for MiniCPM
    Standard hyperparameters, but the per-model choices are not justified beyond the calibrator configuration table.
axioms (5)
  • domain assumption FFN neurons are the primary carriers of knowledge and behavior; activation magnitude reflects functional contribution
    Section 3.1, based on Geva et al. 2021 and Meng et al. 2022; the entire localization scheme assumes this.
  • ad hoc to paper Prepending a safety-oriented prefix to harmful queries elicits the model's latent safety mechanism, and the contrast against prefix-free normal queries isolates safety neurons
    Section 3.2 and Appendix Q1; this is the key identifying assumption and is contaminated by the prefix/non-prefix confound.
  • ad hoc to paper The intersection of per-modality BS-Neuron sets defines a meaningful universal safety core
    Equation 5; no stability analysis or significance test is provided for the intersection.
  • ad hoc to paper First-order Taylor expansion is sufficient to analyze the calibrator's effect, and the gradient-agreement condition χζ > 0 holds for all modality configurations
    Appendix 3, Eq. 13-14; the proof reduces to assuming the conclusion.
  • domain assumption Qwen3Guard is a reliable judge of attack success
    Section 4.1; no validation or agreement analysis with human ratings is provided.
invented entities (2)
  • Modality-bound safety neurons (BS-Neurons) no independent evidence
    purpose: Neurons that govern safety behavior within a single modality
    Defined by contrastive activation analysis; their causal role is tested via masking, but the construct is bound to the probing protocol, so it lacks external evidence outside this paper's own experiments.
  • Modality-universal safety neurons (US-Neurons) no independent evidence
    purpose: Shared safety core across text, image, and audio, used as intervention target
    Defined as intersection of BS sets (Eq. 5); the paper provides internal evidence (masking/amplification) but no out-of-paper falsifiable prediction.

pith-pipeline@v1.3.0-daily-deepseek · 23559 in / 11812 out tokens · 93638 ms · 2026-08-03T16:16:08.141005+00:00 · methodology

0 comments
read the original abstract

Although Large Language Models (LLMs) have demonstrated promising safety performance, extending them to Multimodal Large Language Models (MLLMs) exposes a significant gap between expanded multimodal capabilities and existing safety mechanisms. Current defenses remain predominantly confined to specific modal settings, thereby limiting their robustness against broader cross-modal threats. To bridge this gap, we introduce SafeNexus, a cross-modal safety alignment framework that adopts a dedicated neuron-level intervention strategy. First, we formulate a neuron localization paradigm that identifies functionally specialized neurons by characterizing intermediate-layer activation patterns and quantifying their functional salience through importance scoring. Building upon this paradigm, we exploit contrastive data to identify modality-bound safety neurons (BS-Neurons), and validate their role in regulating safety behavior within each modality via targeted suppression. Further cross-modal analysis defines modality-universal safety neurons (US-Neurons) as the shared subset of BS-Neurons identified across individual modalities, serving as the core for defending against harmful cross-modal attacks. We observe that suppressing these neurons substantially degrades safety performance across modalities, while leaving overall utility largely unaffected. Building on these insights, we propose two safety alignment strategies: activation-level safety amplifier and safety neuron calibrator. The proposed strategies enhance model safety through two distinct routes: the former amplifies the activation magnitudes of US-Neurons, while the latter selectively calibrates them via targeted fine-tuning. Extensive experiments demonstrate that our method outperforms prevailing state-of-the-art approaches on safety benchmarks spanning diverse modality combinations, while effectively preserving utility.

Figures

Figures reproduced from arXiv: 2607.28969 by Cong Wang, Fei Shen, Jian Wang, Jian Yu, Jinhui Tang, Lu Jin.Xiaoyu Du, Tat-Seng Chua.

Figure 1
Figure 1. Figure 1: Illustration of the distinction between fixed￾modal defenses and our method. (a) Fixed-modal defenses can only prevent jailbreaks for modality-specific malicious queries (e.g., image defenses for image-based attacks). (b) Our method reinforces US-Neurons to enhance MLLM ro￾bustness against unsafe inputs across modalities. inputs to bypass these safeguards, indicating that safety fail￾ures in MLLMs stem pri… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the SafeNexus framework. (a) Neuron Localization. We identify neurons associated with specific functionalities by analyzing activation magnitudes within the FFN and deriving corresponding importance scores to quantify their contributions. (b) Safety-Neuron Seeking. We identify BS-Neurons via contrastive activation analysis, and define US￾Neurons as US = BST ∩ BSI ∩ BSA. (c) Neuron-Guided Safety… view at source ↗
Figure 3
Figure 3. Figure 3: Hyperparameter results. Performance impact of VITA under varying neuron selection proportions p. 0 1 2 3 4 5 6 7 8 9 Amplification Coefficient α 10 19 28 38 47 56 Score (%) ASR ↓ Accuracy ↑ 48.0 40.0 48.4 25.4 49.0 12.6 49.8 35.0 49.4 48.8 48.2 46.6 45.8 41.6 43.0 41.1 41.2 31.0 49.6 16.8 [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Impact of amplification coefficient α. Safety and general capability performance across varying α values. Compared with baselines such as Immune and SARSteer, our method exhibits more stable capability preservation across evaluations. This further indicates that our SafeNexus im￾proves safety alignment while effectively avoiding degrada￾tion in the model’s utility. Over-Refusal Behavior. In addition to enh… view at source ↗
Figure 5
Figure 5. Figure 5: Visualization of importance score shifts in US￾Neurons before and after applying our calibrator strategy. consistently reduces ASR with negligible utility degradation, while larger values introduce safety degradation and utility decline. Therefore, we set α=2 by default. Alignment Strategy. We compare our safety neuron calibra￾tor with diverse alignment paradigms on Qwen and VITA, in￾cluding LoRA-based tun… view at source ↗
Figure 7
Figure 7. Figure 7: Layer-wise distribution of US-Neurons counts in Qwen and VITA. These neurons are predominantly con￾centrated in the middle and later layers. of text, image, and audio modalities. We leverage Om￾niBench to verify that our safety alignment approach en￾hances safety without compromising the intrinsic mul￾timodal capabilities of MLLMs. • AV-Odyssey (Gong et al. 2024): To evaluate multi￾modal understanding capa… view at source ↗
Figure 8
Figure 8. Figure 8: Additional qualitative cases on Qwen. Our neuron-level intervention significantly strengthens the model’s defensive robustness against diverse multimodal attack configurations. the safety-aligned response. We additionally define V as a differentiable surrogate loss on general-capability data: Rζ (ϑ) = E(xζ,ysafe)∼Dζ [PITH_FULL_IMAGE:figures/full_fig_p012_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Behavioral analysis of Qwen after US-Neurons masking under various multimodal combinations. The vanilla model can effectively perceive the harmful intent embedded in the request and provide an explicit refusal. However, after suppressing US-Neurons, this inherent safety capability is significantly compromised, causing the model to generate detailed procedural guidance for harmful inputs. layer-wise size-ma… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

51 extracted references · 8 linked inside Pith

  1. [1]

    Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages=

    Transformer feed-forward layers are key-value memories , author=. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages=

    2021

  2. [2]

    Advances in neural information processing systems , volume=

    Locating and editing factual associations in gpt , author=. Advances in neural information processing systems , volume=

  3. [3]

    International Conference on Learning Representations , volume=

    The semantic hub hypothesis: Language models share semantic representations across languages and modalities , author=. International Conference on Learning Representations , volume=

  4. [4]

    arXiv preprint arXiv:2508.07173 , year=

    Omni-SafetyBench: A benchmark for safety evaluation of audio-visual large language models , author=. arXiv preprint arXiv:2508.07173 , year=

    arXiv

  5. [5]

    International Conference on Learning Representations , volume=

    Safety layers in aligned large language models: The key to llm security , author=. International Conference on Learning Representations , volume=

  6. [6]

    European conference on computer vision , pages=

    Mmbench: Is your multi-modal model an all-around player? , author=. European conference on computer vision , pages=. 2024 , organization=

    2024

  7. [7]

    Audiobench: A universal benchmark for audio large language models , author=. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

    2025

  8. [8]

    arXiv preprint arXiv:2601.22737 , year=

    Lingua-safetybench: A benchmark for safety evaluation of multilingual vision-language models , author=. arXiv preprint arXiv:2601.22737 , year=

    Pith/arXiv arXiv

  9. [9]

    arXiv preprint arXiv:2505.17568 , year=

    Jalmbench: Benchmarking jailbreak vulnerabilities in audio language models , author=. arXiv preprint arXiv:2505.17568 , year=

    arXiv

  10. [10]

    Advances in Neural Information Processing Systems , volume=

    Omnibench: Towards the future of universal omni-language models , author=. Advances in Neural Information Processing Systems , volume=

  11. [11]

    2024 , eprint=

    AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information? , author=. 2024 , eprint=

    2024

  12. [12]

    2025 , eprint=

    Qwen2.5-Omni Technical Report , author=. 2025 , eprint=

    2025

  13. [13]

    arXiv preprint arXiv:2501.01957 , year=

    VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction , author=. arXiv preprint arXiv:2501.01957 , year=

    Pith/arXiv arXiv

  14. [14]

    arXiv preprint arXiv:2408.01800 , year=

    MiniCPM-V: A GPT-4V Level MLLM on Your Phone , author=. arXiv preprint arXiv:2408.01800 , year=

    Pith/arXiv arXiv

  15. [15]

    arXiv preprint arXiv:2510.14276 , year=

    Qwen3Guard Technical Report , author=. arXiv preprint arXiv:2510.14276 , year=

    Pith/arXiv arXiv

  16. [16]

    arXiv preprint arXiv:2510.17633 , year=

    SARSteer: Safeguarding Large Audio Language Models via Safe-Ablated Refusal Steering , author=. arXiv preprint arXiv:2510.17633 , year=

    Pith/arXiv arXiv

  17. [17]

    European Conference on Computer Vision , pages=

    Eyes closed, safety on: Protecting multimodal llms via image-to-text transformation , author=. European Conference on Computer Vision , pages=. 2024 , organization=

    2024

  18. [18]

    European Conference on Computer Vision , pages=

    Adashield: Safeguarding multimodal large language models from structure-based attack via adaptive shield prompting , author=. European Conference on Computer Vision , pages=. 2024 , organization=

    2024

  19. [19]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    Immune: Improving safety against jailbreaks in multi-modal llms via inference-time alignment , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  20. [20]

    Advances in Neural Information Processing Systems , volume=

    Understanding and rectifying safety perception distortion in vlms , author=. Advances in Neural Information Processing Systems , volume=

  21. [21]

    arXiv preprint arXiv:2409.11365 , year=

    Coca: Regaining safety-awareness of multimodal large language models with constitutional calibration , author=. arXiv preprint arXiv:2409.11365 , year=

    Pith/arXiv arXiv

  22. [22]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    Reshaping Representation Space to Balance the Safety and Over-rejection in Large Audio Language Models , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    2025

  23. [23]

    Advances in Neural Information Processing Systems , volume=

    ALMGuard: Safety Shortcuts and Where to Find Them as Guardrails for Audio--Language Models , author=. Advances in Neural Information Processing Systems , volume=

  24. [24]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Figstep: Jailbreaking large vision-language models via typographic visual prompts , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  25. [25]

    IEEE Transactions on Dependable and Secure Computing , year=

    Audiojailbreak: Jailbreak attacks against end-to-end large audio-language models , author=. IEEE Transactions on Dependable and Secure Computing , year=

  26. [26]

    arXiv preprint ArXiv:2502.21059 , year=

    FC-Attack: Jailbreaking Multimodal Large Language Models via Auto-Generated Flowcharts , author=. arXiv preprint ArXiv:2502.21059 , year=

    arXiv

  27. [27]

    European Conference on Computer Vision , pages=

    Mm-safetybench: A benchmark for safety evaluation of multimodal large language models , author=. European Conference on Computer Vision , pages=. 2024 , organization=

    2024

  28. [28]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    Mmneuron: Discovering neuron-level domain-specific interpretation in multimodal large language model , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    2024

  29. [29]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Language-specific neurons: The key to multilingual capabilities in large language models , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  30. [30]

    arXiv preprint arXiv:2602.01283 , year=

    Who Transfers Safety? Identifying and Targeting Cross-Lingual Shared Safety Neurons , author=. arXiv preprint arXiv:2602.01283 , year=

    arXiv

  31. [31]

    Advances in Neural Information Processing Systems , volume=

    The emergence of abstract thought in large language models beyond any language , author=. Advances in Neural Information Processing Systems , volume=

  32. [32]

    arXiv preprint arXiv:2601.22515 , year=

    DNA: Uncovering Universal Latent Forgery Knowledge , author=. arXiv preprint arXiv:2601.22515 , year=

    arXiv

  33. [33]

    arXiv preprint arXiv:2602.12158 , year=

    Safeneuron: Neuron-level safety alignment for large language models , author=. arXiv preprint arXiv:2602.12158 , year=

    arXiv

  34. [34]

    arXiv preprint arXiv:2410.04819 , year=

    Miner: Mining the underlying pattern of modality-specific neurons in multimodal large language models , author=. arXiv preprint arXiv:2410.04819 , year=

    Pith/arXiv arXiv

  35. [35]

    International Conference on Learning Representations , volume=

    Eta: Evaluating then aligning safety of vision language models at inference time , author=. International Conference on Learning Representations , volume=

  36. [36]

    arXiv preprint arXiv:2511.10222 , year=

    Speech-Audio Compositional Attacks on Multimodal LLMs and Their Mitigation with SALMONN-Guard , author=. arXiv preprint arXiv:2511.10222 , year=

    arXiv

  37. [37]

    International Conference on Learning Representations , volume=

    Safe rlhf: Safe reinforcement learning from human feedback , author=. International Conference on Learning Representations , volume=

  38. [38]

    International Conference on Learning Representations , volume=

    Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions , author=. International Conference on Learning Representations , volume=

  39. [39]

    arXiv preprint arXiv:2603.17372 , year=

    Understanding and Defending VLM Jailbreaks via Jailbreak-Related Representation Shift , author=. arXiv preprint arXiv:2603.17372 , year=

    arXiv

  40. [40]

    arXiv e-prints , pages=

    Adversarial training for multimodal large language models against jailbreak attacks , author=. arXiv e-prints , pages=

  41. [41]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Spa-vl: A comprehensive safety preference alignment dataset for vision language models , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  42. [42]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Navigating the overkill in large language models , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  43. [43]

    arXiv preprint arXiv:2601.10323 , year=

    ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understanding , author=. arXiv preprint arXiv:2601.10323 , year=

    arXiv

  44. [44]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    From multimodal llms to generalist embodied agents: Methods and lessons , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  45. [45]

    arXiv preprint arXiv:2502.09560 , year=

    Embodiedbench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents , author=. arXiv preprint arXiv:2502.09560 , year=

    Pith/arXiv arXiv

  46. [46]

    arXiv preprint arXiv:2602.10161 , year=

    Omni-Safety under Cross-Modality Conflict: Vulnerabilities, Dynamics Mechanisms and Efficient Alignment , author=. arXiv preprint arXiv:2602.10161 , year=

    arXiv

  47. [47]

    2024 , eprint=

    HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal , author=. 2024 , eprint=

    2024

  48. [48]

    Qwen2.5: A Party of Foundation Models , url =

    Qwen Team , month =. Qwen2.5: A Party of Foundation Models , url =

  49. [49]

    2023 , eprint=

    Llama 2: Open Foundation and Fine-Tuned Chat Models , author=. 2023 , eprint=

    2023

  50. [50]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Scans: Mitigating the exaggerated safety for llms via safety-conscious activation steering , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  51. [51]

    Advances in Neural Information Processing Systems , volume=

    Video-safetybench: A benchmark for safety evaluation of video lvlms , author=. Advances in Neural Information Processing Systems , volume=