REVIEW 7 cited by
Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedback
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Multimodal large language models (MLLMs) are essential for building general-purpose AI assistants; however, they pose increasing safety risks. How can we ensure safety alignment of MLLMs to prevent undesired behaviors? Going further, it is critical to explore how to fine-tune MLLMs to preserve capabilities while meeting safety constraints. Fundamentally, this challenge can be formulated as a min-max optimization problem. However, existing datasets have not yet disentangled single preference signals into explicit safety constraints, hindering systematic investigation in this direction. Moreover, it remains an open question whether such constraints can be effectively incorporated into the optimization process for multi-modal models. In this work, we present the first exploration of the Safe RLHF-V -- the first multimodal safety alignment framework. The framework consists of: $\mathbf{(I)}$ BeaverTails-V, the first open-source dataset featuring dual preference annotations for helpfulness and safety, supplemented with multi-level safety labels (minor, moderate, severe); $\mathbf{(II)}$ Beaver-Guard-V, a multi-level guardrail system to proactively defend against unsafe queries and adversarial attacks. Applying the guard model over five rounds of filtering and regeneration significantly enhances the precursor model's overall safety by an average of 40.9%. $\mathbf{(III)}$ Based on dual preference, we initiate the first exploration of multi-modal safety alignment within a constrained optimization. Experimental results demonstrate that Safe RLHF effectively improves both model helpfulness and safety. Specifically, Safe RLHF-V enhances model safety by 34.2% and helpfulness by 34.3%.
Forward citations
Cited by 7 Pith papers
-
RoCo-ACE: Rollout-Conditioned Online Distillation for Retention-Aware Knowledge Injection
A rollout-conditioned contrastive distillation loss plus sparse anchored cross-entropy improves injected-knowledge accuracy in MLLMs while keeping retention close to the base model.
-
The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating It
LLM safety research at ACL venues from 2020 to 2024 is predominantly English-only, and the language gap is growing over time.
-
USB: A Comprehensive and Unified Safety Evaluation Benchmark for Multimodal Large Language Models
USB-SafeBench is a unified MLLM safety benchmark with 61 risk categories, 4 modality combinations, and dual-language vulnerability and oversensitivity tests.
-
HKGAI-V1: Towards Regional Sovereign Large Language Model for Hong Kong
A DeepSeek-based model fine-tuned for Hong Kong outperforms general models on Hong Kong benchmarks, but most of those benchmarks are self-authored and unreleased.
-
Generative RLHF-V: Learning Principles from Multi-modal Human Preference
A reinforcement-learned multimodal judge with grouped pairwise scoring improves vision-language model alignment on seven benchmarks.
-
Enhancing LLMs through human feedback: a journey towards self-improvement
An auxiliary feedback RAG continuously ingests classified human feedback to iteratively raise a primary RAG system’s answer accuracy and relevance.
-
A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models
A structured literature survey concluding that reasoning capabilities do not automatically make LLMs more trustworthy and can introduce new vulnerabilities in safety, robustness, and privacy.
Discussion (0). Continue with ORCID to comment.