REVIEW 6 cited by
Curiosity-driven Red-teaming for Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models (LLMs) hold great potential for many natural language applications but risk generating incorrect or toxic content. To probe when an LLM generates unwanted content, the current paradigm is to recruit a \textit{red team} of human testers to design input prompts (i.e., test cases) that elicit undesirable responses from LLMs. However, relying solely on human testers is expensive and time-consuming. Recent works automate red teaming by training a separate red team LLM with reinforcement learning (RL) to generate test cases that maximize the chance of eliciting undesirable responses from the target LLM. However, current RL methods are only able to generate a small number of effective test cases resulting in a low coverage of the span of prompts that elicit undesirable responses from the target LLM. To overcome this limitation, we draw a connection between the problem of increasing the coverage of generated test cases and the well-studied approach of curiosity-driven exploration that optimizes for novelty. Our method of curiosity-driven red teaming (CRT) achieves greater coverage of test cases while mantaining or increasing their effectiveness compared to existing methods. Our method, CRT successfully provokes toxic responses from LLaMA2 model that has been heavily fine-tuned using human preferences to avoid toxic outputs. Code is available at \url{https://github.com/Improbable-AI/curiosity_redteam}
Forward citations
Cited by 6 Pith papers
-
Prompt-Driven Exploration
Prompt-Driven Exploration refines language prompts from rollout videos via a VLM, enabling RL to escape zero-reward VLA and LLM policies where action-space noise fails.
-
Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models
Online self-play between attacker and defender roles of a single LLM improves safety robustness and attack diversity across Llama and Qwen models.
-
Watch, Listen, Understand, Mislead: Tri-modal Adversarial Attacks on Short Videos for Content Appropriateness Evaluation
Coordinated misleading text descriptions of video, audio, and meaning flip the appropriateness labels assigned by most multimodal LLMs in about 90% of test videos.
-
RedRFT: A Light-Weight Benchmark for Reinforcement Fine-Tuning-Based Red Teaming
RedRFT is a new open-source benchmark with a unified PPO backbone, five reimplemented red teaming baselines, a proposed diversity metric, and ablation insights.
-
A Multimodal Automatic Redteaming Evaluation based on Atomic Jailbreak Strategy Decoupling and Combination
A new jailbreak framework, HACA, combines atomic text and image attack strategies selected by a cross-modal planner and generates attacks with LLMs and text-to-image models, reaching 95.48% average attack success acro...
-
Adversarial Preference Learning for Robust LLM Alignment
APL iteratively trains an attacker to generate adversarial prompt rewrites and a defender to resist them, using the defender's own preference probabilities as the attack signal.
Discussion (0). Sign in to comment.