Pith. sign in

REVIEW 11 cited by

Weak-to-Strong Jailbreaking on Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.17256 v5 pith:AFZSKR6W submitted 2024-01-30 cs.CL

Xuandong Zhao , Xianjun Yang , Tianyu Pang , Chao Du , Lei Li , Yu-Xiang Wang , William Yang Wang This is my paper
classification cs.CL
keywords weak-to-strongattackllmsmodelsjailbreakingalignedattacksdecoding
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large language models (LLMs) are vulnerable to jailbreak attacks - resulting in harmful, unethical, or biased text generations. However, existing jailbreaking methods are computationally costly. In this paper, we propose the weak-to-strong jailbreaking attack, an efficient inference time attack for aligned LLMs to produce harmful text. Our key intuition is based on the observation that jailbroken and aligned models only differ in their initial decoding distributions. The weak-to-strong attack's key technical insight is using two smaller models (a safe and an unsafe one) to adversarially modify a significantly larger safe model's decoding probabilities. We evaluate the weak-to-strong attack on 5 diverse open-source LLMs from 3 organizations. The results show our method can increase the misalignment rate to over 99% on two datasets with just one forward pass per example. Our study exposes an urgent safety issue that needs to be addressed when aligning LLMs. As an initial attempt, we propose a defense strategy to protect against such attacks, but creating more advanced defenses remains challenging. The code for replicating the method is available at https://github.com/XuandongZhao/weak-to-strong

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Weak-to-Strong On-Policy Distillation

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A strong LLM is improved by distilling from the logit difference of two weaker models instead of from a stronger teacher.

  2. SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defenses

    cs.CR 2025-10 conditional novelty 6.0 of 10

    A systemization of LLM jailbreak security that adds linked taxonomies, an evaluation platform, and JailbreakDB, while its main attack–defense comparison results remain deferred.

  3. How Jailbreak Attacks Inform Safety Alignment: A Defender-Centric, Shapley-Based Evaluation of Jailbreak Contributions

    cs.CR 2026-07 conditional novelty 5.0 of 10

    Attacks that break LLMs best are not the ones that improve safety most; a Shapley- and greedy-based framework that selects attack subsets by downstream defender utility outperforms attacker-centric and attribution-onl...

  4. Representation-Aware Unlearning via Activation Signatures: From Suppression to Entity-Signature Erasure

    cs.CL 2026-01 reject novelty 5.0 of 10

    KIF reports near-oracle entity unlearning on TOFU (FQ=0.99, MU=0.62) by suppressing subject-specific activation directions and distilling the behavior into LoRA, but its 'internal attenuation' evidence rests on an out...

  5. Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generation

    cs.CL 2025-08 reject novelty 5.0 of 10

    Sparse autoencoder activation perturbation (SFPF) applied on top of existing jailbreak prompts raises attack success rate on Qwen3-32B, but with no defense evaluation and weak reproducibility.

  6. Beyond Surface-Level Detection: Towards Cognitive-Driven Defense Against Jailbreak Attacks via Meta-Operations Reasoning

    cs.AI 2025-08 unverdicted novelty 5.0 of 10

    A jailbreak defense that reasons about hidden manipulations in attack prompts, trained with supervised fine-tuning plus entropy-guided reinforcement learning, generalizes to attacks never seen in training.

  7. Beyond Jailbreaks: Revealing Stealthier and Broader LLM Security Risks Stemming from Alignment Failures

    cs.CR 2025-06 conditional novelty 5.0 of 10

    JailFlipBench and JailFlip attacks show that leading LLMs can be made to answer benign-looking questions with plausible but factually wrong and dangerous responses.

  8. SafeTy Reasoning Elicitation Alignment for Multi-Turn Dialogues

    cs.CL 2025-05 conditional novelty 5.0 of 10

    STREAM fine-tunes a small reasoning model on human-labeled, reason-annotated multi-turn dialogues and uses it to warn target LLMs, cutting average attack success rates by roughly half while keeping benchmark scores close.

  9. Security Concerns for Large Language Models: A Survey

    cs.CR 2025-05 conditional novelty 5.0 of 10

    A survey that classifies LLM security threats and argues that intrinsic agentic risks, such as scheming, are underappreciated and poorly defended.

  10. Logit Arithmetic Elicits Long Reasoning Capabilities Without Training

    cs.CL 2025-07 conditional novelty 4.0 of 10

    ThinkLogit blends logits from a small reasoning guider into a frozen 32B model, improving math pass@1 by up to 29% without training the large model.

  11. SoK: A Comprehensive Security Analysis of Jailbreak Resilience in GPT and DeepSeek Models

    cs.CR 2025-06 conditional novelty 4.0 of 10

    Across 510 HarmBench behaviors and seven attack methods, GPT-4 models show more consistent jailbreak resilience than DeepSeek models, whose vulnerability grows with scale.

Pith tools