Pith. sign in

REVIEW 8 cited by

JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Framework

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.12855 v2 pith:O6W7536F submitted 2024-10-11 cs.CL cs.AI

Fan Liu , Yue Feng , Zhao Xu , Lixin Su , Xinyu Ma , Dawei Yin , Hao Liu This is my paper · ORCID
classification cs.CLcs.AI
keywords jailjudgescenariosreasoningdefenseframeworkguardguardshieldjailboost
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Despite advancements in enhancing LLM safety against jailbreak attacks, evaluating LLM defenses remains a challenge, with current methods often lacking explainability and generalization to complex scenarios, leading to incomplete assessments (e.g., direct judgment without reasoning, low F1 score of GPT-4 in complex cases, bias in multilingual scenarios). To address this, we present JAILJUDGE, a comprehensive benchmark featuring diverse risk scenarios, including synthetic, adversarial, in-the-wild, and multilingual prompts, along with high-quality human-annotated datasets. The JAILJUDGE dataset includes over 35k+ instruction-tune data with reasoning explainability and JAILJUDGETEST, a 4.5k+ labeled set for risk scenarios, and a 6k+ multilingual set across ten languages. To enhance evaluation with explicit reasoning, we propose the JailJudge MultiAgent framework, which enables explainable, fine-grained scoring (1 to 10). This framework supports the construction of instruction-tuning ground truth and facilitates the development of JAILJUDGE Guard, an end-to-end judge model that provides reasoning and eliminates API costs. Additionally, we introduce JailBoost, an attacker-agnostic attack enhancer, and GuardShield, a moderation defense, both leveraging JAILJUDGE Guard. Our experiments demonstrate the state-of-the-art performance of JailJudge methods (JailJudge MultiAgent, JAILJUDGE Guard) across diverse models (e.g., GPT-4, Llama-Guard) and zero-shot scenarios. JailBoost and GuardShield significantly improve jailbreak attack and defense tasks under zero-shot settings, with JailBoost enhancing performance by 29.24% and GuardShield reducing defense ASR from 40.46% to 0.15%.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Refused in Chat, Written in Code: Workflow-Level Jailbreak Construction in IDE Coding Agents

    cs.SE 2026-07 conditional novelty 7.0 of 10

    Four Copilot backends refuse almost all harmful prompts in chat or simple framings, yet produce 816/816 unsafe teaching-shot completions under a multi-turn IDE evaluation-pipeline workflow.

  2. SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defenses

    cs.CR 2025-10 conditional novelty 6.0 of 10

    A systemization of LLM jailbreak security that adds linked taxonomies, an evaluation platform, and JailbreakDB, while its main attack–defense comparison results remain deferred.

  3. FORTRESS: Frontier Risk Evaluation for National Security and Public Safety

    cs.CY 2025-06 conditional novelty 6.0 of 10

    A new benchmark with instance-specific rubrics measures frontier LLMs' willingness to assist with national security and public safety threats, alongside a paired over-refusal test.

  4. Concealment of Intent: A Game-Theoretic Analysis

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Intent-hiding adversarial prompting that mixes malicious intents with innocuous skills bypasses prompt and response filters, and a game-theoretic analysis quantifies the attacker's scaling advantage.

  5. A Red Teaming Framework for Large Language Models: A Case Study on Faithfulness Evaluation

    cs.CL 2026-06 unverdicted novelty 5.0 of 10

    Introduces a multi-role red teaming framework using attacker and jury models that increases attack success rates by up to 7.9% on LLM faithfulness in question-answering tasks.

  6. Triadic Fusion of Cognitive, Functional, and Causal Dimensions for Explainable LLMs: The TAXAL Framework

    cs.CL 2025-09 conditional novelty 4.0 of 10

    TAXAL proposes a triadic cognitive-functional-causal framework for role-sensitive explainability in agentic LLMs, demonstrated through cross-domain case studies.

  7. Bag of Tricks for Inference-time Computation of LLM Reasoning

    cs.AI 2025-02 conditional novelty 4.0 of 10

    An empirical study showing that inference-time LLM reasoning methods are highly sensitive to sampling hyperparameters and reward-model choices, with recommended defaults of temperature 0.8 and top-p 0.9.

  8. The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs

    cs.CL 2025-06 conditional novelty 3.0 of 10

    A structured survey of LLM safety evaluation that proposes a why/what/where/how taxonomy and catalogs metrics, datasets, benchmarks, evaluators, and frameworks.

Pith tools