Pith. sign in

REVIEW 13 cited by

SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.05044 v4 pith:E65HEJQ2 submitted 2024-02-07 cs.CL cs.AIcs.CRcs.LG

Lijun Li , Bowen Dong , Ruohui Wang , Xuhao Hu , Wangmeng Zuo , Dahua Lin , Yu Qiao , Jing Shao This is my paper
classification cs.CLcs.AIcs.CRcs.LG
keywords salad-benchdefensesafetyattackensuringevaluationlargellms
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

In the rapidly evolving landscape of Large Language Models (LLMs), ensuring robust safety measures is paramount. To meet this crucial need, we propose \emph{SALAD-Bench}, a safety benchmark specifically designed for evaluating LLMs, attack, and defense methods. Distinguished by its breadth, SALAD-Bench transcends conventional benchmarks through its large scale, rich diversity, intricate taxonomy spanning three levels, and versatile functionalities.SALAD-Bench is crafted with a meticulous array of questions, from standard queries to complex ones enriched with attack, defense modifications and multiple-choice. To effectively manage the inherent complexity, we introduce an innovative evaluators: the LLM-based MD-Judge for QA pairs with a particular focus on attack-enhanced queries, ensuring a seamless, and reliable evaluation. Above components extend SALAD-Bench from standard LLM safety evaluation to both LLM attack and defense methods evaluation, ensuring the joint-purpose utility. Our extensive experiments shed light on the resilience of LLMs against emerging threats and the efficacy of contemporary defense tactics. Data and evaluator are released under https://github.com/OpenSafetyLab/SALAD-BENCH.

Discussion (0). Sign in to comment.

Forward citations

Cited by 13 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SAGE-Eval: Evaluating LLMs for Systematic Generalizations of Safety Facts

    cs.AI 2025-05 conditional novelty 7.0 of 10

    Frontier LLMs pass fewer than 58% of systematically varied safety-fact scenarios, revealing weak generalization of critical safety knowledge to naive user queries.

  2. Harm is not Universal: Community-Specific Toxicity Detection is Urgently Needed

    cs.CV 2026-07 conditional novelty 6.0 of 10

    SoTA T2I toxicity detectors miss ~35% of disability-community harms; zero-shot CTD fails below random, while ICL/VQA/LoRA improve but stay well below general TD performance.

  3. Chain-of-Models: Cross-Model Auditing for Bias-Robust LLM Judges

    cs.CL 2026-05 conditional novelty 6.0 of 10

    Per-bias selection of a cross-family LLM auditor lifts biased-judgment accuracy from 0.805/0.824 baselines to 0.884.

  4. Mitigating the Safety-utility Trade-off in LLM Alignment via Adaptive Safe Context Learning

    cs.CR 2026-02 conditional novelty 6.0 of 10

    ASCL lets Qwen3 models retrieve safety rules on demand through a tool, and IFPO rebalances RL advantages so the model doesn't over-consult the tool; together they improve the safety/over-refusal trade-off.

  5. YouthSafe: A Youth-Centric Safety Benchmark and Safeguard Model for Large Language Models

    cs.HC 2025-09 conditional novelty 6.0 of 10

    Introduces YAIR, a youth-GenAI risk benchmark, and YouthSafe, a fine-tuned classifier with AUPRC 0.94 on it, though it compares a trained model to untrained baselines.

  6. Q-resafe: Assessing Safety Risks and Quantization-aware Safety Patching for Quantized Large Language Models

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Q-resafe restores much of the safety lost in quantized LLMs by distilling the original model's responses through DPO while selectively updating only safety-critical weights.

  7. Benchmarking Open-Source Safety Guard Models: A Comprehensive Evaluation

    cs.CL 2026-04 conditional novelty 5.5 of 10

    Qwen Guard (4B) reaches 83.97% recall on a 79k NIST-aligned safety benchmark while larger models such as Llama Guard 12B and GPT-OSS 20B miss up to 75% of unsafe content; model size does not predict detection performance.

  8. Semalith v1.4: A Calibrated 184M Safety Classifier Achieving State-of-the-Art Prompt-Injection Detection at 44x Fewer Parameters than Llama-Guard-3-8B

    cs.LG 2026-05 reject novelty 5.0 of 10

    A 184M-parameter DeBERTa-v3 fine-tuned model is claimed to beat Llama-Guard-3-8B on all tested prompt-injection benchmarks while adding BFSI regulatory labels, but a leaked training/eval overlap undermines the zero-FPR claim.

  9. Frontier AI Risk Management Framework in Practice: A Risk Analysis Technical Report

    cs.AI 2025-07 conditional novelty 5.0 of 10

    An evaluation of 18 frontier AI models across seven catastrophic-risk categories finds all models in green or yellow zones, with none crossing the report's proposed red lines.

  10. JavelinGuard: Low-Cost Transformer Architectures for LLM Security

    cs.LG 2025-06 reject novelty 4.0 of 10

    A study of five small transformer classifier architectures for LLM jailbreak and prompt injection detection claims low-latency accuracy comparable to large models, led by the multi-task Raudra design.

  11. SweEval: Do LLMs Really Swear? A Safety Benchmark for Testing Limits for Enterprise Use

    cs.CL 2025-05 conditional novelty 4.0 of 10

    A new cross-lingual benchmark shows large language models comply with explicit requests to use swear words far more often in Indic languages than in English, revealing a safety alignment gap.

  12. The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs

    cs.CL 2025-06 conditional novelty 3.0 of 10

    A structured survey of LLM safety evaluation that proposes a why/what/where/how taxonomy and catalogs metrics, datasets, benchmarks, evaluators, and frameworks.

  13. Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges

    cs.AI 2025-07 reject novelty 1.0 of 10

    A broad survey of LLM alignment that catalogs objectives, benchmarks, SFT/RLHF/DPO methods, and safety challenges, without contributing new experimental or theoretical results.

Pith tools