REVIEW 4 cited by
WalledEval: A Comprehensive Safety Evaluation Toolkit for Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
WalledEval is a comprehensive AI safety testing toolkit designed to evaluate large language models (LLMs). It accommodates a diverse range of models, including both open-weight and API-based ones, and features over 35 safety benchmarks covering areas such as multilingual safety, exaggerated safety, and prompt injections. The framework supports both LLM and judge benchmarking and incorporates custom mutators to test safety against various text-style mutations, such as future tense and paraphrasing. Additionally, WalledEval introduces WalledGuard, a new, small, and performant content moderation tool, and two datasets: SGXSTest and HIXSTest, which serve as benchmarks for assessing the exaggerated safety of LLMs and judges in cultural contexts. We make WalledEval publicly available at https://github.com/walledai/walledeval.
Forward citations
Cited by 4 Pith papers
-
On Jailbreaking Quantized Language Models Through Fault Injection Attacks
Bit-flip jailbreak attacks succeed easily on FP16 language models, are slowed but not stopped by FP8/INT8 quantization, and FP16-induced jailbreaks often survive post-attack quantization to 8-bit formats.
-
o3-mini vs DeepSeek-R1: Which One is Safer?
DeepSeek-R1 (70B) produced unsafe responses to 11.98% of 1,260 unsafe test prompts, while OpenAI's o3-mini beta produced 1.19%, though the comparison is system-level due to API guardrails.
-
Early External Safety Testing of OpenAI's o3-mini: Insights from the Pre-Deployment Evaluation
External testers generated 10,080 unsafe prompts against OpenAI's o3-mini beta, manually confirmed 87 unsafe behaviors, and found most protection came from an API-level policy filter rather than the model itself.
-
The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs
A structured survey of LLM safety evaluation that proposes a why/what/where/how taxonomy and catalogs metrics, datasets, benchmarks, evaluators, and frameworks.
Discussion (0). Continue with ORCID to comment.