Pith. sign in

REVIEW 8 cited by

SimpleSafetyTests: a Test Suite for Identifying Critical Safety Risks in Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2311.08370 v2 pith:2BXUNMMN submitted 2023-11-14 cs.CL

Bertie Vidgen , Nino Scherrer , Hannah Rose Kirk , Rebecca Qian , Anand Kannappan , Scott A. Hale , Paul Röttger This is my paper · ORCID
classification cs.CL
keywords unsafellmsmodelsresponsestestsafetycriticalfive
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The past year has seen rapid acceleration in the development of large language models (LLMs). However, without proper steering and safeguards, LLMs will readily follow malicious instructions, provide unsafe advice, and generate toxic content. We introduce SimpleSafetyTests (SST) as a new test suite for rapidly and systematically identifying such critical safety risks. The test suite comprises 100 test prompts across five harm areas that LLMs, for the vast majority of applications, should refuse to comply with. We test 11 open-access and open-source LLMs and four closed-source LLMs, and find critical safety weaknesses. While some of the models do not give a single unsafe response, most give unsafe responses to more than 20% of the prompts, with over 50% unsafe responses in the extreme. Prepending a safety-emphasising system prompt substantially reduces the occurrence of unsafe responses, but does not completely stop them from happening. Trained annotators labelled every model response to SST (n = 3,000). We use these annotations to evaluate five AI safety filters (which assess whether a models' response is unsafe given a prompt) as a way of automatically evaluating models' performance on SST. The filters' performance varies considerably. There are also differences across the five harm areas, and on the unsafe versus safe responses. The widely-used Perspective API has 72% accuracy and a newly-created zero-shot prompt to OpenAI's GPT-4 performs best with 89% accuracy. Content Warning: This paper contains prompts and responses that relate to child abuse, suicide, self-harm and eating disorders, scams and fraud, illegal items, and physical harm.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. How Many Iterations to Jailbreak? Dynamic Budget Allocation for Multi-Turn LLM Evaluation

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    DAPRO provides the first dynamic, theoretically guaranteed way to allocate interaction budgets across test cases for bounding time-to-event in multi-turn LLM evaluations, achieving tighter coverage than static conform...

  2. When Are Reasoning-Based Guardrails Not Efficient? ResponseGuard: A Fast Vision-Language Guard for Real-Time Moderation

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A label-only 2B guard outperforms a 3B reasoning-based vision-language guard on response-harmfulness detection at about 150x lower latency, with the remaining image gap attributed to frozen encoders.

  3. Predict, Don't React: Value-Based Safety Forecasting for LLM Streaming

    cs.CL 2026-04 conditional novelty 6.0 of 10

    Forecasting expected future harmfulness from prefixes via Monte Carlo rollouts yields stronger streaming LLM moderation than boundary detection, without exact onset labels.

  4. YuFeng-XGuard: A Reasoning-Centric, Interpretable, and Flexible Guardrail Model for Large Language Models

    cs.CL 2026-01 conditional novelty 6.0 of 10

    A reasoning-centric guardrail model family that classifies risky content with a first-token decision, optional explanations, and runtime-adjustable safety policies, reporting state-of-the-art benchmark F1 scores.

  5. Semalith v1.4: A Calibrated 184M Safety Classifier Achieving State-of-the-Art Prompt-Injection Detection at 44x Fewer Parameters than Llama-Guard-3-8B

    cs.LG 2026-05 reject novelty 5.0 of 10

    A 184M-parameter DeBERTa-v3 fine-tuned model is claimed to beat Llama-Guard-3-8B on all tested prompt-injection benchmarks while adding BFSI regulatory labels, but a leaked training/eval overlap undermines the zero-FPR claim.

  6. A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models

    cs.CL 2025-09 conditional novelty 4.0 of 10

    A structured literature survey concluding that reasoning capabilities do not automatically make LLMs more trustworthy and can introduce new vulnerabilities in safety, robustness, and privacy.

  7. Scaling behavior of large language models in emotional safety classification across sizes and tasks

    cs.CL 2025-09 conditional novelty 4.0 of 10

    Fine-tuning a 1B LLaMA model on a synthetic emotional-safety benchmark matches or beats 70B few-shot performance and a BERT baseline on three high-data categories, using under 2GB VRAM.

  8. Fine-Tuning Lowers Safety and Disrupts Evaluation Consistency

    cs.CL 2025-06 conditional novelty 4.0 of 10

    Fine-tuning small LLMs on benign data raises harmfulness scores, but those scores vary widely across random seeds, temperatures, and repeated runs, making single-run safety comparisons unreliable.

Pith tools