Pith. sign in

REVIEW 10 cited by

Safety Assessment of Chinese Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2304.10436 v1 pith:PLFCJMHE submitted 2023-04-20 cs.CL

Hao Sun , Zhexin Zhang , Jiawen Deng , Jiale Cheng , Minlie Huang This is my paper
classification cs.CL
keywords safetyllmsbenchmarkmodelschineselanguagelargeassessment
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

With the rapid popularity of large language models such as ChatGPT and GPT-4, a growing amount of attention is paid to their safety concerns. These models may generate insulting and discriminatory content, reflect incorrect social values, and may be used for malicious purposes such as fraud and dissemination of misleading information. Evaluating and enhancing their safety is particularly essential for the wide application of large language models (LLMs). To further promote the safe deployment of LLMs, we develop a Chinese LLM safety assessment benchmark. Our benchmark explores the comprehensive safety performance of LLMs from two perspectives: 8 kinds of typical safety scenarios and 6 types of more challenging instruction attacks. Our benchmark is based on a straightforward process in which it provides the test prompts and evaluates the safety of the generated responses from the evaluated model. In evaluation, we utilize the LLM's strong evaluation ability and develop it as a safety evaluator by prompting. On top of this benchmark, we conduct safety assessments and analyze 15 LLMs including the OpenAI GPT series and other well-known Chinese LLMs, where we observe some interesting findings. For example, we find that instruction attacks are more likely to expose safety issues of all LLMs. Moreover, to promote the development and deployment of safe, responsible, and ethical AI, we publicly release SafetyPrompts including 100k augmented prompts and responses by LLMs.

Discussion (0). Sign in to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Paladin: Defending LLM-enabled Phishing Emails with a New Trigger-Tag Paradigm

    cs.CR 2025-09 conditional novelty 7.0 of 10

    A trigger-tag watermark embedded by fine-tuning lets modified LLMs mark their own phishing outputs for cheap detection.

  2. EPT Benchmark: Evaluation of Persian Trustworthiness in Large Language Models

    cs.CL 2025-09 conditional novelty 6.0 of 10

    A new Persian-Islamic trustworthiness benchmark ranks Claude highest and Qwen lowest across eight LLMs and finds safety is the weakest dimension.

  3. Libra: Large Chinese-based Safeguard for AI Content

    cs.AI 2025-07 conditional novelty 6.0 of 10

    Libra-Guard, a Chinese-language AI content safety model, reaches 86.8% accuracy on the new 5,720-sample Libra-Test benchmark, beating open-source baselines and approaching GPT-4o and Claude-3.5-Sonnet.

  4. Certifiable Safe RLHF: Semantic Grounding and Fixed Penalty Constraint Optimization for Safer LLM Alignment

    cs.LG 2025-10 reject novelty 5.0 of 10

    A fixed ReLU penalty and a semantically labeled cost model are proposed to make RLHF safer, but the 'certifiable' guarantee is not fully supported.

  5. SafeToolBench: Pioneering a Prospective Benchmark to Evaluating Tool Utilization Safety in LLMs

    cs.CR 2025-09 reject novelty 5.0 of 10

    A new benchmark of 1,200 risky tool-use requests plus a nine-dimension scoring prompt claims to improve LLM safety, but the evaluation only includes risky samples, so high scores may just mean the model refuses everything.

  6. GaussMaster: An LLM-based Database Copilot System

    cs.DB 2025-06 conditional novelty 5.0 of 10

    GaussMaster combines retrieval-augmented generation, safety controls, and diagnosis-tree-guided multi-agent orchestration to automate GaussDB Q&A and anomaly repair.

  7. Xinyu AI Search: Enhanced Relevance and Comprehensive Results with Rich Answer Presentations

    cs.IR 2025-05 conditional novelty 5.0 of 10

    Xinyu, an integrated generative AI search engine with query decomposition, multi-source retrieval, and rich answer presentation, outperforms eight existing technologies in human evaluations.

  8. A Comprehensive Evaluation framework of Alignment Techniques for LLMs

    cs.CL 2025-08 unverdicted novelty 4.0 of 10

    The paper proposes a multi-dimensional framework to evaluate and compare LLM alignment techniques.

  9. Fine-Tuning Lowers Safety and Disrupts Evaluation Consistency

    cs.CL 2025-06 conditional novelty 4.0 of 10

    Fine-tuning small LLMs on benign data raises harmfulness scores, but those scores vary widely across random seeds, temperatures, and repeated runs, making single-run safety comparisons unreliable.

  10. The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs

    cs.CL 2025-06 conditional novelty 3.0 of 10

    A structured survey of LLM safety evaluation that proposes a why/what/where/how taxonomy and catalogs metrics, datasets, benchmarks, evaluators, and frameworks.

Pith tools