The reviewed record of science sign in
Pith

arxiv: 2411.07140 · v2 · pith:GMXIGQ72 · submitted 2024-11-11 · cs.CL

Chinese SimpleQA: A Chinese Factuality Evaluation for Large Language Models

Yancheng He , Shilong Li , Jiaheng Liu , Yingshui Tan , Weixun Wang , Hui Huang , Xingyuan Bu , Hangyu Guo
show 10 more authors
Chengwei Hu Boren Zheng Zhuoran Lin Xuepeng Liu Dekai Sun Shirong Lin Zhicheng Zheng Xiaoyong Zhu Wenbo Su Bo Zheng
This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:GMXIGQ72record.jsonopen to challenge →

classification cs.CL
keywords chinesesimpleqamodelsfactualitylanguageanswerscomprehensiveevaluation
0
0 comments X
read the original abstract

New LLM evaluation benchmarks are important to align with the rapid development of Large Language Models (LLMs). In this work, we present Chinese SimpleQA, the first comprehensive Chinese benchmark to evaluate the factuality ability of language models to answer short questions, and Chinese SimpleQA mainly has five properties (i.e., Chinese, Diverse, High-quality, Static, Easy-to-evaluate). Specifically, first, we focus on the Chinese language over 6 major topics with 99 diverse subtopics. Second, we conduct a comprehensive quality control process to achieve high-quality questions and answers, where the reference answers are static and cannot be changed over time. Third, following SimpleQA, the questions and answers are very short, and the grading process is easy-to-evaluate based on OpenAI API. Based on Chinese SimpleQA, we perform a comprehensive evaluation on the factuality abilities of existing LLMs. Finally, we hope that Chinese SimpleQA could guide the developers to better understand the Chinese factuality abilities of their models and facilitate the growth of foundation models.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. MemSearcher: Training LLMs to Reason, Search and Manage Memory via End-to-End Reinforcement Learning

    cs.CL 2025-11 unverdicted novelty 7.0

    MemSearcher trains LLMs to manage compact memory in multi-turn searches via multi-context GRPO for end-to-end RL, outperforming ReAct-style baselines with stable token counts.

  2. Patterns behind Chaos: Forecasting Data Movement for Efficient Large-Scale MoE LLM Inference

    cs.DC 2025-10 conditional novelty 6.0

    Comprehensive profiling of expert selection in frontier MoE models reveals temporal and spatial patterns that enable 6.6x speedup on wafer-scale GPUs and 1.25x on existing systems via targeted optimizations.

  3. LIMO: Less is More for Reasoning

    cs.CL 2025-02 unverdicted novelty 6.0

    LIMO achieves 63.3% on AIME24 and 95.6% on MATH500 via supervised fine-tuning on roughly 1% of the data used by prior models, supporting the claim that minimal strategic examples suffice when pre-training has already ...

  4. Continual LLM Upcycling: A Predictor-Gated Bank-Wise Sparsity Training Recipe for Dense-to-Sparse LLMs

    cs.CL 2026-06 unverdicted novelty 5.0

    Continual training recipe upcycles dense Qwen2.5-8B LLM to 4x channel-sparse model via predictor-gated bank-wise sparsity in SwiGLU FFN with a single-layer repair for long-context failure on RULER-CWE.

  5. Kimi K2: Open Agentic Intelligence

    cs.LG 2025-07 unverdicted novelty 5.0

    Kimi K2 is a 1-trillion-parameter MoE model that leads open-source non-thinking models on agentic benchmarks including 65.8 on SWE-Bench Verified and 66.1 on Tau2-Bench.

  6. KnowRL: Exploring Knowledgeable Reinforcement Learning for Factuality

    cs.AI 2025-06 unverdicted novelty 5.0

    KnowRL integrates a knowledge-verification factuality reward into RL training to enforce fact-based reasoning steps and lower hallucination rates in LLMs.