REVIEW 9 cited by
AlignBench: Benchmarking Chinese Alignment of Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Alignment has become a critical step for instruction-tuned Large Language Models (LLMs) to become helpful assistants. However, the effective evaluation of alignment for emerging Chinese LLMs is still largely unexplored. To fill in this gap, we introduce AlignBench, a comprehensive multi-dimensional benchmark for evaluating LLMs' alignment in Chinese. We design a human-in-the-loop data curation pipeline, containing eight main categories, 683 real-scenario rooted queries and corresponding human verified references. To ensure the correctness of references, each knowledge-intensive query is accompanied with evidences collected from reliable web sources (including URLs and quotations) by our annotators. For automatic evaluation, our benchmark employs a rule-calibrated multi-dimensional LLM-as-Judge~\cite{zheng2023judging} approach with Chain-of-Thought to generate explanations and final ratings, ensuring high reliability and interpretability. All evaluation code, data, and LLM generations are available at \url{https://github.com/THUDM/AlignBench}. Since its release, AlignBench has been adopted by top (Chinese) LLMs for evaluating their alignment capabilities in Chinese, including ChatGLM, Qwen, DeepSeek, Yi, Baichuan, and Abab.
Forward citations
Cited by 9 Pith papers
-
Characterizing Bias: Benchmarking Large Language Models in Simplified versus Traditional Chinese
A new benchmark shows LLMs are more accurate in Simplified Chinese for regional terms but favor Taiwanese names in simulated hiring, revealing task-dependent bias between Chinese script variants.
-
Chain-of-Models: Cross-Model Auditing for Bias-Robust LLM Judges
Per-bias selection of a cross-family LLM auditor lifts biased-judgment accuracy from 0.805/0.824 baselines to 0.884.
-
CompassJudger-2: Towards Generalist Judge Model via Verifiable Rewards
A 7B judge model trained with verifiable reward signals and a margin contrastive loss matches the judgment accuracy of models tens of times larger, and a new benchmark JudgerBenchV2 standardizes judge evaluation.
-
Diagnosing Failures in Large Language Models' Answers: Integrating Error Attribution into Evaluation Framework
A new error-attribution dataset and fine-tuned judge model that outputs score, error category, and feedback for LLM responses.
-
Chengyu-Bench: Benchmarking Large Language Models for Chinese Idiom Understanding and Use
Chengyu-Bench is a 2,937-example human-verified benchmark showing LLMs are strong at idiom sentiment classification but weak at appropriateness and open cloze generation.
-
A Survey of Automatic Evaluation Methods on Text, Visual and Speech Generations
A unified taxonomy and comparative meta-evaluation of automatic evaluation methods across text, vision, and speech generation, concluding that LLM-based evaluators dominate current practice.
-
AgentScope 1.0: A Developer-Centric Framework for Building Agentic Applications
AgentScope 1.0 packages the components needed to build, evaluate, and deploy LLM agent applications into one developer framework.
-
Technical Report of TeleChat2, TeleChat2.5 and T1
The released T1-115B open-weight model outperforms OpenAI's o1-mini and GPT-4o on MATH500, AlignBench, and IFEval, despite using a standard dense transformer architecture.
-
Beyond the Surface: Measuring Self-Preference in LLM Judgments
The DBG metric measures LLM self-preference bias as the gap between a judge model's own win rate and the win rate assigned by an ensemble of gold judges.
Discussion (0). Sign in to comment.