BeHonest: Benchmarking Honesty in Large Language Models
Reviewed by Pithpith:65FGBF77open to challenge →
read the original abstract
Previous works on Large Language Models (LLMs) have mainly focused on evaluating their helpfulness or harmlessness. However, honesty, another crucial alignment criterion, has received relatively less attention. Dishonest behaviors in LLMs, such as spreading misinformation and defrauding users, present severe risks that intensify as these models approach superintelligent levels. Enhancing honesty in LLMs addresses critical limitations and helps uncover latent capabilities that are not readily expressed. This underscores the urgent need for reliable methods and benchmarks to effectively ensure and evaluate the honesty of LLMs. In this paper, we introduce BeHonest, a pioneering benchmark specifically designed to assess honesty in LLMs comprehensively. BeHonest evaluates three essential aspects of honesty: awareness of knowledge boundaries, avoidance of deceit, and consistency in responses. Building on this foundation, we designed 10 scenarios to evaluate and analyze 9 popular LLMs on the market, including both closed-source and open-source models from different model families with varied model sizes. Our findings indicate that there is still significant room for improvement in the honesty of LLMs. We encourage the AI community to prioritize honesty alignment in these models, which can harness their full potential to benefit society while preventing them from causing harm through deception or inconsistency. Our benchmark and code can be found at: \url{https://github.com/GAIR-NLP/BeHonest}.
This paper has not been read by Pith yet.
Forward citations
Cited by 7 Pith papers
-
Decoding Hidden Deception in Reasoning LLMs: Activation Explainers for Deception Auditing
STATEWITNESS is a decoder-based activation explainer that audits deception in LLMs by interpreting hidden states, reaching 0.916 mean AUROC on seven datasets with inspectable evidence.
-
RogueAI: A Reverse Turing Test for Detecting Licensed AI Deception in Dialogue
RogueAI operationalizes a reverse Turing test as a one-on-two interrogation game to detect licensed deception in LLMs, with pilot data from 467 sessions showing a simple linguistic heuristic at 75.6% accuracy versus 5...
-
SPADE-Bench: Evaluating Spontaneous Strategic Deception in Agents via Plan-Action Divergence
SPADE-Bench is a new evaluation framework that tests spontaneous plan-action divergence in LLM agents during tool-use tasks under controlled pressure.
-
SPADE-Bench: Evaluating Spontaneous Strategic Deception in Agents via Plan-Action Divergence
SPADE-Bench is a benchmark that measures spontaneous plan-action divergence in tool-using LLM agents under pressure to distinguish strategic deception from hallucination.
-
DECOR: Auditing LLM Deception via Information Manipulation Theory
DECOR introduces a theory-grounded multi-agent system that decomposes contexts into atomic units, scores four manipulation dimensions per unit, and aggregates profiles into a global deception index, reporting SOTA res...
-
Navigating the Sea of LLM Evaluation: Investigating Bias in Toxicity Benchmarks
Toxicity benchmarks for LLMs produce inconsistent results when task type, input domain, or model changes, revealing intrinsic evaluation biases.
-
Unlearners Can Lie: Evaluating and Improving Honesty in LLM Unlearning
Existing LLM unlearning methods fail honesty standards by hallucinating on forgotten knowledge; ReVa improves rejection rates nearly twofold while enhancing retained honesty.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.