REVIEW 7 cited by
LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scores
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Automatic evaluation of generated textual content presents an ongoing challenge within the field of NLP. Given the impressive capabilities of modern language models (LMs) across diverse NLP tasks, there is a growing trend to employ these models in creating innovative evaluation metrics for automated assessment of generation tasks. This paper investigates a pivotal question: Do language model-driven evaluation metrics inherently exhibit bias favoring texts generated by the same underlying language model? Specifically, we assess whether prominent LM-based evaluation metrics (e.g. BARTScore, T5Score, and GPTScore) demonstrate a favorable bias toward their respective underlying LMs in the context of summarization tasks. Our findings unveil a latent bias, particularly pronounced when such evaluation metrics are used in a reference-free manner without leveraging gold summaries. These results underscore that assessments provided by generative evaluation models can be influenced by factors beyond the inherent text quality, highlighting the necessity of developing more reliable evaluation protocols in the future.
Forward citations
Cited by 7 Pith papers
-
Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge
A regression controlling for human-rated quality detects positive self- and family-bias in several LLM judges, including GPT-4o and Claude 3.5 Sonnet.
-
LLM-as-a-Verifier: A General-Purpose Verification Framework
Expecting over scoring-token logits yields continuous, scalable verification that improves agent trajectory selection and dense RL rewards across coding, robotics, and medical benchmarks.
-
Breaking the Mirror: Activation-Based Mitigation of Self-Preference in LLM Evaluators
Steering vectors flip most unjustified self-preference decisions of an LLM judge but also disturb legitimate ones, showing the bias is not captured by a single linear direction.
-
TripTailor: A Real-World Benchmark for Personalized Travel Planning
A travel-planning benchmark is claimed in the abstract, but the full text is an unrelated supernova spectroscopy paper, leaving the central claim completely unsupported.
-
Adaptive-VP: A Framework for LLM-Based Virtual Patients that Adapts to Trainees' Dialogue to Facilitate Nurse Communication Training
A four-module LLM framework that scores trainee utterances and adjusts a virtual patient's hostility in real time, validated with Korean nursing corpora and expert nurse raters.
-
Beyond the Surface: Measuring Self-Preference in LLM Judgments
The DBG metric measures LLM self-preference bias as the gap between a judge model's own win rate and the win rate assigned by an ensemble of gold judges.
-
Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
A broad survey of LLM alignment that catalogs objectives, benchmarks, SFT/RLHF/DPO methods, and safety challenges, without contributing new experimental or theoretical results.
Discussion (0). Continue with ORCID to comment.