LLM judges show low intra-rater reliability, producing inconsistent and sometimes arbitrary ratings across repeated evaluations on NLG tasks.
Output: A single number0for consistent sum- mary and1for inconsistent summary
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.CL 1years
2025 1verdicts
UNVERDICTED 1representative citing papers
citing papers explorer
-
Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks
LLM judges show low intra-rater reliability, producing inconsistent and sometimes arbitrary ratings across repeated evaluations on NLG tasks.