REVIEW 3 major objections 5 minor 5 cited by
Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge
T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read Some LLM judges systematically rate their own outputs higher than equally good responses from other models, even after accounting for human-rated quality.
desk verdict A careful regression-based framework for self-bias in LLM judges, backed by a reusable dataset and strong robustness checks; the significance tests need cluster-robust SEs before taking the headline numbers at face value. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The regression in Equation (1), which models an LLM judge's rating as a linear function of the human reference score plus a judge-specific self-bias indicator and a family-bias indicator. The self-bias coefficient $\gamma_j$ is the additive boost a judge gives to its own completions after adjusting for quality, and $\lambda_{F(j)}$ is the same for completions from models of the judge's family. The human reference score $S_{idm}$ is the component that makes the identification possible: it absorbs genuine quality differences, so the indicators capture only systematic favoritism.
What would settle it
Take a benchmark with objectively correct answers (e.g., math or code generation), have GPT-4o and Claude 3.5 Sonnet score their own and other models' answers, and use the objective correctness score as the reference $S_{idm}$ in Equation (1). If the estimated $\gamma_j$ drops to zero or loses significance compared with the human-reference version, the apparent self-bias is not identity-based favoritism but a shared human-judge preference.
Extended reading notes
Core claim
The central discovery, on the paper's own terms, is that self-bias and family-bias are identifiable parameters in a regression of judge ratings on an independent reference score. Fitting $\tilde{S}_{idmj} = \alpha + \delta_j + \beta_j S_{idm} + \gamma_j \mathbf{1}_{j=m} + \lambda_{F(j)}\mathbf{1}_{F(j)=F(m)} + \eta_d + \epsilon_{idmj}$ to 5,364 completions of 596 prompts scored by nine LLM judges and expert humans, the paper finds $\gamma_j$ significantly positive for GPT-4o and Claude 3.5 Sonnet, and $\lambda$ significantly positive for the Claude and GPT families. The identification comes from the offset between how a judge rates its own completions versus other models' completions at the
Load-bearing premise
The method assumes the average human score is an unbiased, interval-scale measure of a completion's true quality, and that any systematic judge-specific preference for length, style, or other features is shared by the human raters; if that fails, the estimated self-bias coefficients absorb those differences and are misread as favoritism.
Editorial extensions
If this is right
- LLM-judge rankings can be misleading when comparing models of similar measured quality; a score gap of about 0.02, comparable to the estimated self-bias, can reverse a ranking.
- If human reference scores are available, practitioners can estimate and subtract $\gamma_j$ and $\lambda_{F(j)}$ from judge ratings to obtain debiased evaluation scores at deployment.
- The framework generalizes to other biases that admit a control group of completions where the bias does not apply, with any reference score one wants to imitate.
- Family-bias implies that a panel of judges from a single family, such as multiple GPT models, can over-reward that family's outputs even when no single judge favors its own generation.
Reading between the lines
- If human raters share the LLM judges' stylistic preferences, the estimated self-bias could partly reflect genuine taste overlap rather than identity-driven favoritism; the paper's robustness check using LLM reference scores is a step toward testing this, but an objective ground truth such as verifiable answers would settle it.
- The negative self-bias found for Llama 3 8B suggests the framework detects both directions of favoritism; a natural next experiment is to swap completions across models to test whether the effect is actually about recognizing one's own outputs or about recognizing a family's typical style or length.
- Because the paper pools dimensions with different Likert scales, the linear model imposes interval-scale comparability; the ordered-logit robustness check partially relaxes this, but a full ordinal mixed model that treats each dimension's scale separately would be a more faithful extension.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes a statistical regression framework for measuring self-bias and family-bias in LLM-as-a-judge evaluations. The central model, Equation (1), regresses an LLM judge's score on a human reference score with judge-specific alignment/intercept terms, a self-bias dummy, a family-bias dummy, and dimension fixed effects. The authors release a new dataset of 596 prompts, nine model completions, six evaluation dimensions, expert human annotations, and LLM-as-a-judge ratings. They report statistically significant positive self-bias for GPT-4o and Claude 3.5 Sonnet, positive family-bias for the Claude and GPT families, and negative self-bias for Llama 3 8B. Several robustness checks are presented, including length control, ordinal logit and GAM specifications, alternative reference scores, and removal of weaker models.
Significance. If the estimates are valid, this is a useful contribution to the growing literature on LLM-as-a-judge biases. The method improves on simple comparisons of self-scores by explicitly controlling for human-rated quality and judge-specific alignment, and the released dataset is a valuable resource for future evaluation research. The paper is also transparent about many of its identifying assumptions. However, the empirical headline rests on pooled OLS with White standard errors under an independence assumption that is unlikely to hold, and on pooling Likert scales of different widths as continuous variables. These issues are load-bearing for the significance claims, so the current evidence is suggestive rather than definitive.
major comments (3)
- [Section 5.2, Appendix A] The paper reports 90% Wald confidence intervals based on White (heteroskedasticity-robust) standard errors. The pooled dataset, however, is highly clustered: each prompt contributes 9 completions x 6 dimensions x 9 judges = 486 observations, and the six ratings of the same completion are almost surely correlated. White standard errors are not valid under within-cluster correlation and will be anti-conservative if correlations are positive. The central claims that GPT-4o and Claude 3.5 Sonnet have significant self-bias, and that GPT/Claude families have family-bias, rely on these intervals in Figure 3 excluding zero. Please re-estimate the variances with cluster-robust estimators (at least clustered by prompt; ideally two-way by prompt and completion) and report whether the 90% intervals still exclude zero. The point estimates are unchanged, but the significance statements may be.
- [Equation (1), Section 4.2] The main model pools ordinal Likert ratings on 3-, 5-, and 7-point scales and treats them as continuous outcomes, with only dimension-specific intercepts eta_d. A constant intercept shift does not account for differences in scale widths or in the mapping from latent quality to observed category. If the judge's own completions are not uniformly distributed across dimensions, or if the judge-reference relationship is nonlinear in different ways across dimensions, gamma_j can absorb scale artifacts rather than self-bias. The per-dimension ordinal logit robustness check is not a full substitute because it estimates separate dimension-level coefficients and does not produce the pooled gamma_j used in the headline. Please either normalize scores within dimension, include dimension-specific scale parameters, or present an ordinal model that yields a single pooled gamma_j for the main claim.
- [Ethics statement, Section 3] The paper correctly acknowledges that identification of gamma_j as self-bias assumes human ratings are unbiased reference scores and that all other systematic judge/reference differences are shared. This is a strong assumption, and unmodeled judge preferences (e.g., for length or style) that are not shared by humans can be absorbed into gamma_j and lambda_F. Since the abstract and conclusions present the positive coefficients as evidence of systematic bias, I ask for either a formal sensitivity analysis or consistent use of the hedged phrasing 'relative to human reference ratings' in all summary statements, so that the gap between the statistical estimand and the causal interpretation is explicit in the claims that reach the reader.
minor comments (5)
- [Throughout] Typos: 'asself-bias' in the abstract, 'extend' for 'extent' in Section 5.1, 'cofficient' in Appendix A, and 'Kripendorff's' in Table 4. Please proofread.
- [Section 6.2 / Figure 10] The text says the robustness check removes Mistral 7B and Llama 3 8B, while the Figure 10 caption lists 'Claude v2, Llama 3 8B, and Mistral 7B' and the plotted axes still include Claude v2. Please reconcile the text, caption, and figure.
- [Figure 8 caption] The caption says 'left' and 'right' panels, but the figure as displayed appears to be a single panel. Update the caption to match the layout.
- [References] Several references are incomplete, e.g., 'Zhu et al., Li et al., a' and 'Li et al., b'. Please disambiguate and complete all entries.
- [Abstract, Section 7] The word 'systematically' overstates the dimension-level results; Section 6.1 shows that self-bias is concentrated in particular dimensions for several models. Qualify the summary claims accordingly.
Circularity Check
No significant circularity: self-bias is estimated from independent human reference scores and the main claim is not equivalent to its inputs.
full rationale
Equation (1) estimates gamma_j and lambda_F from the difference between judge ratings and an independent human reference score S_idm; the self-bias coefficient is not defined by the judge's own outputs but by a regression residual after controlling for S_idm. The main result is therefore a fit to external data, not a prediction forced by construction. The only potentially circular-looking robustness check (using LLM ratings as reference scores) explicitly acknowledges that naive replacement would introduce circularity and instead removes the judged family before constructing the alternative reference; it is a sensitivity analysis, not the main identification. The paper's self-citations (Fogliato et al. 2021, 2024) support standard-error computation and a future efficiency tool, and are not load-bearing for the central claim. The stated limitations, such as 'assumes human ratings as unbiased reference scores', are assumptions about external validity rather than circular definitions. Statistical concerns about White versus cluster-robust standard errors are correctness risks, not circularity.
Assumptions & free parameters
free parameters (5)
- Judge-specific self-bias coefficient gamma_j =
approx. 0.01 to 0.02 for GPT-4o and Claude 3.5 Sonnet; negative for Llama 3 8B
- Family-bias coefficient lambda_F(j) =
approx. 0.005 to 0.01 for Claude and GPT families
- Judge-specific alignment slope beta_j =
not reported numerically
- Judge fixed effects delta_j and global intercept alpha =
not reported
- Dimension fixed effects eta_d =
not reported
assumptions (4)
- domain assumption Human average ratings S_idm are an unbiased, interval-scale measure of completion quality.
- domain assumption Apart from self- and family-bias, systematic rating differences between human and LLM judges are absent.
- domain assumption Averaged Likert scores from 3-, 5-, and 7-point scales can be pooled in a linear model after dimension intercept shifts.
- standard math OLS asymptotic normality and White standard errors are valid for inference.
Cite this review
Pith. "Pith review of Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge." pith.science (2026). https://pith.science/paper/YLAUA24J
@misc{pith2026250806709,
author = {Pith},
title = {Pith review of: Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge},
year = {2026},
howpublished = {\url{https://pith.science/paper/YLAUA24J}},
note = {Machine review of arXiv:2508.06709}
}
read the original abstract
Large language models (LLMs) can serve as judges that offer rapid and reliable assessments of other LLM outputs. However, models may systematically assign overly favorable ratings to their own outputs, a phenomenon known as self-bias, which can distort evaluations of true model performance. Previous studies often conflate genuine differences in model quality with bias or incorrectly assume that evaluations from LLMs and humans follow the same rating distributions. In this work, we present a statistical framework that explicitly formalizes assumptions under which self-bias can be identified and estimated. Our method models the difference in the scoring distribution that LLM-as-a-judge assigns to its own completions compared to other models, while accounting for the underlying quality of the completions provided by an independent, third-party judge (e.g., humans). Our method reliably isolates and quantifies self-bias, even when models vary in ability, ensuring that genuine performance differences are not mistaken for self-bias. We conduct an empirical analysis of self-bias on a large dataset (>5000 prompt-completion pairs) consisting of expert human annotations and judgments from nine different LLM judges. We find that some models, such as GPT-4o and Claude 3.5 Sonnet, systematically assign higher scores to their own outputs. These models also display family-bias; systematically assigning higher ratings to outputs produced by other models of the same family. Our findings highlight potential pitfalls of using LLM judges and offer practical guidance to mitigate biases when interpreting automated evaluations.
Figures
Figures from the paper (8 more)
Forward citations
Cited by 5 Pith papers
-
Hy-MultiTurn: A Six-Dimensional Benchmark for Deep Multi-Turn Dialogue Understanding
A 209-task Chinese benchmark across six dialogue-failure modes shows that even the top frontier model fully satisfies only 41.1% of long multi-turn requests.
-
Self-Preference Bias in Rubric-Based Evaluation of Large Language Models
Self-preference bias persists in rubric-based LLM evaluation even with fully objective, programmatically verifiable rubrics, and can shift subjective medical-chat scores by up to ~10 points.
-
Eval-Pair Matrix: Answer-Paired Meta-Evaluation of LLM Judges for Grounded RAG
Answer-paired analysis of a 3×3 GPT/Grok/Gemini judge matrix finds near-zero same-model recall bias for induced RAG grounding errors; remaining flag gaps reflect label-task mismatch.
-
Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluation
Validity in agentic AI evaluation degrades multiplicatively across task generation, simulation, and judging, so most reported benchmark scores retain far less information than they appear to.
-
Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations
Using an output-matched proxy baseline across 16 models on 9 datasets, 89.6% of measured self-preference bias disappears and roughly half of prior findings lose significance.
Reference graph
Works this paper leans on
-
[1]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,
-
[3]
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862 , 2022a. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy J...
-
[5]
High There are only 7 restaurants with a yelp rating of 5 in North Platsville. Here they are: [list of 7 restaurants] Low There are only 7 restaurants with a yelp rating of 5 in North Platsville. Conciseness Who was president of the US in 1954? High Dwight Eisenhower was presi- dent of the US in
work page 1954
-
[7]
maars: Tidy Inference under the 'Models as Approximations' Framework in R
Riccardo Fogliato, Shamindra Shrotriya, and Arun Kumar Kuchibhotla. maars: Tidy inference under the’models as approximations’ framework in r.arXiv preprint arXiv:2106.11188 ,
-
[9]
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 ,
-
[11]
URLhttps://arxiv.org/ abs/2310.06825. Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source language model specialized in evaluating other language models. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Pro...
arXiv 2024
-
[12]
Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. Bench- marking cognitive biases in large language models as evaluators.arXiv preprint arXiv:2309.17012 ,
-
[13]
Model-free study of ordinary least squares linear regression
Arun K Kuchibhotla, Lawrence D Brown, and Andreas Buja. Model-free study of ordinary least squares linear regression. arXiv preprint arXiv:1809.10538 ,
Show all 31 references
-
[17]
Llm evaluators recognize and favor their own generations
Arjun Panickssery, Samuel R Bowman, and Shi Feng. Llm evaluators recognize and favor their own generations. arXiv preprint arXiv:2404.13076 ,
-
[19]
Red teaming language models with language models
Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pag...
2022
-
[20]
Verbosity bias in preference labeling by large language models.arXiv preprint arXiv:2310.10076 ,
Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto. Verbosity bias in preference labeling by large language models.arXiv preprint arXiv:2310.10076 ,
-
[22]
Replacing judges with juries: Evaluating llm generations with a panel of diverse models.arXiv preprint arXiv:2404.18796 ,
Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhang- orodsky, Minjie Xu, Naomi White, and Patrick Lewis. Replacing judges with juries: Evaluating llm generations with a panel of diverse models.arXiv preprint arXiv:2404.18796 ,
-
[23]
Self-preference bias in llm-as-a-judge.arXiv preprint arXiv:2410.21819,
Koki Wataoka, Tsubasa Takahashi, and Ryokan Ri. Self-preference bias in llm-as-a-judge.arXiv preprint arXiv:2410.21819,
-
[24]
Justice or prejudice? quantifying biases in llm-as-a-judge
Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, et al. Justice or prejudice? quantifying biases in llm-as-a-judge. arXiv preprint arXiv:2410.02736 ,
-
[25]
Flask: Fine-grained language model evaluation based on alignment skill sets
Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. Flask: Fine-grained language model evaluation based on alignment skill sets. InICLR 2024 Workshop on Large Language Model (LLM) Agents . Yian Zhang, Yif...
2024
-
[26]
Xinshu Zhao, Jun S Liu, and Ke Deng
URLhttps://crfm.stanford.edu/2024/02/18/helm-instruct.html. Xinshu Zhao, Jun S Liu, and Ke Deng. Assumptions behind intercoder reliability indices.Annals of the International Communication Association , 36(1):419–480,
2024
-
[27]
Judgelm: Fine-tuned large language models are scalable judges
Lianghui Zhu, Xinggang Wang, and Xinlong Wang. Judgelm: Fine-tuned large language models are scalable judges. In The Thirteenth International Conference on Learning Representations . 16 A Additional Details on the Methods We provide some details on the estimation of the regres...
2010
-
[28]
The process is analogous for the cofficient corresponding to family-bias
In other words, we assess whether the coefficientγj is equal to0. The process is analogous for the cofficient corresponding to family-bias. A two-sided Wald test of level α will rejectH0 if |dVar(ˆγj)−1/2ˆγj| > z1−α/2 where z1−α/2 is the1 − α/2 percentile of a standard Normal....
2013
-
[31]
I can’t answer that
Claude v2 Claude 3 Sonnet Claude 3.5 Sonnet GPT−3.5 Turbo GPT−4o Llama 3 8B Llama 3 70B Mistral 7B Mistral Large −1 0 1 2 3 4 Estimate of self−bias Dimension Conciseness Completeness Faithfulness Helpfulness Logical robustness Logical correctness Figure 8: Robustness check: Es...
2024
-
[1954]
Is a kilo of feathers heavier than a pound of steel? High One pound is equal to about 0.45 kilograms
Low Adlai Stevenson lost the presi- dential race in 1954 Logical Robust- ness Please give me a brief, formal answer. Is a kilo of feathers heavier than a pound of steel? High One pound is equal to about 0.45 kilograms. Therefore, one kilogram of feathers is heavier than one po...
1954
-
[1961]
Large language models are inconsistent and biased evaluators.arXiv preprint arXiv:2405.01724 ,
Rickard Stureborg, Dimitris Alikaniotis, and Yoshi Suhara. Large language models are inconsistent and biased evaluators.arXiv preprint arXiv:2405.01724 ,
-
[2002]
Offsetbias: Leveraging debiased data for tuning evaluators
Junsoo Park, Seungyeon Jwa, Ren Meiying, Daeyoung Kim, and Sanghyuk Choi. Offsetbias: Leveraging debiased data for tuning evaluators. InFindings of the Association for Computational Linguistics: EMNLP 2024 , pages 1043–1067,
2024
-
[2004]
G-eval: Nlg evaluation using gpt-4 with better human alignment
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 2511–2522, 2023a. Yiqi Liu, Nafise Sad...
2023 arXiv
-
[2006]
Human-like summarization evaluation with chatgpt.arXiv preprint arXiv:2304.02554 ,
Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan. Human-like summarization evaluation with chatgpt.arXiv preprint arXiv:2304.02554 ,
-
[2015]
Humans or llms as the judge? a study on judgement bias
Guiming Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. Humans or llms as the judge? a study on judgement bias. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8301–8327,
2024
-
[2016]
Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization
Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 1797–1807,
2018
-
[2017]
Gpt-4o system card.arXiv preprint arXiv:2410.21276,
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276,
-
[2018]
From generation to judgment: Opportunities and challenges of llm-as-a-judge.arXiv preprint arXiv:2411.16594 ,
14 Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. From generation to judgment: Opportunities and challenges of llm-as-a-judge.arXiv preprint arXiv:2411.16594 ,
-
[2022]
Bert: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human langua...
2019
-
[2023]
On the limitations of reference-free evaluations of generated text
Daniel Deutsch, Rotem Dror, and Dan Roth. On the limitations of reference-free evaluations of generated text. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10960–10977,
2022
-
[2024]
Reference-guided verdict: Llms-as-judges in automatic evaluation of free-form text
12 Sher Badshah and Hassan Sajjad. Reference-guided verdict: Llms-as-judges in automatic evaluation of free-form text. arXiv preprint arXiv:2408.09235 ,
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.