Pith. sign in

REVIEW 2 major objections 27 references

An LLM-as-judge score can move when only the evaluator changes; practical judge upgrades are not interchangeable, and reliability needs bias, dependence, and audit reports beyond accuracy.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 05:46 UTC pith:SE6CXDTV

load-bearing objection Solid empirical audit: under shared prompts and parse-shared tests, Qwen3 1.7B→4B is the only robust adjacent gain, MiniMax adjacent releases are flat, and residual bias plus high ρ make the reporting checklist the real product. the 2 major comments →

arxiv 2607.08535 v1 pith:SE6CXDTV submitted 2026-07-09 cs.CL cs.AI

When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability

Zongyou Yang , Yinghan Hou , Xiaokun Yang This is my paper
classification cs.CL cs.AI
keywords LLM-as-judgeautomatic evaluationmodel scalingevaluation reliabilitybiasjury aggregationposition biasprotocol auditability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper treats LLM-as-judge evaluation as a measuring instrument that can itself introduce ambiguity: when the score changes after you replace the judge, you cannot tell from accuracy alone whether capability, bias, the benchmark slice, or the parsing pipeline caused the shift. Across four judgment datasets it compares two everyday upgrade paths—scaling Qwen3 dense models from 1.7B to 32B and stepping through MiniMax M2–M2.7 released APIs—and finds that upgrades do not behave alike. Only the jump from Qwen3 1.7B to 4B survives strict paired testing as a robust adjacent gain; adjacent MiniMax releases do not. Stronger judges cut position and verbosity bias but leave it nonzero; majority-vote juries barely help when errors are highly correlated; and structured debate can swing decisions a lot, yet those swings cannot be credited to deliberation without parser and fallback logs. The practical upshot is a reporting standard: slice definition, bias probes, error-dependence estimates, and protocol audit trails so a judge score can be read as evidence about the candidates rather than about the instrument.

Core claim

Judge upgrades available in practice are not interchangeable under a shared measurement protocol. After parse-shared McNemar tests and Holm correction over eighteen adjacent comparisons, only the Qwen3 1.7B→4B steps on LLMBar and Arena remain significant; no MiniMax adjacent release step is significant. Higher accuracy reduces but does not remove position and verbosity bias, and repeated-sample juries add little once error correlation is accounted for. Protocol interventions such as debate can move outcomes substantially, but without parser and fallback logs those moves cannot be attributed to deliberation.

What carries the argument

Evaluator-replacement ambiguity framed as a multi-component measurement pipeline: judgment validity (accuracy, κ), bias robustness (position flip, verbosity padding), aggregation independence (error correlation ρ and ρ-corrected beta-binomial jury predictions), and protocol auditability (parse status, fallbacks, intermediate verdicts), tested on a Qwen3 parameter axis versus a MiniMax release-generation axis.

Load-bearing premise

That four English judgment slices, one main fixed prompt, and the chosen proxies for accuracy, bias, correlation, and parse logs are enough to decide whether practical upgrade paths improve reliability in general.

What would settle it

Under the same datasets, prompts, parse-shared paired tests, and Holm correction, find at least one MiniMax adjacent release step that is significant while the Qwen3 1.7B→4B gain disappears, or measure low error correlation with large majority-vote jury gains that match the independence prediction.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 0 minor

Summary. The paper treats LLM-as-judge evaluation as a measurement-validity problem under evaluator-replacement ambiguity: scores can change when the judge changes even if candidate responses are fixed. Across four judgment datasets (LLMBar, PandaLM, Arena sample, Judge's Verdict), it compares two practical upgrade paths—Qwen3 dense scaling (1.7B–32B) and MiniMax M2–M2.7 released APIs—using parse-shared McNemar tests with Holm correction, bias probes (position flip, verbosity, granularity), homogeneous/heterogeneous juries with estimated error correlation ρ and a beta-binomial correction, and a structured-debate protocol. Main findings: upgrades are not interchangeable (only Qwen3 1.7B→4B remains robustly significant after correction; MiniMax adjacent steps do not); higher accuracy reduces but does not eliminate bias; jury gains are small under high ρ; debate can shift decisions substantially but lacks parser/fallback logs needed for causal attribution. The authors recommend reporting slices, bias probes, ρ, and protocol audit trails.

Significance. If the reported pattern holds, the paper usefully reframes LLM-as-judge work away from one-dimensional model ordering toward auditable measurement design. The empirical core is carefully executed: parse-shared paired tests, Holm correction over 18 adjacent comparisons, Wilson intervals, residual bias rates, and a ρ-corrected beta-binomial model that tracks observed jury accuracy far better than independence (median absolute error drops to ~0.008/0.004). The reporting checklist (Table 4) is operationally concrete and would improve reproducibility of preference evaluation. Strengths include explicit non-causal framing of the MiniMax release path, separation of capability from bias/dependence/protocol artifacts, and transparent threats to validity. The contribution is primarily methodological and empirical rather than theoretical, but it addresses a real practice gap in how judge upgrades and multi-agent protocols are currently reported.

major comments (2)
  1. §4.4 / Finding 4 and Table 1 (protocol auditability): The debate experiment produces the largest protocol-level shifts (+0.289 to +0.317 for cross-capability pairs) yet is presented without raw responses, parse-success flags, or fallback rates. The manuscript itself states that round-1 parse failures fall back to A and later failures retain the previous verdict, so final-vs-round-1 accuracy cannot be attributed to deliberation. This is load-bearing for RQ3 and for the claim that protocol upgrades require audit trails. Either re-run with full logs and report parse/fallback rates, or demote the debate results more clearly to a negative case study (protocol pattern only) and remove any residual implication of a deliberation effect from the abstract and Table 2.
  2. §3.2–3.3, Table 3, and Threats §6 (external validity): The non-interchangeability claim rests on four English slices, a fixed main prompt, one primary decoding regime per experiment, and two model families (plus two reference judges). Arena is a seed-42 2k subsample; Judge's Verdict is only 200 examples and shows non-monotone Qwen3 behavior. The authors already flag limited transfer, but the abstract and RQ1 answer still read as a general statement about 'judge upgrades' and 'released-model upgrade paths.' The central claim should be scoped more tightly in the abstract and conclusion to the tested panel and protocol, or strengthened with at least one additional domain/language slice or a controlled prompt-factorial check beyond the limited robustness note.

Circularity Check

0 steps flagged

No significant circularity: empirical measurements on external judgment datasets, not predictions forced by fitted inputs or self-citation.

full rationale

The paper’s load-bearing claims are empirical comparisons of judge accuracy, bias flip rates, jury error correlation, and debate shifts on four external human-preference / judgment slices (LLMBar, PandaLM, Arena sample, Judge’s Verdict), using parse-shared McNemar tests with Holm correction and fixed main prompts. Nothing is derived by defining a quantity in terms of the target it then ‘predicts.’ The β-binomial jury check estimates ρ from the vote matrix and shows that the ρ-corrected prediction tracks observed majority-vote accuracy better than independence; that is ordinary model calibration against the same votes, not a headline prediction of a quantity that was fitted into the model. MiniMax is explicitly treated as an observed release path rather than a controlled ablation, and debate is framed as an auditability case study precisely because parser/fallback logs are missing—so the authors do not smuggle causal deliberation claims. References are to external benchmarks and prior LLM-judge literature; there is no load-bearing self-citation of a uniqueness theorem or ansatz that forces the non-interchangeability result. The study is self-contained against external benchmarks under the stated panel and protocol.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 2 invented entities

The paper is an empirical audit, not a first-principles derivation. Load-bearing background is standard paired statistics plus domain practice that human labels on preference/judgment sets are the external validity target. Free choices are experimental design knobs (temperatures, jury sizes, one Arena seed, fixed main prompt, padding string, Holm family size). Invented content is mostly conceptual framing and an operational four-part reliability construct, not new physical entities.

free parameters (6)
  • main_decoding_temperature
    Near-greedy T=0.1 for single-judge reliability estimates; chosen by design to reduce sampling noise, not fitted, but affects measured accuracy and parse behavior.
  • jury_sampling_temperature
    T=0.7 for homogeneous repeated-sample juries; chosen so dependence is not understated by low-T repeats.
  • jury_sizes_K
    Reported K=1,3,5 for homogeneous juries; discrete design choice that bounds observed aggregation gains.
  • Arena_subsample_seed_and_N
    2,000-example seed-42 sample (1,997 valid) from Chatbot Arena; sampling choice affects slice-level conclusions.
  • verbosity_padding_string
    Fixed generic innocuous padding used for verbosity bias probe; probe magnitude depends on this hand-chosen string.
  • Holm_family_size
    Family of 18 adjacent tests for multiple-comparison control; defines which adjacent gains survive as 'robust'.
axioms (5)
  • domain assumption Human preference/judgment labels on the four datasets are a valid external target for judge accuracy, Cohen's κ, and Spearman correlation.
    Invoked throughout §3–4 as the ground for 'judgment validity'; standard in LLM-as-judge work but not re-validated here beyond human-ceiling calibration.
  • standard math Exact two-sided McNemar tests on parse-shared examples, with Holm correction, are appropriate for declaring adjacent upgrade significance.
    §3.3 and Fig. 3; standard paired binary test, but assumes parser-shared subset is not itself a biased selection.
  • domain assumption Independence (or Poisson-binomial) jury predictions are the right null against which to measure dependence-limited gains; β-binomial with estimated ρ is a suitable correction.
    §4.3 and classical Condorcet/correlated-voter citations; modeling choice that shapes Finding 3.
  • ad hoc to paper MiniMax M2–M2.7 API sequence and Qwen3 dense sizes are meaningful practical 'upgrade paths' for studying evaluator-replacement ambiguity without needing causal training ablations.
    Stated in Introduction and §3.2; authors disclaim MiniMax causal claims, yet interchangeability conclusions rest on treating both axes as comparable interventions.
  • domain assumption A fixed main prompt with limited prompt-sensitivity checks is enough to compare judges on reliability rather than prompt engineering.
    Related Work and robustness checks; prompt is held fixed by design.
invented entities (2)
  • evaluator-replacement ambiguity no independent evidence
    purpose: Name the non-identifiability of score changes when only the judge is swapped.
    Framing device in Introduction; organizes the study but is not an independently measured physical object.
  • four-component reliability construct (validity, bias robustness, aggregation independence, protocol auditability) no independent evidence
    purpose: Operationalize multi-dimensional judge reliability via Table 1 measures and Experiments 1–4.
    Paper-defined measurement framework; proxies are standard metrics recombined under a new reporting standard.

pith-pipeline@v1.1.0-grok45 · 14817 in / 3853 out tokens · 46861 ms · 2026-07-10T05:46:46.217819+00:00 · methodology

0 comments
read the original abstract

An LLM-as-judge score can move even when the candidate responses stay fixed, simply because the evaluator has changed. We treat this evaluator-replacement ambiguity as a measurement-validity problem. Across four judgment datasets, we compare two upgrade paths available in practice: scaling Qwen3 dense judges from 1.7B to 32B parameters and moving across MiniMax M2-M2.7 released APIs. The main pattern is that judge upgrades are not interchangeable: only Qwen3 1.7B to 4B gives a robust adjacent gain, while MiniMax adjacent releases do not. Stronger judges reduce but do not remove position and verbosity bias. Repeated-sample juries add little when errors are correlated. Structured debate can move decisions substantially, but without parser and fallback logs those shifts cannot be attributed to deliberation. We argue that LLM-as-judge reports should include dataset slices, bias probes, error-dependence estimates, and protocol audit trails.

Figures

Figures reproduced from arXiv: 2607.08535 by Xiaokun Yang, Yinghan Hou, Zongyou Yang.

Figure 1
Figure 1. Figure 1: LLM-as-judge as a measurement pipeline. A reported score depends not only on candidate responses, but also on the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Study design. The paper uses a Qwen3 parameter axis and a MiniMax release-generation axis as observable evaluator [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Capability–fairness association on LLMBar. Higher [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Jury behavior on LLMBar. Majority voting provides [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Structured-debate accuracy shift versus single-judge [PITH_FULL_IMAGE:figures/full_fig_p005_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

27 extracted references · 27 canonical work pages · 21 internal anchors

  1. [1]

    Vittorio Castelli, Rishav Chakravarti, Saswati Dana, Anthony Ferritto, Radu Florian, Martin Franz, Dinesh Garg, Dinesh Khandelwal, Scott McCarley, Mike McCawley, Mohamed Nasr, Lin Pan, Cezar Pendus, John Pitrelli, Saurabh Pujar, Salim Roukos, Andrzej Sakrajda, Avirup Sil, Rosario Uceda-Sosa, Todd Ward, and Rong Zhang. 2019. The TechQA Dataset.arXiv prepri...

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  2. [2]

    Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate.arXiv preprint arXiv:2308.07201(2023). arXiv:2308.07201

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  3. [3]

    1785.Essai sur l’application de l’analyse à la probabilité des décisions rendues à la pluralité des voix

    Marie Jean Antoine Nicolas Caritat de Condorcet. 1785.Essai sur l’application de l’analyse à la probabilité des décisions rendues à la pluralité des voix. Imprimerie Royale, Paris

    work page

  4. [4]

    Improving Factuality and Reasoning in Language Models through Multiagent Debate

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mor- datch. 2023. Improving Factuality and Reasoning in Language Models through Multiagent Debate. InInternational Conference on Machine Learning (ICML). arXiv:2305.14325

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  5. [5]

    Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators

    Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. 2024. Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators. arXiv preprint arXiv:2404.04475(2024). arXiv:2404.04475

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  6. [6]

    Steve Han, Gilberto Titericz Junior, Tom Balough, and Wenfei Zhou. 2025. Judge’s Verdict: A Comprehensive Analysis of LLM Judge Capability Through Human Agreement.arXiv preprint arXiv:2510.09738(2025). arXiv:2510.09738

    work page arXiv 2025

  7. [7]

    Debating with More Persuasive LLMs Leads to More Truthful Answers

    Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R. Bowman, Tim Rocktäschel, and Ethan Perez. 2024. Debating with More Persuasive LLMs Leads to More Truthful Answers.International Conference on Machine Learning (ICML)(2024). arXiv:2402.06782

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  8. [8]

    Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo

    work page

  9. [9]

    Prometheus: Inducing Fine-grained Evaluation Capability in Language Models

    Prometheus: Inducing Fine-grained Evaluation Capability in Language Models.International Conference on Learning Representations (ICLR)(2024). arXiv:2310.08491

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  10. [10]

    Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024. Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models.arXiv preprint arXiv:2405.01535(2024). arXiv:2405.01535

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  11. [11]

    Krishna K. Ladha. 1992. The Condorcet Jury Theorem, Free Speech, and Correlated Votes.American Journal of Political Science36, 3 (1992), 617–634. doi:10.2307/2111584

    work page doi:10.2307/2111584 1992

  12. [12]

    RewardBench: Evaluating Reward Models for Language Modeling

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. 2024. RewardBench: Evaluating Reward Models for Language Modeling.arXiv preprint arXiv:2403.13787(2024). arXiv:2403.13787

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  13. [13]

    Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu

    work page

  14. [14]

    Generative Judge for Evaluating Alignment

    Generative Judge for Evaluating Alignment.arXiv preprint arXiv:2310.05470 (2023). arXiv:2310.05470

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  15. [15]

    Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. 2023. Encouraging Divergent Think- ing in Large Language Models through Multi-Agent Debate.arXiv preprint arXiv:2305.19118(2023). arXiv:2305.19118

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  16. [16]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. InConference on Empirical Methods in Natural Language Processing (EMNLP). arXiv:2303.16634

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  17. [17]

    MiniMax. 2026. The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligence.arXiv preprint arXiv:2605.26494(2026). arXiv:2605.26494

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  18. [18]

    Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto. 2023. Ver- bosity Bias in Preference Labeling by Large Language Models.arXiv preprint arXiv:2310.10076(2023). arXiv:2310.10076

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  19. [19]

    Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023. Large Language Models are not Fair Evaluators.arXiv preprint arXiv:2305.17926(2023). arXiv:2305.17926

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  20. [20]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models.International Conference on Learning Representations (ICLR)(2023). arXiv:2203.11171

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  21. [21]

    Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, Wei Ye, Shikun Zhang, and Yue Zhang. 2023. PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization.arXiv preprint arXiv:2306.05087(2023). arXiv:2306.05087

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  22. [22]

    Yusuke Yamauchi, Taro Yano, and Masafumi Oyamada. 2025. An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliability.arXiv preprint arXiv:2506.13639(2025). arXiv:2506.13639 doi:10.48550/arXiv.2506.13639

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2506.13639 2025

  23. [23]

    An Yang et al. 2025. Qwen3 Technical Report.arXiv preprint arXiv:2505.09388 (2025). arXiv:2505.09388

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  24. [24]

    Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen

    work page

  25. [25]

    Evaluating Large Language Models at Evaluating Instruction Following

    Evaluating Large Language Models at Evaluating Instruction Following. InInternational Conference on Learning Representations (ICLR). arXiv:2310.07641

    work page internal anchor Pith review Pith/arXiv arXiv

  26. [26]

    Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT- Bench and Chatbot Arena. InAdvances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track. arXiv:2306.05685

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  27. [27]

    Lianghui Zhu, Xinggang Wang, and Xinlong Wang. 2023. JudgeLM: Fine-tuned Large Language Models are Scalable Judges.arXiv preprint arXiv:2310.17631 (2023). arXiv:2310.17631

    work page internal anchor Pith review Pith/arXiv arXiv 2023