Pith. sign in

REVIEW 3 major objections 6 minor 3 cited by

Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments

T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A multi-agent panel of LLM meta-judges, scored with a weighted rubric and majority voting, selects correct judgments at 77.26% precision on JudgeBench, up from 61.71% for raw judgments.

desk verdict Useful meta-judge pipeline with a sensible design, but the headline precision gains are likely overstated because the threshold and agent panel were chosen on the same test set; needs a validation split before the numbers stand up. read the letter →

arxiv 2504.17087 v1 pith:VXRAOCJ6 submitted 2025-04-23 cs.AI

Yuran Li , Jama Hussein Mohamud , Chongren Sun , Di Wu , Benoit Boulet This is my paper · ORCID
classification cs.AI
keywords LLM-as-a-judgemeta-judgemulti-agentevaluationmajorityvotingrubric-basedscoringprecisejudgmentselectionJudgeBenchpreferencedatasetconstruction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the judgments LLMs produce — deciding which of two answers is better — can themselves be screened by other LLMs acting as meta-judges, and that this screening works better with a small panel of models and a detailed scoring rubric than with a single-model self-review. It proposes a three-stage pipeline: a human writes a basic rubric, GPT-4 expands it into a seven-criterion weighted scoring guide, several LLM agents independently score each judgment, and only judgments with an aggregate score above a fixed threshold are kept. On the JudgeBench dataset, the best configuration (majority voting across GPT-4o-mini and Claude) raises selection precision from 61.71% for raw judgments and 68.89% for a single-agent baseline to 77.26%. If the result holds, meta-judge scores can be used to build preference datasets for training judges without human annotation, simply by discarding low-scoring judgments.

What carries the argument

The mechanism is a three-stage meta-judge selection pipeline. First, a human-authored basic rubric is expanded by GPT-4 into a weighted seven-criterion scoring system (accuracy, logical soundness, completeness, fairness, relevance, clarity, impactfulness, with weights summing to 1). Second, several LLM agents independently score each judgment against the rubric; in the majority-voting strategy, the final score is set to 5 only when more than half the agents give a score above the threshold $T=4.5$, and to 1 otherwise. Third, judgments with scores above $T$ are selected as trustworthy. The key load-bearing identity is the threshold-based precision calculation $ ext{Precision} = ext{TP}/( ext{TP}+ ext{FP})$: if the meta-judge score actually tracks correctness, filtering on it trades away some recall for higher precision.

What would settle it

Hold out half of the JudgeBench judgments, sweep the selection threshold from 3 to 5 on the training half, apply the best threshold to the held-out half with majority voting, and compare precision against the 68.89% single-agent baseline; if the held-out gain largely disappears, the headline improvement is an artifact of choosing $T=4.5$ on the test set.

Watch

Extended reading notes

Core claim

The central discovery is that a meta-judge can be treated as a selection problem with a measurable precision: give several LLMs a rubric, have them score a judgment, and keep only the judgments that score above a threshold; then measure how often the kept judgments match the ground truth. In their runs, majority voting across two models improves precision by 15.55 percentage points over raw judgments and by 8.37 points over the single-agent baseline, with the largest gains in coding (19.04 points) and knowledge (6.61 points). The paper also finds that no single rubric or model dominates: complex reasoning favors long, detailed rubrics; coding favors shorter rubrics; and panel discussion, where agents share opinions, underperforms late aggregation like voting because opinions converge on hard cases.

Load-bearing premise

The load-bearing premise is that the score cutoff of 4.5 was fixed in advance and reliably separates correct from incorrect judgments, since no validation split or threshold sweep is shown to justify it.

Editorial extensions

If this is right

  • Meta-judge scores can serve as an automated filter for building preference datasets used to train better judge models via reinforcement learning from AI feedback, removing the need for human labels on every judgment.
  • Majority voting across diverse models yields higher precision than any single agent tested, suggesting that panels of smaller models can replace a single large meta-judge at lower cost.
  • Task-specific rubric design is consequential: long rubrics help complex reasoning, short rubrics help coding, and irrelevant criteria can drag precision down.
  • Panel discussion, despite promising convergence, hurts meta-judging on hard problems; late aggregation strategies that preserve independent opinions are preferable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the precision gain generalizes, the same pipeline could be applied to screen human-written evaluations, not just LLM judgments, since the rubric scores the quality of the decisive conclusion rather than its alignment with human preference.
  • The fixed threshold of 4.5 is the fragile link: without a validation split or threshold sweep, the reported gains may be partly an artifact of choosing the cutoff on the test set; a held-out threshold selection would settle this.
  • A natural next experiment is to train a judge model on the high-precision subset selected by this meta-judge and measure whether the judge's accuracy improves on unseen tasks; the paper identifies this direction but does not run it.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper proposes a three-stage meta-judge selection pipeline for LLM evaluation: (1) a rubric refined by GPT-4 from human-drafted criteria; (2) multi-agent scoring using weighted averaging, majority voting, or panel discussion; and (3) threshold-based filtering of judged responses. Using JudgeBench as external ground truth, the authors report that the pipeline improves selection precision over raw judgments by roughly 15.55% and over a single-agent baseline by roughly 8.37%, with Table 5 showing 77.26% overall precision for majority voting. The paper also includes single-agent rubric comparisons, a LLaMA judge experiment, and an ablation study of panel discussion structures.

Significance. If the reported gains hold under proper validation, the contribution is useful: it directly evaluates meta-judge selection precision against objective labels, which is a relatively underexplored aspect of LLM-as-a-judge research, and it systematically compares rubric lengths and multi-agent aggregation strategies. The use of JudgeBench's objective ground truth rather than human preference is a strength, as is the inclusion of multiple agents and several ablations. However, the headline numbers currently rest on test-set-informed choices of the threshold and panel members, with no held-out split, no sensitivity analysis, and no statistical uncertainty, so the quantitative claims are not yet established.

major comments (3)
  1. [§4.1 and §3.3] The threshold T=4.5 is introduced in Section 4.1 with the statement that it is 'set to a fixed value of 4.5 to ensure both high precision and robustness,' but no threshold sweep, calibration set, or sensitivity analysis is provided. Since T is used both in the majority-voting rule (Eq. 2) and in the final selection step (Section 3.3), and precision is measured only on the selected subset, the headline 15.55% and 8.37% improvements could partly reflect a threshold chosen after inspecting the same 350 judgments. Please report precision as a function of T, select T on a validation split, and show that the qualitative conclusions are stable over a range of T.
  2. [§4.2.2, Table 5] The winning multi-agent configuration in Section 4.2.2 ('the two best-performing models based on meta-judging: GPT-4o-mini and Claude') is selected using Table 3, which is computed on the same 350 raw judgments that produce the headline numbers in Table 5. No held-out split or cross-validation is described anywhere in the manuscript, so the comparison in Table 5 is a test-set-informed model selection rather than an unbiased evaluation. Please add a validation split for selecting the threshold, rubric variant, and agent panel, and report final precision on a disjoint test set (or use nested cross-validation).
  3. [Tables 3–5] Tables 3–5 never report the number of selected judgments, coverage, or recall; precision alone is insufficient because every row's denominator is a different filtered subset. For example, the coding category contains 42 pairs, so the 85.71% precision in Table 5 could be 18/21 or 36/42, which carry very different evidentiary weight, and the reported gains are not accompanied by error bars, multiple seeds, or significance tests. Please report selected counts per category, coverage/recall, and uncertainty estimates such as bootstrap intervals.
minor comments (6)
  1. [§4.1] The phrase 'Arena-Hard Judge on JudgeBench' appears to conflate Arena-Hard with the JudgeBench dataset; please clarify the judge model and prompt used to generate the raw judgments.
  2. [§6] There is a typo in the Conclusion: 'JudgBench' should be 'JudgeBench'.
  3. [References] The references list Chan et al. as both 2023a and 2023b for what appears to be the same ChatEval paper; please merge the duplicate entry.
  4. [§3.2.3, Eq. (3)] Equation (3) uses S_j without defining it in the panel-discussion setting; please state explicitly whether S_j is the summarization agent's score for criterion j.
  5. [Tables 3 and 5] The table captions use inconsistent terminology ('Raw Judgments' vs. 'raw judgment collection') and Table 4's 'Selection' row does not specify the exact rubric configuration; please make captions self-contained.
  6. [§5] The Limitations paragraph appropriately acknowledges the limited 350-judgment set, but it does not mention the absence of a validation split for threshold and model selection, which is a more pressing limitation that should be disclosed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the evaluation is anchored to external JudgeBench ground-truth labels, and the reported precision gains are empirical measurements rather than fitted predictions.

full rationale

The paper's central claim is an empirical precision comparison on the external JudgeBench benchmark (Tan et al., 2024). The meta-judge score is produced by prompted LLM agents using a rubric (Sections 3.1-3.2); the prompt template in Table 10 contains the question, answers, judgment, decision, and meta-judgment history but not the ground-truth label, so the score is not constructed from the label it is later compared against. Precision (Eq. 4) is therefore a measurement of agreement between an independently generated score and an external label, not a quantity equal to an input by construction. The fixed threshold T=4.5 and the choice of GPT-4o-mini and Claude as the two-agent panel are potential sources of test-set selection bias, and the paper's own Limitations section concedes the small 350-judgment dataset; but the paper does not fit a parameter to a subset and then predict a held-out quantity, nor does any equation reduce to another by definition. The citation to Li et al. (2024) for role-based panel discussion is not load-bearing for the headline majority-voting result, which is separately measured in Table 5 and shows panel discussion underperforming. Under the quoted-reduction standard, no circular step is present; the threshold and selection concerns are validity risks, not circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

No formal derivation; the pipeline is empirical. Main externally supplied quantities are JudgeBench labels. The threshold, weights, and model choices are set by the authors or GPT-4 on the evaluation set, so the central claim rests on these choices.

free parameters (3)
  • Selection threshold T = 4.5
    Chosen to 'ensure both high precision and robustness' (Section 4.1) with no validation split or sensitivity analysis; likely tuned on the same JudgeBench set used for reported precision.
  • Rubric criterion weights wc_j = 0.2, 0.2, 0.15, 0.1, 0.15, 0.1, 0.1 (Table 2)
    Assigned by GPT-4 during rubric refinement; these weights determine all meta-judge scores and are not justified by data or ablation.
  • Agent weights wagent_i = equal by default; adjustable
    The paper allows dynamic weights based on task performance, which would be fitted to data; experiments appear to use equal weights, but the freedom is a free parameter.
assumptions (3)
  • domain assumption JudgeBench ground-truth labels are correct and objective for the response pairs.
    Used as the label for judging whether a meta-judge-selected judgment is true (Section 4.1, Precision equation). If labels are wrong or ambiguous, precision is mismeasured.
  • domain assumption A meta-judge score above a threshold reliably separates correct from incorrect judgments.
    The entire selection mechanism assumes rubric-based scores correlate with correctness on unseen judgments; only tested on one set of 350 judgments.
  • domain assumption Scores from different LLM agents on a 1-5 scale are comparable and aggregate meaningfully.
    Weighted averaging and majority voting treat agent scores as commensurable, with no calibration between models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments." pith.science (2026). https://pith.science/paper/VXRAOCJ6

@misc{pith2026250417087,
  author       = {Pith},
  title        = {Pith review of: Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VXRAOCJ6}},
  note         = {Machine review of arXiv:2504.17087}
}
read the original abstract

Large language models (LLMs) are being widely applied across various fields, but as tasks become more complex, evaluating their responses is increasingly challenging. Compared to human evaluators, the use of LLMs to support performance evaluation offers a more efficient alternative. However, most studies focus mainly on aligning LLMs' judgments with human preferences, overlooking the existence of biases and mistakes in human judgment. Furthermore, how to select suitable LLM judgments given multiple potential LLM responses remains underexplored. To address these two aforementioned issues, we propose a three-stage meta-judge selection pipeline: 1) developing a comprehensive rubric with GPT-4 and human experts, 2) using three advanced LLM agents to score judgments, and 3) applying a threshold to filter out low-scoring judgments. Compared to methods using a single LLM as both judge and meta-judge, our pipeline introduces multi-agent collaboration and a more comprehensive rubric. Experimental results on the JudgeBench dataset show about 15.55\% improvement compared to raw judgments and about 8.37\% improvement over the single-agent baseline. Our work demonstrates the potential of LLMs as meta-judges and lays the foundation for future research on constructing preference datasets for LLM-as-a-judge reinforcement learning.

Figures

Figures reproduced from arXiv: 2504.17087 by the authors.

Figure 1
Figure 1. Method components and interactions. The actor [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. LLM-as-meta-judge framework. The rubric is predefined in the prompt design stage. We benchmark the judgment [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Cooperative discussion diagram. Each agent is [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Generation of Raw Judgments from the JudgeBench Dataset. A judgment is considered true if it matches the label for the answer pairs; otherwise, it is false. Evaluation Metric: JudgeBench provides the ground truth labels for the response pairs. The judgment GT la￾bel is…
Figure 5
Figure 5. Figure 5: Ablation study on the impact of the number and roles of agents across different tasks. The vertical axis represents [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. HIERA: Hierarchical Multi-Agent Relevance Assessment for Content Discovery Systems

    cs.MA 2026-08 conditional novelty 6.0 of 10

    A hierarchical multi-agent LLM framework, in which a coordinating analyzer synthesizes query and item specialist outputs, beats flat, staged, and ensemble LLM relevance judges on five content search datasets.

  2. Multi-Agent Debate Strategies: Survey, Taxonomy, and Challenges

    cs.SE 2026-07 accept novelty 6.0 of 10

    A systematic review of 141 papers derives a three-axis taxonomy of multi-agent debate design (participants, interaction, agreement) and shows the field has converged on a narrow default pattern.

  3. ABot-AgentOS: A General Robotic Agent OS with Lifelong Multi-modal Memory

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A robotic agent operating system with source-grounded graph memory and split-wise self-evolution improves long-horizon embodied task success and memory QA scores over baseline controllers.

Reference graph

Works this paper leans on

21 extracted references · 2 canonical work pages · cited by 3 Pith papers

  1. [1]

    Chateval: Towards better llm-based evaluators through multi-agent debate

    Chan, C.-M., Chen, W., Su, Y ., Yu, J., Xue, W., Zhang, S., Fu, J., and Liu, Z. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201, 2023a. Chan, C.-M., Chen, W., Su, Y ., Yu, J., Xue, W., Zhang, S., Fu, J., and Liu, Z. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv pre...

    arXiv

  2. [3]

    Livecodebench: Holistic and contamination free eval- uation of large language models for code

    Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. Livecodebench: Holistic and contamination free eval- uation of large language models for code. arXiv preprint arXiv:2403.07974,

    arXiv

  3. [5]

    Lin, S., Hilton, J., and Evans, O

    URL https://arxiv.org/abs/2403.19305. Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958,

    arXiv

  4. [8]

    Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615,

    arXiv

  5. [10]

    Y ., Cuadron, A., Wang, C., Popa, R

    Tan, S., Zhuang, S., Montgomery, K., Tang, W. Y ., Cuadron, A., Wang, C., Popa, R. A., and Stoica, I. Judgebench: A benchmark for evaluating llm-based judges. arXiv preprint arXiv:2410.12784,

    arXiv

  6. [11]

    S., Choudhary, K., Ramayapally, V

    Thakur, A. S., Choudhary, K., Ramayapally, V . S., Vaidyanathan, S., and Hupkes, D. Judging the judges: Evaluating alignment and vulnerabilities in llms-as- judges. arXiv preprint arXiv:2406.12624,

    arXiv

  7. [12]

    A., Rama- murthy, R., Stevens, K., Chaudhery, T

    Trivedi, P., Gulati, A., Molenschot, O., Rajeev, M. A., Rama- murthy, R., Stevens, K., Chaudhery, T. S., Jambholkar, J., Zou, J., and Rajani, N. Self-rationalization improves llm as a fine-grained judge. arXiv preprint arXiv:2410.05495,

    arXiv

  8. [13]

    Replacing judges with juries: Evaluating llm gener- ations with a panel of diverse models

    Verga, P., Hofstatter, S., Althammer, S., Su, Y ., Piktus, A., Arkhangorodsky, A., Xu, M., White, N., and Lewis, P. Replacing judges with juries: Evaluating llm gener- ations with a panel of diverse models. arXiv preprint arXiv:2404.18796,

    arXiv

Show all 21 references
  1. [14]

    Aligning large language models with human: A survey

    9 Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments Wang, Y ., Zhong, W., Li, L., Mi, F., Zeng, X., Huang, W., Shang, L., Jiang, X., and Liu, Q. Aligning large language models with human: A survey. arXiv preprint arXiv:2307.12966,

    arXiv

  2. [15]

    Mmlu-pro: A more robust and challenging multi-task language under- standing benchmark

    Wang, Y ., Ma, X., Zhang, G., Ni, Y ., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., et al. Mmlu-pro: A more robust and challenging multi-task language under- standing benchmark. arXiv preprint arXiv:2406.01574,

    arXiv

  3. [16]

    Livebench: A challenging, contamination-free llm benchmark

    White, C., Dooley, S., Roberts, M., Pal, A., Feuer, B., Jain, S., Shwartz-Ziv, R., Jain, N., Saifullah, K., Naidu, S., et al. Livebench: A challenging, contamination-free llm benchmark. arXiv preprint arXiv:2406.19314,

    arXiv

  4. [17]

    Meta-rewarding language models: Self-improving alignment with llm-as-a-meta- judge

    Wu, T., Yuan, W., Golovneva, O., Xu, J., Tian, Y ., Jiao, J., Weston, J., and Sukhbaatar, S. Meta-rewarding language models: Self-improving alignment with llm-as-a-meta- judge. arXiv preprint arXiv:2407.19594,

    arXiv

  5. [18]

    Do large language models know what they don’t know? arXiv preprint arXiv:2305.18153,

    Yin, Z., Sun, Q., Guo, Q., Wu, J., Qiu, X., and Huang, X. Do large language models know what they don’t know? arXiv preprint arXiv:2305.18153,

    arXiv

  6. [19]

    Mme-crs: multi-metric evaluation based on cor- relation re-scaling for evaluating open-domain dialogue

    Zhang, P., Hu, X., Yu, K., Wang, J., Han, S., Liu, C., and Yuan, C. Mme-crs: multi-metric evaluation based on cor- relation re-scaling for evaluating open-domain dialogue. arXiv preprint arXiv:2206.09403,

    arXiv

  7. [20]

    Q., and Artzi, Y

    Zhang, T., Kishore, V ., Wu, F., Weinberger, K. Q., and Artzi, Y . Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675,

    1904 arXiv

  8. [21]

    Detail of different rubrics In the single-agent precision comparison section, we analyzed the impact of four different rubric configurations

    10 Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments A. Detail of different rubrics In the single-agent precision comparison section, we analyzed the impact of four different rubric configurations. Section 3.1 provides an example of the ’logi...

    2024

  9. [2020]

    W., Iyyer, M., Zettlemoyer, L., and Hajishirzi, H

    Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-t., Koh, P. W., Iyyer, M., Zettlemoyer, L., and Hajishirzi, H. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv:2305.14251,

    arXiv

  10. [2021]

    Manakul, P., Liusie, A., and Gales, M. J. Mqag: Multiple- choice question answering and generation for assessing information consistency in summarization. arXiv preprint arXiv:2301.12307, 2023a. Manakul, P., Liusie, A., and Gales, M. J. Selfcheckgpt: Zero- resource black-box h...

    2005 arXiv

  11. [2022]

    Exploring llm prompting strategies for joint es- say scoring and feedback generation

    Stahl, M., Biermann, L., Nehring, A., and Wachsmuth, H. Exploring llm prompting strategies for joint es- say scoring and feedback generation. arXiv preprint arXiv:2404.15845,

    arXiv

  12. [2023]

    Guo, Z., Jin, R., Liu, C., Huang, Y ., Shi, D., Yu, L., Liu, Y ., Li, J., Xiong, B., Xiong, D., et al

    Accessed: 2024-10-13. Guo, Z., Jin, R., Liu, C., Huang, Y ., Shi, D., Yu, L., Liu, Y ., Li, J., Xiong, B., Xiong, D., et al. Evaluating large lan- guage models: A comprehensive survey. arXiv preprint arXiv:2310.19736,

    2024 arXiv

  13. [2024]

    R., Rockt¨aschel, T., and Perez, E

    Khan, A., Hughes, J., Valentine, D., Ruis, L., Sachan, K., Radhakrishnan, A., Grefenstette, E., Bowman, S. R., Rockt¨aschel, T., and Perez, E. Debating with more persua- sive llms leads to more truthful answers. arXiv preprint arXiv:2402.06782,

    arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.