{"id":"99f3dde8-e2a1-4032-be2c-188c5056bae3","arxiv_id":"2504.17087","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A multi-agent, rubric-based meta-judge pipeline improves the precision of selecting correct LLM judgments on JudgeBench from 61.71% to 77.26%, with majority voting.","lead":"This paper proposes a three-stage pipeline in which multiple LLMs score whether another LLM's judgment is trustworthy, then a threshold filters out low-quality judgments. On the JudgeBench dataset, majority voting raised the precision of selected judgments from 61.7% to 77.3%.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline gains may be an artifact of test-set-informed choices: T=4.5 and the two-agent panel were selected after inspecting precision on the same JudgeBench judgments, with no calibration split or threshold sensitivity analysis.","rationale":"The reader's weakest assumption (unvalidated T=4.5) is correct as far as it goes, but the more load-bearing issue is the broader evaluation protocol: threshold, agent selection, and per-task rubric choices are all made with access to the same test judgments used for the headline numbers. This creates several degrees of freedom for the reported improvements, and the absence of any calibration split, uncertainty estimate, or coverage statistic makes the gains hard to interpret. I would not move the verdict to REJECT because the method is clearly described, reproducible in principle, and a held-out evaluation could easily confirm the claim; the appropriate disposition is the same conditional verdict the reader reached, with the additional requirement that a calibration split and coverage reporting be provided. This is a methodological concern, not an attack on the authors, and the paper's own Limitations section acknowledges the small dataset, which is consistent with a conditional acceptance rather than an unconditional one.","tokens_in":11806,"tokens_out":5190,"duration_ms":47889,"concrete_test":"Split the 350 raw GPT-4o-mini judgments into a calibration set and a held-out evaluation set (stratified by JudgeBench category and pair ID). On the calibration set only, sweep T over {4.0, 4.2, 4.5, 4.8, 5.0}, select the best single-agent configuration, and fix the two-agent panel; then compute the majority-voting precision on the held-out set at the chosen T, along with the number of selected judgments and a 95% Clopper-Pearson interval. If the held-out precision is not clearly above the single-agent baseline (68.89) and raw judgments (61.71), the headline claim is weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central precision gains rest on several choices made after seeing the test data. Section 4.1 fixes T=4.5 'to ensure both high precision and robustness' without reporting a threshold sweep or a validation split, and T is used twice in majority voting (Eq. 2 and the final filter). Section 4.2.2 then builds the winning configuration from 'the two best-performing models based on meta-judging' from Table 3, which evaluates the same 350 raw judgments used for the headline numbers in Table 5. No held-out split is described anywhere in the manuscript. Because precision is computed only on the subset surviving the threshold, large reported gains could come from selecting an easy subset rather than genuinely better meta-judging: the paper never reports the number of selected judgments or any coverage/recall statistic, and the coding category has only 42 pairs. These are not internal inconsistencies, but they mean the 15.55%/8.37% improvements may partly reflect fitting the threshold, agents, and rubric to the evaluation set.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a three-stage meta-judge selection pipeline for LLM evaluation: (1) a rubric refined by GPT-4 from human-drafted criteria; (2) multi-agent scoring using weighted averaging, majority voting, or panel discussion; and (3) threshold-based filtering of judged responses. Using JudgeBench as external ground truth, the authors report that the pipeline improves selection precision over raw judgments by roughly 15.55% and over a single-agent baseline by roughly 8.37%, with Table 5 showing 77.26% overall precision for majority voting. The paper also includes single-agent rubric comparisons, a LLaMA judge experiment, and an ablation study of panel discussion structures.","tokens_in":12047,"tokens_out":5416,"duration_ms":47590,"significance":"If the reported gains hold under proper validation, the contribution is useful: it directly evaluates meta-judge selection precision against objective labels, which is a relatively underexplored aspect of LLM-as-a-judge research, and it systematically compares rubric lengths and multi-agent aggregation strategies. The use of JudgeBench's objective ground truth rather than human preference is a strength, as is the inclusion of multiple agents and several ablations. However, the headline numbers currently rest on test-set-informed choices of the threshold and panel members, with no held-out split, no sensitivity analysis, and no statistical uncertainty, so the quantitative claims are not yet established.","major_comments":[{"comment":"The threshold T=4.5 is introduced in Section 4.1 with the statement that it is 'set to a fixed value of 4.5 to ensure both high precision and robustness,' but no threshold sweep, calibration set, or sensitivity analysis is provided. Since T is used both in the majority-voting rule (Eq. 2) and in the final selection step (Section 3.3), and precision is measured only on the selected subset, the headline 15.55% and 8.37% improvements could partly reflect a threshold chosen after inspecting the same 350 judgments. Please report precision as a function of T, select T on a validation split, and show that the qualitative conclusions are stable over a range of T.","section":"§4.1 and §3.3"},{"comment":"The winning multi-agent configuration in Section 4.2.2 ('the two best-performing models based on meta-judging: GPT-4o-mini and Claude') is selected using Table 3, which is computed on the same 350 raw judgments that produce the headline numbers in Table 5. No held-out split or cross-validation is described anywhere in the manuscript, so the comparison in Table 5 is a test-set-informed model selection rather than an unbiased evaluation. Please add a validation split for selecting the threshold, rubric variant, and agent panel, and report final precision on a disjoint test set (or use nested cross-validation).","section":"§4.2.2, Table 5"},{"comment":"Tables 3–5 never report the number of selected judgments, coverage, or recall; precision alone is insufficient because every row's denominator is a different filtered subset. For example, the coding category contains 42 pairs, so the 85.71% precision in Table 5 could be 18/21 or 36/42, which carry very different evidentiary weight, and the reported gains are not accompanied by error bars, multiple seeds, or significance tests. Please report selected counts per category, coverage/recall, and uncertainty estimates such as bootstrap intervals.","section":"Tables 3–5"}],"minor_comments":[{"comment":"The phrase 'Arena-Hard Judge on JudgeBench' appears to conflate Arena-Hard with the JudgeBench dataset; please clarify the judge model and prompt used to generate the raw judgments.","section":"§4.1"},{"comment":"There is a typo in the Conclusion: 'JudgBench' should be 'JudgeBench'.","section":"§6"},{"comment":"The references list Chan et al. as both 2023a and 2023b for what appears to be the same ChatEval paper; please merge the duplicate entry.","section":"References"},{"comment":"Equation (3) uses S_j without defining it in the panel-discussion setting; please state explicitly whether S_j is the summarization agent's score for criterion j.","section":"§3.2.3, Eq. (3)"},{"comment":"The table captions use inconsistent terminology ('Raw Judgments' vs. 'raw judgment collection') and Table 4's 'Selection' row does not specify the exact rubric configuration; please make captions self-contained.","section":"Tables 3 and 5"},{"comment":"The Limitations paragraph appropriately acknowledges the limited 350-judgment set, but it does not mention the absence of a validation split for threshold and model selection, which is a more pressing limitation that should be disclosed.","section":"§5"}],"recommendation":"major_revision","confidential_remarks":"The core idea is plausible and the paper fits the journal's scope, but the experimental validation needs a proper split and uncertainty quantification before acceptance. I recommend major revision rather than rejection because the issues are fixable with additional experiments; however, if the authors cannot provide a held-out evaluation, the quantitative claims in the abstract should be substantially weakened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper proposes a three-stage meta-judge pipeline: rubric refinement with GPT-4, multi-agent scoring, and a threshold filter to select correct LLM judgments from a mixed pool. The core idea is sensible and the evaluation directly targets judge precision on JudgeBench's ground-truth labels, which is a step beyond the usual alignment-with-human approach. Worth reading, but the headline gains (15.55% over raw, 8.37% over single-agent) are probably inflated.\n\nWhat's actually new: prior work had single-agent meta-judging (Wu et al., Trivedi et al.) and multi-agent LLM panels (Verga, Chan), but this combination with a GPT-4-refined rubric and a score threshold used for filtering is new. The paper also benchmarks several agents and rubrics and includes ablations on panel size and summarization, which is more thorough than many papers in this subfield. The writing is clear, and the authors honestly note in Limitations that the dataset is only 350 judgments.\n\nThe soft spots are real. The threshold T=4.5 is fixed with no explanation of how it was derived, and the winning two-agent panel is chosen as the two best from Table 3 on the same 350 judgments used in Table 5. No validation split, no threshold sweep, no sensitivity analysis, so the precision gains likely include some curve-fitting to the test set. Also, precision-only reporting means we never learn how many judgments are discarded; there is no coverage or recall statistic, so the method could be cherry-picking easy cases. The coding category has only 42 pairs, so the 85.71% precision there could come from a handful of selected items. No error bars or significance tests anywhere.\n\nNone of this is fatal to the idea. The pipeline is plausible and the direction is useful for RLAIF and automated preference-data construction. But the specific numbers should not be trusted as they stand.\n\nWho should read it: anyone working on LLM-as-a-judge, meta-evaluation, or automated preference-data generation. It's a useful empirical datapoint, though not a definitive one.\n\nRecommendation: send to peer review with a request for major revision. Add a calibration/validation split, a threshold sensitivity curve, report selected-judgment counts and maybe recall, and give bootstrap confidence intervals. With those, it could be a solid contribution.","headline":"Useful meta-judge pipeline with a sensible design, but the headline precision gains are likely overstated because the threshold and agent panel were chosen on the same test set; needs a validation split before the numbers stand up.","tokens_in":12570,"tokens_out":2794,"would_cite":false,"duration_ms":24965,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A multi-agent panel of LLM meta-judges, scored with a weighted rubric and majority voting, selects correct judgments at 77.26% precision on JudgeBench, up from 61.71% for raw judgments.","keywords":["LLM-as-a-judge","meta-judge","multi-agent evaluation","majority voting","rubric-based scoring","precise judgment selection","JudgeBench","preference dataset construction"],"falsifier":"Hold out half of the JudgeBench judgments, sweep the selection threshold from 3 to 5 on the training half, apply the best threshold to the held-out half with majority voting, and compare precision against the 68.89% single-agent baseline; if the held-out gain largely disappears, the headline improvement is an artifact of choosing $T=4.5$ on the test set.","tokens_in":11646,"feed_emoji":"🗳️","tokens_out":6365,"duration_ms":50037,"temperature":0.7,"pith_summary":"The paper argues that the judgments LLMs produce — deciding which of two answers is better — can themselves be screened by other LLMs acting as meta-judges, and that this screening works better with a small panel of models and a detailed scoring rubric than with a single-model self-review. It proposes a three-stage pipeline: a human writes a basic rubric, GPT-4 expands it into a seven-criterion weighted scoring guide, several LLM agents independently score each judgment, and only judgments with an aggregate score above a fixed threshold are kept. On the JudgeBench dataset, the best configuration (majority voting across GPT-4o-mini and Claude) raises selection precision from 61.71% for raw judgments and 68.89% for a single-agent baseline to 77.26%. If the result holds, meta-judge scores can be used to build preference datasets for training judges without human annotation, simply by discarding low-scoring judgments.","feed_headline":"Majority-voting meta-judges lift selection precision to 77.3%","feed_subtitle":"Three-agent rubric scoring filters LLM verdicts, beating raw judgments by 15.6 points on JudgeBench.","key_machinery":"The mechanism is a three-stage meta-judge selection pipeline. First, a human-authored basic rubric is expanded by GPT-4 into a weighted seven-criterion scoring system (accuracy, logical soundness, completeness, fairness, relevance, clarity, impactfulness, with weights summing to 1). Second, several LLM agents independently score each judgment against the rubric; in the majority-voting strategy, the final score is set to 5 only when more than half the agents give a score above the threshold $T=4.5$, and to 1 otherwise. Third, judgments with scores above $T$ are selected as trustworthy. The key load-bearing identity is the threshold-based precision calculation $\text{Precision} = \text{TP}/(\text{TP}+\text{FP})$: if the meta-judge score actually tracks correctness, filtering on it trades away some recall for higher precision.","core_discovery":"The central discovery is that a meta-judge can be treated as a selection problem with a measurable precision: give several LLMs a rubric, have them score a judgment, and keep only the judgments that score above a threshold; then measure how often the kept judgments match the ground truth. In their runs, majority voting across two models improves precision by 15.55 percentage points over raw judgments and by 8.37 points over the single-agent baseline, with the largest gains in coding (19.04 points) and knowledge (6.61 points). The paper also finds that no single rubric or model dominates: complex reasoning favors long, detailed rubrics; coding favors shorter rubrics; and panel discussion, where agents share opinions, underperforms late aggregation like voting because opinions converge on hard cases.","pith_inferences":["If the precision gain generalizes, the same pipeline could be applied to screen human-written evaluations, not just LLM judgments, since the rubric scores the quality of the decisive conclusion rather than its alignment with human preference.","The fixed threshold of 4.5 is the fragile link: without a validation split or threshold sweep, the reported gains may be partly an artifact of choosing the cutoff on the test set; a held-out threshold selection would settle this.","A natural next experiment is to train a judge model on the high-precision subset selected by this meta-judge and measure whether the judge's accuracy improves on unseen tasks; the paper identifies this direction but does not run it."],"forward_implications":["Meta-judge scores can serve as an automated filter for building preference datasets used to train better judge models via reinforcement learning from AI feedback, removing the need for human labels on every judgment.","Majority voting across diverse models yields higher precision than any single agent tested, suggesting that panels of smaller models can replace a single large meta-judge at lower cost.","Task-specific rubric design is consequential: long rubrics help complex reasoning, short rubrics help coding, and irrelevant criteria can drag precision down.","Panel discussion, despite promising convergence, hurts meta-judging on hard problems; late aggregation strategies that preserve independent opinions are preferable."],"supporting_citations":[{"why":"Supplies the JudgeBench dataset with objective ground-truth labels for response pairs, the source of the raw judgments and the precision targets.","marker":"Tan et al., 2024"},{"why":"Provides the single-agent meta-judge baseline and the meta-rewarding idea that this pipeline extends.","marker":"Wu et al., 2024"},{"why":"Establishes the LLM-as-a-judge paradigm and documents its biases, motivating the need for an independent meta-judge step.","marker":"Zheng et al., 2023"},{"why":"Shows that panels of diverse models outperform single large judges, supporting the multi-agent aggregation design.","marker":"Verga et al., 2024"},{"why":"Basis for the panel discussion strategy in which agents exchange opinions during evaluation.","marker":"Chan et al., 2023b"},{"why":"Cited for the design of role-based collaborative discussion among agents, adapted into the meta-judge panel.","marker":"Li et al., 2024"},{"why":"Shows self-rationalization improves fine-grained judging, providing evidence for the meta-judge direction.","marker":"Trivedi et al., 2024"}],"fun_headline_variants":["Meta-judges lift LLM judgment precision by 15.6%","Three-agent meta-judging beats single-agent by 8.4%","Voting beats panel talk for meta-judging LLMs","Rubric-scored meta-judges boost coding verdicts 19 points","Late aggregation beats agent discussion for meta-judging"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the score cutoff of 4.5 was fixed in advance and reliably separates correct from incorrect judgments, since no validation split or threshold sweep is shown to justify it.","fun_headline_variants_meta":{"raw":{"variants":["Meta-judges lift LLM judgment precision by 15.6%","Three-agent meta-judging beats single-agent by 8.4%","Voting beats panel talk for meta-judging LLMs","Rubric-scored meta-judges boost coding verdicts 19 points","Late aggregation beats agent discussion for meta-judging"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000688,"raw_usage":{"total_tokens":3115,"prompt_tokens":939,"completion_tokens":2176,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":2088}},"tokens_in":555,"tokens_out":2176,"duration_ms":14583,"temperature":1.0,"reasoning_tokens":2088,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:49:05.791258+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold out half of the JudgeBench judgments, sweep the selection threshold from 3 to 5 on the training half, apply the best threshold to the held-out half with majority voting, and compare precision against the 68.89% single-agent baseline; if the held-out gain largely disappears, the headline improvement is an artifact of choosing $T=4.5$ on the test set.","supporting_citations":[],"review_version":1}