Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

Efficient MAP Estimation of LLM Judgment Performance with Prior Transfer

T0 review · 5 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A mixture of two Beta-Binomial distributions plus conformal stopping estimates an LLM ensemble judge's error rate from ten samples, with a 3.37 percent margin on TruthfulQA.

desk verdict The empirical finding that Binomial is a bad fit for LLM ensemble judgments is solid, but the theoretical guarantee advertised in the abstract is not proven and the supporting math has real holes. read the letter →

arxiv 2504.12589 v1 pith:6WMOTI6R submitted 2025-04-17 cs.LG

Huaizhi Qu , Inyoung Choi , Zhen Tan , Song Wang , Sukwon Yun , Qi Long , Faizan Siddiqui , Kwonjoon Lee
show 1 more author
Tianlong Chen
This is my paper · ORCID
classification cs.LG
keywords LLM-as-a-judgeensemblejudgmentdistributionBeta-Binomialmixtureconformalpredictionadaptivestoppingpriortransfersampleefficiencymaximumaposterioriestimation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to show that the reliability of an ensemble of LLM judges can be estimated accurately from very few labeled samples, by replacing the standard Binomial assumption with a more realistic distributional model. The central claim is that a mixture of two Beta-Binomial distributions, one for easy questions and one for hard ones, captures how the number of correct judgments is distributed across an ensemble, and that a conformal-prediction rule can decide when enough samples have been seen. On top of this, a text-embedding similarity mechanism transfers distribution priors from open-source datasets to sharpen estimates when annotations are scarce. If the claim is right, auditing an LLM judge on a new dataset costs tens of labeled examples rather than thousands, and the headline result is an error margin as small as 3.37 percent on TruthfulQA from ten samples.

What carries the argument

The load-bearing object is the mixture-of-Beta-Binomials model, where a Beta-Binomial count is a Binomial whose success probability is itself drawn from a Beta distribution, permitting overdispersion beyond what a single Binomial allows; the two components reproduce the empirical two-peak pattern and keep the majority-voting error rate from decaying monotonically with ensemble size. The second mechanism is the conformal-prediction stopping rule, which monitors the empirical $(1-\epsilon)$-quantile of the nonconformity scores $|S_i - E[S]|$ and halts when the quantile moves by less than $\xi$; the paper derives from this rule the sample-size formula $\tau(1/\sqrt{r-1} - 1/\sqrt{r}) \leq \xi$ and the error-rate interval $(1 \pm \min(\xi, \tau/\sqrt{r})) \tilde{P}_{\mathrm{BB}}$. Finally, the transfer weight $\lambda_i = \log(r_i) \cdot \sigma(\alpha \cdot (\mathrm{CosSim}(\bar{E}_0, \bar{E}_i) - \beta))$ mediates how much each source dataset's fitted parameters contribute to the target estimate, suppressing contributions from textually distant datasets.

What would settle it

Take a fully labeled dataset, fix the ensemble, and run BetaConform's adaptive stopping on many random subsamples; compute the true majority-voting error rate from all labels and check whether the fraction of runs whose estimate falls inside the Proposition 5.2 interval $(1 \pm \min(\xi, \tau/\sqrt{r})) \tilde{P}_{\mathrm{BB}}$ is at least the nominal conformal confidence $1 - \epsilon$. If coverage is lower, the claimed guarantee fails at the sample sizes the paper uses.

Watch

Extended reading notes

Core claim

Stated on its own terms, the paper's discovery is that the judgment distribution of an LLM ensemble has two peaks, near all wrong and all correct, and is therefore modeled as $S \sim w \cdot \mathrm{BB}(k, \alpha_1, \beta_1) + (1 - w) \cdot \mathrm{BB}(k, \alpha_2, \beta_2)$, a mixture of two Beta-Binomial distributions that stratify questions by difficulty. Instead of assuming a fixed per-judge accuracy, the mixture is fitted to labeled samples by an expectation-maximization algorithm, and the majority-voting error rate is read off the mixture's cumulative distribution. To answer, the paper borrows conformal prediction: with nonconformity scores $s_i = |S_i - E[S]|$, sampling stops when the $(1-\epsilon)$-quantile changes by at most $\xi$, which, by the asymptotic rate $O_p(1/\sqrt{r})$ of quantile concentration, yields a required sample count $r$ satisfying $\tau(1/\sqrt{r-1} - 1/\sqrt{r}) \leq \xi$ and an error-rate bound $(1 \pm \min(\xi, \tau/\sqrt{r})) \tilde{P}_{\mathrm{BB}}$. When only ten samples are available, the estimate is improved by transferring distribution parameters from source datasets, weighted by $\lambda_i = \log(r_i) \cdot \sigma(\alpha \cdot (\mathrm{CosSim}(\bar{E}_0, \bar{E}_i) - \beta))$, so that textually similar datasets contribute more. The empirical demonstration includes a 3.37 percent error margin for a Llama ensemble on TruthfulQA with ten samples.

Load-bearing premise

The load-bearing premise is that the empirical quantile of the nonconformity scores converges to its population value at the $1/\sqrt{r}$ rate promised by large-sample theory, even when only tens of samples, sometimes ten, are used; if convergence is slower in that regime, the stopping rule, the sample counts, and the error-rate bound all lose their footing.

Editorial extensions

If this is right

  • An audit of an LLM judge ensemble needs only tens of labeled examples, around 56 for the thresholds $\xi = 0.01$, $\tau = 25$, to reach the paper's stopping criterion, instead of full-dataset annotation.
  • The two-peak mixture predicts that adding more judges to an ensemble does not keep lowering the majority-voting error rate; the rate plateaus, contradicting the Condorcet-style conclusion drawn from a Binomial model.
  • The conformal stopping rule provides a concrete, threshold-driven answer to how many samples are enough, with sample counts scaling roughly as $\tau(1/\sqrt{r-1} - 1/\sqrt{r}) \leq \xi$.
  • Prior transfer helps only when source and target datasets are textually similar; the sigmoid on embedding cosine similarity prevents distinct datasets from degrading the estimate.
  • Under the stopping threshold, the estimated majority-voting error rate stays within a relative factor $(1 \pm \min(\xi, \tau/\sqrt{r}))$ of the true mixture error rate.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Not drawn in the paper: the same conformal stopping logic should apply to other ensemble statistics besides the count of correct judgments, since its nonconformity score only requires a stable expectation-like quantity.
  • The sigmoid transfer weight implies a natural rejection rule: below a similarity threshold the prior contributes almost nothing, so practitioners could decide a priori whether transfer will help for their target dataset.
  • The 3.37 percent figure is an empirical error margin, not a demonstrated consequence of the asymptotic bound, because the proofs rely on quantile concentration at sample sizes where the asymptotics may be unreliable; treating the guarantee as literal would require a finite-sample check.
  • A testable extension: datasets with more than two difficulty strata might be better fit by three or more Beta-Binomial components, and model-selection criteria such as BIC could decide the component count per dataset.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. BetaConform models the number of correct judgments in an LLM ensemble as a two-component Beta-Binomial mixture, fits it with an EM-style estimator, uses a conformal-prediction-inspired quantile-stability rule to decide how many samples to label, and optionally transfers mixture parameters from textually similar source datasets. The experiments compare error margins against Binomial and single Beta-Binomial baselines on ten benchmarks with five LLM families and report lower average margins, including a 10-sample transfer result on TruthfulQA with a 3.37% error margin. The paper's headline contribution is a theoretical guarantee: Eq. (11) determines the required sample amount and Proposition 5.2 bounds the majority-voting error rate under adaptive stopping.

Significance. The problem is practically relevant, and the empirical evaluation is reasonably broad: ten datasets, five LLM families, thirty repetitions, and ablations of the transfer design. The observation that a Binomial model understates the plateau in majority-voting error is well supported by the figures. However, the load-bearing claim is the theoretical guarantee, and that claim is not established. The derivation of Eq. (11) is invalid, the step from a bound on E[S] to a bound on the tail error probability in Proposition 5.2 does not follow, and the EM update in Eq. (7) is internally inconsistent. The 10-sample transfer headline is a favorable single cell that is not covered by any stated guarantee and is contradicted qualitatively by several other cells in Table 2. If the theoretical-claim language were removed, the paper would be a heuristic empirical study; as submitted, the advertised central contribution fails.

major comments (5)
  1. [Appendix A.1, Eqs. (21)-(25)] The derivation of the sample-size formula is invalid. Combining q^r = q + O_p(1/sqrt(r)) and q^{r-1} = q + O_p(1/sqrt(r-1)) gives q^r - q^{r-1} = O_p(1/sqrt(r) + 1/sqrt(r-1)), whose leading order is O_p(1/sqrt(r-1)); there is no cancellation that yields O_p(1/sqrt(r-1) - 1/sqrt(r)). Moreover, an O_p statement is a stochastic asymptotic order and does not imply a deterministic inequality of the form tau*(1/sqrt(r-1) - 1/sqrt(r)) <= xi with a fixed tau. Consequently Proposition 5.1 and Eq. (11) are not proved, and the claimed r >= 56 (or the r >= 51 stated in Section 8) has no theoretical basis.
  2. [Appendix A.2, Eqs. (39)-(41), Proposition 5.2] The step from a multiplicative bound on E[S] to the same multiplicative bound on P(S < ceil(k/2)) is invalid. Monotonicity of the Beta-Binomial CDF implies only that probability statements are ordered as the distribution shifts; it does not control the ratio of tail probabilities. A two-component mixture whose mass is concentrated near the voting threshold can undergo a small relative change in its mean while the tail probability changes by a much larger factor, so Eq. (12) does not follow from Eq. (39). The proof also never uses the finite-sample coverage property of conformal prediction, so the phrase 'theoretically guaranteed' in the abstract is unsupported.
  3. [Section 4.3, Eq. (7)] The M-step update is inconsistent with the mixture model. Both mixture components are updated using the same responsibilities gamma_i^1, whereas the second component should use gamma_i^2 = 1 - gamma_i^1; as printed, the algorithm cannot maximize the mixture likelihood. In addition, the E-step in Eq. (6) uses the Beta density evaluated at p_i = S_i/k rather than the Beta-Binomial probability of S_i, so it is not the EM for the model in Assumption 4.1. The authors should either correct the updates to a proper Beta-Binomial EM or explicitly state that the procedure is a heuristic approximation.
  4. [Sections 5.1 and 8] The reported thresholds and the implied sample counts are inconsistent. Section 5 states that xi = 0.03 and tau = 25 lead to r >= 56, while Section 8 sets xi = 0.01 and says this requires at least r >= 51 samples. Substituting xi = 0.01 into Eq. (11) gives r approximately 116, not 51. The sample counts in Table 1 (roughly 40 to 61) therefore cannot validate Eq. (11) under the stated experimental configuration, and the text in Section 8.1 describing deviations of 3.14 to 12.86 samples from the 'estimated value 56' mixes two different threshold settings.
  5. [Section 6 and Table 2] The claim that the transfer mechanism will not cause performance degradation when source and target datasets are distinct is contradicted by the reported numbers. For example, HalluDial with GPT-3.5 degrades from 16.62 to 23.79, JudgeBench with GPT-3.5 from 8.23 to 14.36, RewardBench with Qwen from 9.47 to 12.84, and LLMBar with GPT-4 from 12.90 to 17.70. The abstract's 3.37% result on TruthfulQA is a single favorable cell and is not representative of the average transfer margins in Table 2.
minor comments (5)
  1. [Abstract and Section 4.3] The paper is described as MAP estimation, but no prior distribution is defined; the EM updates in Eq. (7) are maximum-likelihood updates. The MAP interpretation should be made explicit or the terminology should be changed.
  2. [Table 1] The Binomial rows appear to contain fewer values than the column headers imply, and the sample counts and standard deviations for the Binomial baseline are not reported. Please reformat the table or clarify the reported entries.
  3. [Section 6, Eq. (14)] The transfer hyperparameters alpha and beta are not specified anywhere in the paper, and no sensitivity analysis is reported for the sigmoid threshold embedded in Eq. (14). These values are needed for reproducibility.
  4. [Appendix A.2] The same quantile-stability passage appears twice in Appendix A.2, once around Eqs. (29)-(38) and again around Eqs. (33)-(40); this is an editing error and obscures the absence of a real proof of Proposition 5.2.
  5. [Section 3] The symbol S is used both for the random variable denoting the number of correct judgments and for the set of labeled samples; please disambiguate the notation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the mixture fit, conformal stopping, and transfer are validated against held-out full-data error rates and do not reduce to their inputs.

full rationale

BetaConform's central derivation is not circular. The mixture-of-Beta-Binomial parameters are estimated by EM from labeled samples (Eqs. 4-7), and the reported error margins compare the resulting estimates against the actual full-data majority-voting error rates (Tables 1-2), so the validation is external to the fitted values. The conformal stopping rule (Eq. 10) and sample-amount formula (Eq. 11) are derived from quantile asymptotics (Appendix A.1), not from the error-rate quantities being predicted. The transfer weights (Eq. 14) depend only on text embeddings and source sample counts, not on target labels or target error rates, and the transferred parameters (Eq. 15) are evaluated on held-out target data. No load-bearing self-citation or imported uniqueness theorem appears. The paper does contain proof gaps that undermine the advertised guarantee, notably Appendix A.2's unsupported jump from a proportional bound on E[S] (Eq. 39) to the same proportional bound on the tail probability P(S < ceil(k/2)) via monotonicity of the Beta-Binomial CDF, and the questionable O_p combination in Eq. (24), but these are correctness and rigor failures, not circular reductions of the outputs to the inputs.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The framework rests on a parametric mixture assumption, hand-chosen stopping constants, and unreported transfer hyperparameters. These are the main quantities the reader must accept on faith.

free parameters (3)
  • Beta-Binomial mixture parameters (w, alpha1, beta1, alpha2, beta2) per dataset/model = Estimated via EM on labeled samples (Eq. 7)
    The central claim depends on these fitted parameters to compute the ensemble error rate via Eq. (5). They are not derived from first principles.
  • Transfer hyperparameters alpha, beta (Eq. 14) = Not reported
    The sigmoid weight for source-to-target transfer is controlled by alpha and beta; the paper never states their values or a tuning procedure, so the transfer results are not reproducible without these choices.
  • Stopping threshold xi and scale tau = xi=0.01 or 0.03, tau=25
    The adaptive stopping criterion and the claimed sample requirement r>=56 (or r>=51) depend on these hand-chosen constants; the paper applies different xi values without resolving the inconsistency.
assumptions (3)
  • domain assumption Assumption 4.1: The judgment count S follows a mixture of two Beta-Binomial distributions (Eq. 4).
    Used throughout to derive the error rate (Eq. 5) and the theoretical bounds. The paper gives empirical motivation from two-peak histograms, but the exact two-component form is assumed, not derived.
  • standard math Samples are i.i.d. and the empirical quantile of nonconformity scores converges as in Bahadur's representation (Eq. 19) at the sample sizes used.
    Appendix A.1/A.2 rely on sqrt(r) asymptotic normality of sample quantiles to derive Eqs. (11) and (12). This is a standard asymptotic result but is applied at r around 40 to 60 and even r=10, where finite-sample coverage is not guaranteed.
  • domain assumption Text embedding cosine similarity is a sufficient measure of judgment-distribution transferability between datasets.
    The prior transfer mechanism (Eq. 14) assumes that semantically close datasets have similar LLM judgment distributions; the paper shows correlational evidence in Figure 5 but does not prove this.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient MAP Estimation of LLM Judgment Performance with Prior Transfer." pith.science (2026). https://pith.science/paper/6WMOTI6R

@misc{pith2026250412589,
  author       = {Pith},
  title        = {Pith review of: Efficient MAP Estimation of LLM Judgment Performance with Prior Transfer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6WMOTI6R}},
  note         = {Machine review of arXiv:2504.12589}
}
read the original abstract

LLM ensembles are widely used for LLM judges. However, how to estimate their accuracy, especially in an efficient way, is unknown. In this paper, we present a principled maximum a posteriori (MAP) framework for an economical and precise estimation of the performance of LLM ensemble judgment. We first propose a mixture of Beta-Binomial distributions to model the judgment distribution, revising from the vanilla Binomial distribution. Next, we introduce a conformal prediction-driven approach that enables adaptive stopping during iterative sampling to balance accuracy with efficiency. Furthermore, we design a prior transfer mechanism that utilizes learned distributions on open-source datasets to improve estimation on a target dataset when only scarce annotations are available. Finally, we present BetaConform, a framework that integrates our distribution assumption, adaptive stopping, and the prior transfer mechanism to deliver a theoretically guaranteed distribution estimation of LLM ensemble judgment with minimum labeled samples. BetaConform is also validated empirically. For instance, with only 10 samples from the TruthfulQA dataset, for a Llama ensembled judge, BetaConform gauges its performance with error margin as small as 3.37%.

Figures

Figures reproduced from arXiv: 2504.12589 by the authors.

Figure 1
Figure 1. In this paper, we aim to answer (1) how to estimate the judgment distribution of LLM ensemble on a dataset, and (2) how to achieve efficient estimation to reduce annotation effort. evaluate the performance of LLM judges. However, these datasets rely on human annotations, which are impractical at large scale due to the substantial time and financial costs of annotating. This challenge highlights the need of how to es… view at source ↗
Figure 2
Figure 2. Overview of BetaConform. Given a target dataset, adaptive stopping is adopted to determine the sample amount (b, Section 5). During iterative sampling, the sampling deviation is monitored by using conformal prediction. The sampling process stops when the deviation is sufficiently low. Next, the estimation of the small number of samples from the previous step is further enhanced by transferring distribution priors fr… view at source ↗
Figure 3
Figure 3. Comparison of judgment distributions among actual, Binomial, and ours. Llama-3.3-70B and GPT-4 ensembles of 11 models are tested on HaluEval and JudgeBench, respectively. The Binomial distribution is estimated by using single judge accuracy p. Our mixture distribution is estimated with 100 samples and scaled to the full dataset. Our distribution is consistently closer to the actual one. 1 3 5 7 9 11 Number of Judges… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Majority voting error rate of actual, Binomial, and our mixture distribution. Binomial uses single judge accuracy p. Our distribution is estimated with 100 random samples and tested for 3 times. The line denotes the average error rate and the shadow represents the stan…
Figure 5
Figure 5. Figure 5: Examples of distribution prior transfer. Splits from HaluEval form distinct clusters in the embedding space, and transfer does not degrade performance compared to only using target dataset samples. In contrast, topics in TruthfulQA exhibit closer proximity, where trans…
Figure 6
Figure 6. Figure 6: The actual number of samples under various thresholds ξ versus the theoretical value from Equation (11). The actual sample numbers match with the theoretical bound. HaluEval TruthfulQA HalluDial PRM800K BIG-bench TRAM JudgeBench RewardBench LLMBar ICE-Score COMP-Analys…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Knowing When to Stop: Predicting Execution-Consistency Convergence in Text-to-SQL

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Learned 1-D predictors of execution-consistency convergence stop Text-to-SQL sampling adaptively, beating fixed budgets and a Beta-Bernoulli rule on BIRD and two customer sets.

Reference graph

Works this paper leans on

35 extracted references · 5 canonical work pages · cited by 1 Pith paper

  1. [2]

    (27) The sample size requirement is given by Equation (11): τ 1√r− 1− 1√r ≤ξ

    (26) 12 Efficient MAP Estimation of LLM Judgment Performance with Prior Transfer The adaptive stopping criterion is given by Equation (10): qr 1−ϵ−qr−1 1−ϵ ≤ξ. (27) The sample size requirement is given by Equation (11): τ 1√r− 1− 1√r ≤ξ. (28) Based on the two equations and large number theory, we know that the difference between the quantile on samplesqr ...

    work page 2020

  2. [7]

    Lambert, N., Pyatkin, V ., Morrison, J., Miranda, L., Lin, B

    URL https: //arxiv.org/abs/2309.17012. Lambert, N., Pyatkin, V ., Morrison, J., Miranda, L., Lin, B. Y ., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y ., et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787,

    arXiv

  3. [8]

    Lee, H., Phatale, S., Mansoor, H., Mesnard, T., Ferret, J., Lu, K., Bishop, C., Hall, E., Carbune, V ., Rastogi, A., and Prakash, S

    URL https://arxiv.org/abs/2405.17428. Lee, H., Phatale, S., Mansoor, H., Mesnard, T., Ferret, J., Lu, K., Bishop, C., Hall, E., Carbune, V ., Rastogi, A., and Prakash, S. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback,

    arXiv

  4. [9]

    Li, D., Jiang, B., Huang, L., Beigi, A., Zhao, C., Tan, Z., Bhattacharjee, A., Jiang, Y ., Chen, C., Wu, T., et al

    URL https://arxiv.org/abs/2309.00267. Li, D., Jiang, B., Huang, L., Beigi, A., Zhao, C., Tan, Z., Bhattacharjee, A., Jiang, Y ., Chen, C., Wu, T., et al. From generation to judgment: Opportunities and challenges of llm-as-a-judge. arXiv preprint arXiv:2411.16594, 2024a. Li, J., Cheng, X., Zhao, W. X., Nie, J.-Y ., and Wen, J.- R. Halueval: A large-scale h...

    arXiv

  5. [10]

    Let’s verify step by step

    Lightman, H., Kosaraju, V ., Burda, Y ., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let’s verify step by step. arXiv preprint arXiv:2305.20050,

    arXiv

  6. [11]

    Truthfulqa: Measuring how models mimic human falsehoods

    Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958,

    arXiv

  7. [13]

    doi: 10.18653/v1/2023.emnlp-main

    Association for Computa- tional Linguistics. doi: 10.18653/v1/2023.emnlp-main

    doi:10.18653/v1/2023.emnlp-main 2023

  8. [15]

    FActScore: Fine-grained atomic evaluation of factual precision in long form text generation

    Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-t., Koh, P., Iyyer, M., Zettlemoyer, L., and Hajishirzi, H. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12076–12100, Si...

    work page 2023

Show all 35 references
  1. [16]

    doi: 10.18653/v1/2023

    Association for Computational Linguistics. doi: 10.18653/v1/2023. emnlp-main.741. URL https://aclanthology. org/2023.emnlp-main.741/. Mohri, C. and Hashimoto, T. Language models with conformal factuality guarantees,

    2023 doi

  2. [17]

    Moniri, B., Hassani, H., and Dobriban, E

    URL https: //arxiv.org/abs/2402.10978. Moniri, B., Hassani, H., and Dobriban, E. Evaluating the performance of large language models via debates,

    arXiv

  3. [18]

    OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F

    URL https://arxiv.org/abs/2406.11044. OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Bal- aji, S., Balcom, V ., Baltescu, P., Bao, H., Bavarian, M., Belgum, J....

    arXiv

  4. [19]

    Panickssery, A., Bowman, S

    URL https://arxiv.org/abs/2303.08774. Panickssery, A., Bowman, S. R., and Feng, S. Llm eval- uators recognize and favor their own generations,

    arXiv

  5. [20]

    Park, J., Jwa, S., Ren, M., Kim, D., and Choi, S

    URL https://arxiv.org/abs/2404.13076. Park, J., Jwa, S., Ren, M., Kim, D., and Choi, S. Offsetbias: Leveraging debiased data for tuning evaluators,

    arXiv

  6. [21]

    Qiu, J., Guo, D., Natalie, P., Noelle, P., Cheri, L., and Henry, T

    URL https://arxiv.org/abs/2407.06551. Qiu, J., Guo, D., Natalie, P., Noelle, P., Cheri, L., and Henry, T. R. Ensemble of large language models for curated labeling and rating of free-text data,

    arXiv

  7. [22]

    Schoenegger, P., Tuminauskaite, I., Park, P

    URL https://arxiv.org/abs/2501.08413. Schoenegger, P., Tuminauskaite, I., Park, P. S., and Tetlock, P. E. Wisdom of the silicon crowd: Llm ensemble predic- 10 Efficient MAP Estimation of LLM Judgment Performance with Prior Transfer tion capabilities rival human crowd accuracy,

  8. [23]

    Shafer, G

    URL https://arxiv.org/abs/2402.19379. Shafer, G. and V ovk, V . A tutorial on conformal prediction. Journal of Machine Learning Research, 9(3),

    arXiv

  9. [26]

    org/abs/2305.17926

    URL https://arxiv. org/abs/2305.17926. Wang, T., Kulikov, I., Golovneva, O., Yu, P., Yuan, W., Dwivedi-Yu, J., Pang, R. Y ., Fazel-Zarandi, M., Weston, J., and Li, X. Self-taught evaluators,

    arXiv

  10. [27]

    URL https: //arxiv.org/abs/2408.02666. Wang, Y . and Zhao, Y . Tram: Benchmarking temporal reasoning for large language models. arXiv preprint arXiv:2310.00835,

    arXiv

  11. [28]

    Yadkori, Y

    URL https://arxiv.org/ abs/2407.19594. Yadkori, Y . A., Kuzborskij, I., Stutz, D., Gy¨orgy, A., Fisch, A., Doucet, A., Beloshapka, I., Weng, W.-H., Yang, Y .-Y ., Szepesv´ari, C., Cemgil, A. T., and Tomasev, N. Miti- gating llm hallucinations via conformal abstention,

    arXiv

  12. [29]

    Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al

    URL https://arxiv.org/abs/2405.01563. Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115,

    arXiv

  13. [30]

    URL https://arxiv.org/abs/2402. 15043. Yuan, T., He, Z., Dong, L., Wang, Y ., Zhao, R., Xia, T., Xu, L., Zhou, B., Li, F., Zhang, Z., et al. R-judge: Benchmark- ing safety risk awareness for llm agents. arXiv preprint arXiv:2401.10019, 2024a. Yuan, W., Pang, R. Y ., Cho, K., S...

    arXiv

  14. [31]

    Zhao, R., Zhang, W., Chia, Y

    URL https://arxiv.org/abs/2501.07301. Zhao, R., Zhang, W., Chia, Y . K., Xu, W., Zhao, D., and Bing, L. Auto-arena: Automating llm evaluations with agent peer battles and committee discussions, 2024a. URL https://arxiv.org/abs/2405.20267. Zhao, Y ., Luo, Z., Tian, Y ., Lin, H....

    arXiv

  15. [32]

    Zhuo, T. Y . Ice-score: Instructing large language models to evaluate code. arXiv preprint arXiv:2304.14317,

    arXiv

  16. [34]

    and InternLM-2.5-20B (Cai et al., 2024)). We choose domains of hallucination detection (HaluEval, Li et al., 2023a; TruthfulQA, Lin et al., 2021; HalluDial, Luo et al., 2024), reasoning (PRM800K, Lightman et al., 2023; BIG-bench, Srivastava et al., 2022; TRAM, Wang & Zhao, 202...

    2024

  17. [35]

    Throughout all the experiments, the sampling temperature of all LLMs is set to

    and alignment (JudgeBench, Tan et al., 2024; RewardBench, Lambert et al., 2024; LLMBar, Zeng et al., 2023). Throughout all the experiments, the sampling temperature of all LLMs is set to

    2024

  18. [153]

    emnlp-main.153/

    URL https://aclanthology.org/2023. emnlp-main.153/. Luo, W., Shen, T., Li, W., Peng, G., Xuan, R., Wang, H., and Yang, X. Halludial: A large-scale benchmark for automatic dialogue-level hallucination evaluation. arXiv preprint arXiv:2406.07070,

    2023 arXiv

  19. [1996]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901,

    1901

  20. [2008]

    Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615,

    arXiv

  21. [2014]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  22. [2020]

    Internlm2 technical report

    Cai, Z., Cao, M., Chen, H., Chen, K., Chen, K., Chen, X., Chen, X., Chen, Z., Chen, Z., Chu, P., et al. Internlm2 technical report. arXiv preprint arXiv:2403.17297,

    arXiv

  23. [2021]

    G-eval: NLG evaluation using gpt-4 with better human alignment

    9 Efficient MAP Estimation of LLM Judgment Performance with Prior Transfer Liu, Y ., Iter, D., Xu, Y ., Wang, S., Xu, R., and Zhu, C. G-eval: NLG evaluation using gpt-4 with better human alignment. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conferen...

    2023

  24. [2022]

    Y ., Cuadron, A., Wang, C., Popa, R

    Tan, S., Zhuang, S., Montgomery, K., Tang, W. Y ., Cuadron, A., Wang, C., Popa, R. A., and Stoica, I. Judgebench: A benchmark for evaluating llm-based judges. arXiv preprint arXiv:2410.12784,

    arXiv

  25. [2023]

    Austen-Smith, D

    URL https: //arxiv.org/abs/2208.02814. Austen-Smith, D. and Banks, J. S. Information aggregation, rationality, and the condorcet jury theorem. American political science review, 90(1):34–45,

    arXiv

  26. [2024]

    org/abs/2404.04475

    URL https://arxiv. org/abs/2404.04475. Fontana, M., Zeni, G., and Vantini, S. Conformal prediction: a unified review of theory and new challenges. Bernoulli, 29(1):1–23,

    arXiv

  27. [2025]

    Chen, B., Wang, X., Peng, S., Litschko, R., Korho- nen, A., and Plank, B

    URL https://arxiv.org/abs/2501.10970. Chen, B., Wang, X., Peng, S., Litschko, R., Korho- nen, A., and Plank, B. ” seeing the big through the small”: Can llms approximate human judgment distri- butions on nli from a few explanations? arXiv preprint arXiv:2406.17600,

    arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.