Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Generalised Probabilistic Modelling and Improved Uncertainty Estimation in Comparative LLM-as-a-judge

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that picking pairwise LLM comparisons by the probability their ranking is reversed, computed under a Gaussian approximation of the score posterior, cuts the number of comparisons needed by about half to reach the same…

desk verdict Probability-of-reordering acquisition is the standout contribution to efficient LLM-as-a-judge; the Laplace-approximation caveat is real but acknowledged, and the empirical case is solid enough to warrant peer review. read the letter →

arxiv 2505.15240 v1 pith:UAXYNKYT submitted 2025-05-21 cs.AI cs.LGstat.ML

Yassir Fathullah , Mark J. F. Gales This is my paper
classification cs.AIcs.LGstat.ML
keywords LLM-as-a-judgepairwisecomparisonProduct-of-ExpertsuncertaintyestimationprobabilityofreorderingLaplaceapproximationactiveselectionrankingentropy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that choosing which pairwise comparisons to ask an LLM judge can be made much smarter by estimating, under a Gaussian approximation, the probability that the current ranking between two candidates is wrong. This 'probability of reordering' selection rule reaches the same ranking accuracy as previous uncertainty-based selection using roughly half the comparisons, cutting evaluation cost. The paper also shows that the previously proposed soft Bradley-Terry expert is one instance of a wider family of expert models, and that combining absolute and comparative scores improves rankings. If correct, LLM-based evaluation of candidate responses can be made substantially cheaper without losing fidelity to human judgements.

What carries the argument

The central object is the Laplace approximation of the posterior over candidate scores, $p(s|C_{1:K}) \approx \mathcal{N}(\mu, \Sigma)$, with $\mu$ the mode of the log-posterior and $\Sigma^{-1}$ the negative Hessian at the mode. This single Gaussian turns intractable pairwise and ranking uncertainties into closed forms: the variance of any score difference is $\Sigma_{ii} - 2\Sigma_{ij} + \Sigma_{jj}$, and the probability of reordering between $i$ and $j$ is $\Phi((\mu_j - \mu_i)/\sqrt{\Sigma_{ii} - 2\Sigma_{ij} + \Sigma_{jj}})$, which for selection simplifies to maximizing $(\Sigma_{ii} - 2\Sigma_{ij} + \Sigma_{jj})/(\mu_i - \mu_j)^2$. The second ingredient is the generalized expert decomposition $p(s_i - s_j | C_k) = f'(s_i - s_j) \tilde p(f(s_i - s_j) | C_k)$, which unifies previous expert choices and lets home-advantage bias enter as a shift $\Delta$ inside $f$. Together these give a tractable loop: fit the posterior, compute the covariance, pick the pair most likely to invert, query the LLM, and repeat.

What would settle it

Run the iterative selection on a benchmark where the true candidate scores are known and the posterior is deliberately multimodal (e.g., two clusters of near-tied candidates), comparing the Laplace-based probability-of-reordering rule against selection made with an exact posterior sampler (MCMC) on the same comparison budget; if the Laplace rule needs no fewer comparisons than random or minimum-uncertainty baselines, the claimed ~50% efficiency gain does not transfer to non-Gaussian posteriors.

Watch

Extended reading notes

Core claim

The paper establishes that the Product-of-Experts model for comparative LLM judging can be written through a generic monotone transform $f$ of score differences, with the familiar soft Bradley-Terry expert recovered by choosing $f$ as the logistic function and a Beta distribution over $f$. Within this framework, the paper introduces a new acquisition rule for iteratively selecting pairwise comparisons: maximize $(\Sigma_{ii} - 2\Sigma_{ij} + \Sigma_{jj})/(\mu_i - \mu_j)^2$, the ratio of the posterior variance of the score difference to the squared current score gap, which equals the Gaussian-approximated probability that the pair's ranking is reversed. Empirically this rule beats the prior minimum-uncertainty baseline, reaching 90% of the full-comparison Spearman correlation with roughly 50% fewer comparisons, and is more robust to batch selection on the 1056-candidate HANNA set. The paper further argues that the exact expert model matters little for final rankings, but that ranking-level uncertainty from the Laplace posterior entropy can flag unreliable rankings, and that adding absolute-score experts improves ranking quality.

Load-bearing premise

The whole efficiency argument depends on the Laplace approximation being a faithful picture of the score posterior, so that the variance and reordering probabilities it produces reflect the true uncertainty; the paper itself notes this fails for multimodal or skewed posteriors.

Editorial extensions

If this is right

  • LLM-as-a-judge evaluation can be roughly halved in cost: on SummEval and HANNA, the reordering rule reaches 90% of the full-comparison ranking accuracy with about 50% fewer queries than the previous minimum-uncertainty strategy.
  • Practitioners can choose the expert distribution (Beta, Gaussian, soft BT) for convenience, since the expert model has limited impact on final rankings once a reasonable set of comparisons is available.
  • Adding cheap absolute-score prompts alongside pairwise comparisons improves ranking accuracy in the Product-of-Experts model, without needing the full quadratic comparison set.
  • Ranking-level entropy computed from the Laplace posterior can flag low-quality rankings, with better detection for sigmoid-like link functions than for linear-Gaussian models.
  • Home-advantage bias in LLM judges can be corrected generically by estimating a shift parameter $\Delta$ in the transform $f$ by maximum likelihood, rather than bespoke rules per expert.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's ablation of the exponent in the selection rule suggests the best metric may not be exactly the reordering probability: $\epsilon = 0.5$ outperformed $\epsilon = 2$ on coherency, hinting that the optimal compromise between variance and gap may be task-dependent.
  • Because temperature annealing calibrates the LLM's probabilities without changing the implied ranking, a testable prescription follows for fixing the overconfidence the paper documents: ensemble several judge backbones or prompt variants, then apply the reordering rule to the averaged probabilities, which should restore monotone improvement in Spearman correlation as comparisons accumulate.
  • The same entropy proxy could be repurposed as a stopping rule for active evaluation, letting a system halt acquisition once the posterior entropy indicates a trustworthy ranking, which the paper motivates but does not explore.
  • The generic $f$-parameterization slots other classical pairwise models (Thurstone, cumulative link) into the same Product-of-Experts machinery, so the unification may extend beyond the Beta and Gaussian instances tested here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a generalized Product-of-Experts framework for comparative LLM-as-a-judge, where pairwise experts are constructed by a monotone change of variables applied to a base distribution. It introduces two acquisition functions for active comparison selection—variance of the score difference and probability of reordering—and an entropy-based measure of overall ranking uncertainty. It also combines comparative and absolute scoring within the same framework. Experiments on SummEval and HANNA with Flan-T5 and Qwen2.5 judges show that the choice of expert has little effect on final Spearman rank correlation, while the probability-of-reordering acquisition reduces the number of comparisons needed to reach 90% of full-set performance by roughly 50–60% relative to the minimum-uncertainty baseline. The paper also reports that entropy can identify low-performing context-level rankings with AUROC above random.

Significance. If the efficiency gains hold, the paper offers a simple and practical way to reduce the cost of LLM-based evaluation while preserving ranking quality. The probability-of-reordering metric is well motivated, is derived cleanly from the Gaussian posterior, and shows consistent improvements over the minimum-uncertainty baseline across two datasets and several judge models in the small-N regime. The unified view of experts and the comparative-plus-absolute combination are useful contributions that may open further work. The paper is also honest about its main limitation, the reliance on Laplace's approximation. The central caveat is that the acquisition metrics are derived from this approximation, and the paper does not verify its fidelity in the low-comparison regime where active selection is most consequential; the efficiency claim therefore needs a strengthening diagnostic before it can be fully accepted.

major comments (3)
  1. [Section 4.1/4.2, Eq. (5), Section 8] The central efficiency claim is load-bearing on the Laplace approximation. At the start of the active-learning loop only a handful of comparisons are available, and the posterior over the N scores is far from Gaussian; for the Φ-Gaussian expert in Eq. (2) the log-posterior is not concave, so a single mode and its local curvature can be a poor summary of the posterior. The paper's Limitations section concedes this, but no diagnostic is offered. Please add a concrete check, for example comparing the Laplace covariance with a Monte Carlo estimate of the posterior covariance for small K on a subset of SummEval contexts, and verifying that the pairs selected by Eq. (5) under the Laplace approximation are the same pairs that maximize the exact reordering probability. The efficiency numbers in Table 2 should be re-reported under the more exact selection, or the claim should be restricted to the settings where the approximation is verified.
  2. [Section 4.3, Table 1] The claim that the entropy of the Gaussian posterior is a useful proxy for ranking uncertainty is supported only by AUROC values relative to a random classifier. To establish that this measure adds value, the paper should compare it against simpler baselines such as the variance of the posterior score means, the average pairwise confidence, or the number of comparisons performed, and should report uncertainty intervals over contexts or seeds. Without this, it is unclear whether the reported AUROC gains (e.g., 64–67% for sigmoid-based experts versus 54.6% for the linear-Gaussian model) are due to the entropy metric itself or to the shape of the likelihood.
  3. [Section 6.3, Figure 3] The large-scale HANNA experiment demonstrates that probability-of-reordering batch selection beats random and minimum-uncertainty selection up to 1% of comparisons, but it does not report the number of comparisons needed to reach a 90% performance threshold. Since the abstract and Section 6.2 make quantitative efficiency claims based on SummEval (N=16), the paper should either provide an equivalent threshold-based comparison on HANNA or qualify the generality of the ~50% reduction claim to the small-N setting.
minor comments (4)
  1. [Abstract and Section 6.2] The phrase 'reducing the number of needed comparisons by ~50%' is relative to the minimum-uncertainty baseline, not to the full set of N(N-1) comparisons; please state this explicitly to avoid misinterpretation.
  2. [Table 2] Please describe how the 90% threshold and the reported standard deviations are computed (across contexts, seeds, or bootstrap samples); the methodology section currently defines the threshold only in words.
  3. [Appendix B] The statement that temperature annealing 'has no impact at all on the predicted rankings' is only true for the MAP scores under the soft Bradley-Terry model; temperature scaling does change the Hessian and therefore the uncertainty estimates used in the selection loop, so the sentence should be qualified.
  4. [Section 3.1] The change-of-variables formula for the expert should specify the support of \tilde{p} and state explicitly that the resulting p(s_i - s_j | C_k) is an unnormalized density; this will make the construction in Eq. (2) easier to follow.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the proposed uncertainty metrics are derived from the Laplace posterior and validated against human rankings; self-citations serve only as baselines.

full rationale

The paper's central efficiency claim is empirical and self-contained. The probability-of-reordering acquisition function (Eq. 5) is derived from the Laplace posterior by a monotone transformation of a Gaussian tail probability (Appendix C), and its evaluation uses Spearman correlation against human labels on SummEval/HANNA, not against the model's own uncertainty. No parameter is fitted to the target quantity and then reported as a prediction; the generalized expert model is a change-of-variables identity, not a derived prediction. Self-citations to Liusie et al. [2024b,c] supply the soft-BT baseline and the iterative selection loop, but these are published methods used as comparators, and their correctness is not assumed in deriving the new metrics. The Limitations section's caveat that Laplace's approximation may fail under multimodality or skewness is a correctness risk, not evidence of circularity. No uniqueness theorem or ansatz is imported via self-citation, and the efficiency gains are measured against an external benchmark (90% of the baseline's full-set Spearman performance). Accordingly, no specific circular step can be identified.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper introduces a generalised family of expert models via a change of variables, but the central efficiency claim rests on the Laplace approximation and the PoE factorization, both inherited or flagged as assumptions. Several hyperparameters (α, β, ε, prior variance) are free in the sense of being set by hand or swept on the test benchmark. No new physical or conceptual entities are introduced.

free parameters (4)
  • Beta shape offsets α, β = swept in ablation: (0,0), (0.1,0.1), (1,1), (2,2), (0.1,2), (2,0.1)
    Eq. (1) introduces arbitrary offsets to the Beta distribution; ablations show negligible impact on selection and final performance.
  • Uncertainty exponent ε = 0.5 chosen as best in Figure 2c; main results use ε=2
    Eq. (6) generalises variance and reordering metrics; exponent is swept on SummEval COH and ε=0.5 gives best performance, indicating the exact form is data-dependent.
  • Home advantage parameter Δ = estimated by maximizing likelihood
    Section 3.3: Δ models positional bias as a score shift f(s_i-s_j-Δ); fitted from data by MLE.
  • Unit Gaussian prior variance = 1
    Section 4.2: 'Starting with a unit Gaussian expert/prior on every score'; the prior scale is set by hand to 1.
assumptions (5)
  • domain assumption Bradley-Terry model assumptions: pairwise comparison probability depends only on score difference via sigmoid
    Section 2 (Background), soft BT expert from Liusie et al. [2024c] is the starting point.
  • domain assumption Product-of-Experts factorization: joint distribution of scores factorizes as product of pairwise experts
    Section 2 (Product-of-Experts), p(s|C1:K) ∝ ∏_k p(s_i-s_j|C_k); this is the core modelling assumption inherited from prior work.
  • domain assumption Laplace approximation yields a sufficiently accurate Gaussian posterior for uncertainty estimation
    Section 4.1 and Limitations: approximate p(s|C1:K) as N(μ,Σ); explicitly flagged as potentially inaccurate for multimodal/skewed posteriors.
  • domain assumption LLM judge probabilities p_ij are treated as pairwise preference probabilities, with debiasing via permutation or home advantage
    Section 3.3: inconsistent p_ij ≠ 1-p_ji are handled by permutation debiasing or home advantage; assumes debiased probabilities are meaningful.
  • standard math Change-of-variables formula for expert densities: p(s_i-s_j|C_k) = f'(s_i-s_j) \tilde{p}(f(s_i-s_j)|C_k)
    Section 3.1: standard probability density transformation; requires f monotone increasing and differentiable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Generalised Probabilistic Modelling and Improved Uncertainty Estimation in Comparative LLM-as-a-judge." pith.science (2026). https://pith.science/paper/UAXYNKYT

@misc{pith2026250515240,
  author       = {Pith},
  title        = {Pith review of: Generalised Probabilistic Modelling and Improved Uncertainty Estimation in Comparative LLM-as-a-judge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UAXYNKYT}},
  note         = {Machine review of arXiv:2505.15240}
}
read the original abstract

This paper explores generalised probabilistic modelling and uncertainty estimation in comparative LLM-as-a-judge frameworks. We show that existing Product-of-Experts methods are specific cases of a broader framework, enabling diverse modelling options. Furthermore, we propose improved uncertainty estimates for individual comparisons, enabling more efficient selection and achieving strong performance with fewer evaluations. We also introduce a method for estimating overall ranking uncertainty. Finally, we demonstrate that combining absolute and comparative scoring improves performance. Experiments show that the specific expert model has a limited impact on final rankings but our proposed uncertainty estimates, especially the probability of reordering, significantly improve the efficiency of systems reducing the number of needed comparisons by ~50%. Furthermore, ranking-level uncertainty metrics can be used to identify low-performing predictions, where the nature of the probabilistic model has a notable impact on the quality of the overall uncertainty.

Figures

Figures reproduced from arXiv: 2505.15240 by the authors.

Figure 1
Figure 1. The Spearman Rank Correlation when iteratively selecting the next examples of lowest confidence/highest [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Extended results on Flan-T5: (a) Comparing the soft BT model to the combined model which relies on additional absolute experts. Both use the probability of reordering as the selection criteria. (b) Selection performance. Varying the parameters α and β of the Beta distribution in Eq. (1). (c) Selection performance. Varying the parameter ϵ in Eq. (6) [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. The Spearman Rank Correlation when iteratively selecting the next batch [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: The reliability diagram of biased and debiased, standard and calibrated systems on the [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: The accuracy at a comparison-level when the examples of lowest confidence/highest uncertainty are rejected. [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: The reliability diagram of biased systems on the [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Qwen2.5-3B-Instruct: The Spearman Rank Correlation when iteratively selecting the next examples of lowest confidence/highest uncertainty. The baseline is the soft Bradley-Terry model with the minimum uncertainty metric. We also report the proposed variance and probabil…
Figure 8
Figure 8. Figure 8: Qwen2.5-7B-Instruct: The Spearman Rank Correlation when iteratively selecting the next examples of lowest confidence/highest uncertainty. The baseline is the soft Bradley-Terry model with the minimum uncertainty metric. We also report the proposed variance and probabil…
Figure 9
Figure 9. Figure 9: Qwen2.5-14B-Instruct: The Spearman Rank Correlation when iteratively selecting the next examples of lowest confidence/highest uncertainty. The baseline is the soft Bradley-Terry model with the minimum uncertainty metric. We also report the proposed variance and probabi…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth

    stat.ML 2026-01 conditional novelty 6.0 of 10

    A judge-aware Bradley–Terry model with per-judge discrimination parameters estimates LLM quality and judge reliability from unlabeled pairwise comparisons, with identifiability and asymptotic normality guarantees.

Reference graph

Works this paper leans on

49 extracted references · 26 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Categorical data analysis

    Alan Agresti. Categorical data analysis. John Wiley & Sons, 1990

    work page 1990

  3. [3]

    A computationally intensive ranking system for paired comparison data

    David Beaudoin and Tim Swartz. A computationally intensive ranking system for paired comparison data. Operations Research Perspectives, 5: 0 105--112, 2018

    work page 2018

  4. [4]

    Rank analysis of incomplete block designs: I

    Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952

    1952

  5. [5]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  6. [6]

    Sparks of artificial general intelligence: Early experiments with gpt-4

    S \'e bastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023

    arXiv 2023

  7. [7]

    Learning to rank: from pairwise approach to listwise approach

    Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. Learning to rank: from pairwise approach to listwise approach. In Proceedings of the 24th international conference on Machine learning, pages 129--136, 2007

    2007

  8. [8]

    Efficient bayesian inference for generalized bradley--terry models

    Francois Caron and Arnaud Doucet. Efficient bayesian inference for generalized bradley--terry models. Journal of Computational and Graphical Statistics, 21 0 (1): 0 174--196, 2012

    2012

Show all 49 references
  1. [9]

    Models for paired comparison data: A review with emphasis on dependent data

    Manuela Cattelan. Models for paired comparison data: A review with emphasis on dependent data. 2012

    2012

  2. [10]

    Humans or llms as the judge? a study on judgement biases, 2024 a

    Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. Humans or llms as the judge? a study on judgement biases, 2024 a

    2024

  3. [11]

    Chi, Xuezhi Wang, and Denny Zhou

    Xinyun Chen, Ryan A. Chi, Xuezhi Wang, and Denny Zhou. Premise order matters in reasoning with large language models, 2024 b . URL https://arxiv.org/abs/2402.08939

    2024 arXiv

  4. [12]

    Of human criteria and automatic metrics: A benchmark of the evaluation of story generation

    Cyril Chhun, Pierre Colombo, Fabian Suchanek, and Chlo \'e Clavel. Of human criteria and automatic metrics: A benchmark of the evaluation of story generation. In Proceedings of the 29th International Conference on Computational Linguistics, pages 5794--5836, 2022

    2022

  5. [13]

    Can large language models be an alternative to human evaluations? arXiv preprint arXiv:2305.01937, 2023

    Cheng-Han Chiang and Hung-yi Lee. Can large language models be an alternative to human evaluations? arXiv preprint arXiv:2305.01937, 2023

    2023 arXiv

  6. [14]

    Scaling instruction-finetuned language models

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 2024

    2024

  7. [15]

    Ranking by pairwise comparisons for swiss-system tournaments

    L \'a szl \'o Csat \'o . Ranking by pairwise comparisons for swiss-system tournaments. Central European Journal of Operations Research, 21: 0 783--803, 2013

    2013

  8. [16]

    The method of paired comparisons, volume 12

    Herbert Aron David. The method of paired comparisons, volume 12. London, 1963

    1963

  9. [17]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  10. [18]

    Rank aggregation methods for the web

    Cynthia Dwork, Ravi Kumar, Moni Naor, and Dandapani Sivakumar. Rank aggregation methods for the web. In Proceedings of the 10th international conference on World Wide Web, pages 613--622, 2001

    2001

  11. [19]

    Summeval: Re-evaluating summarization evaluation

    Alexander R Fabbri, Wojciech Kry \'s ci \'n ski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9: 0 391--409, 2021

    2021

  12. [20]

    Gptscore: Evaluate as you desire

    Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166, 2023

    2023 arXiv

  13. [21]

    Trueskill™: a bayesian skill rating system

    Ralf Herbrich, Tom Minka, and Thore Graepel. Trueskill™: a bayesian skill rating system. Advances in neural information processing systems, 19, 2006

    2006

  14. [22]

    Geoffrey E. Hinton. Products of experts. In Artificial Neural Networks, 1999. ICANN 99. Ninth International Conference on (Conf. Publ. No. 470), volume 1, pages 1--6. IET, 1999

    1999

  15. [23]

    Large language models are state-of-the-art evaluators of translation quality

    Tom Kocmi and Christian Federmann. Large language models are state-of-the-art evaluators of translation quality. arXiv preprint arXiv:2302.14520, 2023

    2023 arXiv

  16. [24]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12: 0 157--173, 2024 a . doi:10.1162/tacl_a_00...

    2024 doi

  17. [25]

    Learning to rank for information retrieval

    Tie-Yan Liu et al. Learning to rank for information retrieval. Foundations and Trends in Information Retrieval , 3 0 (3): 0 225--331, 2009

    2009

  18. [26]

    G -eval: NLG evaluation using gpt-4 with better human alignment

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G -eval: NLG evaluation using gpt-4 with better human alignment. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro...

    2023 doi

  19. [27]

    Aligning with human judgement: The role of pairwise preference in large language model evaluators, 2024 b

    Yinhong Liu, Han Zhou, Zhijiang Guo, Ehsan Shareghi, Ivan Vulić, Anna Korhonen, and Nigel Collier. Aligning with human judgement: The role of pairwise preference in large language model evaluators, 2024 b

    2024

  20. [28]

    Teacher-student training for debiasing: General permutation debiasing for large language models

    Adian Liusie, Yassir Fathullah, and Mark JF Gales. Teacher-student training for debiasing: General permutation debiasing for large language models. arXiv preprint arXiv:2403.13590, 2024 a

    2024 arXiv

  21. [29]

    LLM comparative assessment: Zero-shot NLG evaluation through pairwise comparisons using large language models

    Adian Liusie, Potsawee Manakul, and Mark Gales. LLM comparative assessment: Zero-shot NLG evaluation through pairwise comparisons using large language models. In Yvette Graham and Matthew Purver, editors, Proceedings of the 18th Conference of the European Chapter of the Associ...

    2024

  22. [30]

    Efficient llm comparative assessment: a product of experts framework for pairwise comparisons

    Adian Liusie, Vatsal Raina, Yassir Fathullah, and Mark Gales. Efficient llm comparative assessment: a product of experts framework for pairwise comparisons. arXiv preprint arXiv:2405.05894, 2024 c

    2024 arXiv

  23. [31]

    Stated choice methods: analysis and applications

    Jordan J Louviere, David A Hensher, and Joffre D Swait. Stated choice methods: analysis and applications. Cambridge university press, 2000

    2000

  24. [32]

    The structure of random utility models

    Charles F Manski. The structure of random utility models. Theory and decision, 8 0 (3): 0 229, 1977

    1977

  25. [33]

    Trueskill 2: An improved bayesian skill rating system

    Tom Minka, Ryan Cleven, and Yordan Zaykov. Trueskill 2: An improved bayesian skill rating system. Technical Report, 2018

    2018

  26. [34]

    Efficient computation of rankings from pairwise comparisons

    MEJ Newman. Efficient computation of rankings from pairwise comparisons. Journal of Machine Learning Research, 24 0 (238): 0 1--25, 2023

    2023

  27. [35]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 2...

    2022

  28. [36]

    Paireval: Open-domain dialogue evaluation with pairwise comparison

    ChaeHun Park, Minseok Choi, Dohyun Lee, and Jaegul Choo. Paireval: Open-domain dialogue evaluation with pairwise comparison. arXiv preprint arXiv:2404.01015, 2024

    2024 arXiv

  29. [37]

    Large language models are effective text rankers with pairwise ranking prompting

    Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, et al. Large language models are effective text rankers with pairwise ranking prompting. arXiv preprint arXiv:2306.17563, 2023

    2023 arXiv

  30. [38]

    Qwen2.5: A party of foundation models, September 2024

    Qwen Team . Qwen2.5: A party of foundation models, September 2024. URL https://qwenlm.github.io/blog/qwen2.5/

    2024

  31. [39]

    Finetuning llms for comparative assessment tasks, 2024

    Vatsal Raina, Adian Liusie, and Mark Gales. Finetuning llms for comparative assessment tasks, 2024. URL https://arxiv.org/abs/2409.15979

    2024 arXiv

  32. [40]

    Stanford alpaca: An instruction-following llama model, 2023

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-following llama model, 2023

    2023

  33. [41]

    Is chatgpt a good nlg evaluator? a preliminary study

    Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. Is chatgpt a good nlg evaluator? a preliminary study. arXiv preprint arXiv:2303.04048, 2023 a

    2023 arXiv

  34. [42]

    Large language models are not fair evaluators, 2023 b

    Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators, 2023 b

    2023

  35. [43]

    Primacy effect of C hat GPT

    Yiwei Wang, Yujun Cai, Muhao Chen, Yuxuan Liang, and Bryan Hooi. Primacy effect of C hat GPT . In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 108--115, Singapore, December 202...

    2023 doi

  36. [44]

    Self-instruct: Aligning language models with self-generated instructions

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560, 2022

    2022 arXiv

  37. [45]

    M. Welling. P roduct of experts. Scholarpedia, 2 0 (10): 0 3879, 2007. doi:10.4249/scholarpedia.3879. revision \#137078

    2007 doi

  38. [46]

    Die berechnung der turnier-ergebnisse als ein maximumproblem der wahrscheinlichkeitsrechnung

    Ernst Zermelo. Die berechnung der turnier-ergebnisse als ein maximumproblem der wahrscheinlichkeitsrechnung. Mathematische Zeitschrift, 29 0 (1): 0 436--460, 1929

    1929

  39. [47]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685, 2023

    2023 arXiv

  40. [48]

    Lima: Less is more for alignment

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, LILI YU, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. Lima: Less is more for alignment. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hard...

    2023

  41. [49]

    Judgelm: Fine-tuned large language models are scalable judges

    Lianghui Zhu, Xinggang Wang, and Xinlong Wang. Judgelm: Fine-tuned large language models are scalable judges. 2023

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.