Pith. sign in

REVIEW 4 major objections 7 minor 36 references

Uncertainty-Aware Trust Estimation for Multi-LLM Systems via Structured Expert Judgement

T0 review · 4 major / 7 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLM ensembles should weight each model by its seed-question log score, not trust it equally.

desk verdict A clean, honest application of Cooke's classical expert weighting to LLM ensembles; the per-subject twist is useful, but the paper never validates that the verbalised probabilities it trusts are trustworthy enough to carry the load. read the letter →

arxiv 2607.20529 v1 pith:EZZQM7JX submitted 2026-07-10 cs.LG cs.AI

Jiawei Zheng , Jiazhen Zhang This is my paper
classification cs.LGcs.AI
keywords LLMensemblestrustestimationlog-scoreweightingcalibrationquestionsoverconfidenterrorsuncertaintycontext-awareaggregationMMLU-Pro
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that combining several language models by equal trust is the wrong default: an ensemble should first estimate which models deserve trust, using a small set of calibration questions with known answers. It adapts a classical expert-judgement rule in which each model's weight is the exponential of its average log-probability on correct seed answers, computed per subject, and the final output is the weighted mixture of all models' probability distributions. On multiple-choice benchmarks the rule behaves like simpler aggregators when all models are strong, but once weak or deliberately noisy models join, it keeps accuracy high while producing far fewer high-confidence errors than majority voting or equal averaging. The paper treats this as evidence that uncertainty-aware trust estimation, not accuracy or global reliability alone, is what makes an LLM ensemble robust.

What carries the argument

The mechanism has three parts: a logarithmic scoring rule, a softmax weighting, and a per-subject context split. For each subject c, every expert model i receives s_{i,c}, the average log probability it placed on the correct answers in that subject's seed questions; the model's trust weight is w_{i,c} = exp(tau s_{i,c}) / sum_m exp(tau s_{m,c}); and the ensemble prediction is the weighted mixture p_agg(y|x) = sum_i w_{i,c} p_i(y|x). The log score does the main work because it punishes confident wrong answers much more heavily than uncertain ones. The context split is what lets the same expert be trusted in one subject and ignored in another, which is the capability that global weighting lack

What would settle it

Take a heterogeneous panel and compute two sets of seed weights for the same experts: one from each model's verbalised probabilities, one from its token-level likelihoods (or temperature-scaled logits). If the two weight vectors rank the models differently on seed questions, or if the verbalised weights do not predict held-out seed performance, the calibration signal is an elicitation artefact and the central claim fails. A complementary check: add a strategically 'good on seeds, bad on targets' expert; if log-score weighting still outperforms equal averaging in that panel, the robustness clai

Watch

Extended reading notes

Core claim

The paper's central claim is that per-subject log-score trust weighting—giving each model a softmax weight derived from its average log probability on calibration questions in that subject—achieves the best accuracy-reliability balance among aggregation methods. The evidence is empirical: in heterogeneous and contaminated panels on MMLU and MMLU-Pro, the weighting rule attains the highest or near-highest accuracy while maintaining low overconfident-error rates, and it shifts weight away from noisy experts and toward subject specialists only inside their specialty. The mechanism is the log score's asymmetry: a model that assigns high probability to a wrong seed answer is penalised far more th

Load-bearing premise

The whole scheme rests on two linked premises: that the probabilities each model gives on seed questions are faithful enough to rank models by true reliability, and that seed questions come from the same distribution as the questions the ensemble will be asked; if either fails, the weights become noise and the mixture is not trust-weighted.

Editorial extensions

If this is right

  • Equal-weight and majority-vote ensembles are fragile once model quality varies; a calibration-weighted mixture degrades more slowly as unreliable experts are added.
  • Small seed sets already provide useful trust signals, and larger seed sets improve the per-subject weights, so calibration questions are a practical tool rather than a large data requirement.
  • Aggregation can reduce overconfident errors without sacrificing accuracy, which matters for high-stakes deployment where confident wrong answers are costly.
  • Subject-specific weighting recovers domain expertise: a specialist model is upweighted only in its domain, and a corrupted model is downweighted only where it fails.
  • Proper scoring rules, not raw accuracy, are the right basis for deciding which LLM to trust, because they distinguish confidently wrong from merely uncertain predictions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves open whether the same log-score weighting would transfer to open-ended generation, where probability distributions are not naturally defined; a testable extension is to elicit probabilities over sampled continuations and compare.
  • The method's success depends on verbalised probabilities being faithful; one can probe this by recomputing seed weights from token-level likelihoods or logits and checking whether the model ranking survives.
  • An adversarial expert could hide its failure mode from seed questions by behaving well on them; constructing seed sets that deliberately stress known weaknesses would test whether the robustness claim holds under strategic contamination.
  • The temperature parameter tau controls trust concentration; the paper shows moderate values help and very large values hurt, so an adaptive tau selected on held-out calibration data is a natural next step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper adapts Cooke's classical model from structured expert judgment to aggregate probabilistic predictions from multiple LLMs. It estimates subject-specific trust weights from held-out seed questions using log scores of verbalised probability distributions, then linearly pools experts with softmax-normalized weights (Eqs. 4-6). Experiments on MMLU and MMLU-Pro compare Cooke weighting against majority vote, equal averaging, global weighting, and accuracy weighting under homogeneous, heterogeneous, and contaminated panels, including subject-specific synthetic experts. The paper claims that Cooke weighting provides the best accuracy-reliability balance and remains robust when unreliable experts are introduced.

Significance. If the central premise is valid, the paper makes a useful contribution: it reframes LLM ensembling as trust estimation rather than prediction combination, and it provides a transparent held-out seed/target protocol with controlled synthetic contamination. Strengths include the clean separation of seed and target questions (§4.1), full tables with standard deviations (Tables 5-6), and ablations on seed size and temperature. However, the load-bearing verbalised-probability construct is not validated, and the headline comparisons lack statistical testing. The result is plausible but not yet fully established.

major comments (4)
  1. [§3.2, §A.2, Eqs. (4)-(6)] The method's core input is the verbalised probability distribution p_i(y|x) elicited by the prompt in §A.2. The paper treats these as faithful expert forecasts, but it also cites Xiong et al. (2023) showing verbalised confidence is often miscalibrated, and the Limitations acknowledge that 'inaccuracies in probability estimates may affect weighting.' No evidence is given that per-model elicitation distortions are small or comparable. If some models hedge toward uniform distributions while others overconfidently spike, the log scores in Eq. (4) rank by elicitation style, Eq. (5) turns this into weights, and Eq. (6) is not trust-weighted. Because NLL/Brier/OE are computed from the same verbalised distributions, the reported probabilistic advantages could be artifacts. Please validate the construct: per-model reliability diagrams, comparison with token-level probabilities on a subset, or agr
  2. [§4.1, Tables 2, 5, 6] The headline 'substantially outperforming' claim is not backed by significance tests or confidence intervals. For example, Table 2 shows Cooke MMLU accuracy 94.27 vs global weighting 93.66 and equal averaging 92.10; with only 5 splits and std devs of 0.17/0.28, the first gap may be within noise. OE differences such as 1.65 vs 1.68 vs 1.82 are very small. Please report paired tests or bootstrap CIs across the 5 random splits for the key comparisons, or state which differences are not significant. Without this, 'superior accuracy-reliability balance' is not statistically substantiated.
  3. [§5.3, Figs. 2, 6] The robustness advantage is asserted from visual curves: no numeric accuracy values at contamination ratios, error bars, or confidence intervals are given in the main text. The synthetic experts are described only narratively ('random', 'overconfident-wrong', 'biased') without an exact algorithm for generating their probability distributions, so the contaminated-panel results are hard to reproduce or verify. Please provide a numeric table with means and CIs at each contamination ratio and specify the generative procedure for each noisy-expert type, including the specialist and corrupted experts used in Tables 3-4.
  4. [Limitations] The Limitations correctly note that if the seed set does not reflect the target distribution, weights may not generalise. Because the experiments use random within-subject splits, seed-target shift is absent by construction, so the context-aware advantage (Tables 3-4) may overstate deployment performance. This does not invalidate the controlled comparison, but §6's generalisation language should be softened, or a covariate-shift experiment (e.g., training seeds on a subset of subjects and testing on others) should be added.
minor comments (7)
  1. [Eq. (4)] The summation index is inconsistent: it writes log p_i(y_j|x_j) but D^c_seed is defined over j; please clarify indices.
  2. [§2, §6] The disagreement-based method of Kruse et al. (2025) is discussed and argued to be fragile, but no experiment includes this or any other modern uncertainty-aware ensemble baseline. Adding such a comparison would strengthen the paper's positioning.
  3. [Appendix A.2] Model version details and inference settings (e.g., decoding temperature, number of JSON parsing retries) are not specified, which limits reproducibility.
  4. [General] No code or data availability statement is provided. Releasing the evaluation scripts and the verbalised probability outputs would help verify the results.
  5. [Figure 1] The legend entry 'accuracy' appears to label the accuracy-weighting baseline; consider renaming it to 'accuracy weighting' for consistency.
  6. [§3.2] Minor typo: 'In practise' should be 'In practice'.
  7. [Abstract] The abstract uses 'adversarial experts' but the synthetic experts are not adversarial in a game-theoretic sense; consider 'noisy or unreliable experts'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: weights are fit on held-out seed questions and evaluated on disjoint targets.

full rationale

The derivation chain is Eq. 4 (per-context seed log score) -> Eq. 5 (softmax weights) -> Eq. 6 (weighted mixture). The paper explicitly separates estimation from evaluation: 'For each subject, we split questions into a seed set and a target set. Seed questions are used only to estimate expert reliability and compute aggregation weights. Target questions are used only for the final evaluation.' Therefore the aggregated predictions on target questions are not used to determine the weights, and no target-set statistic is fed back into the weighting formula. The temperature τ is fixed at 1 for the main results, so there is no tuning on the evaluation set that would force the reported comparison. The accuracy/NLL/Brier/OE comparisons are empirical measurements on disjoint targets, not algebraic consequences of the fitting procedure. There are no author self-citations, and the load-bearing methodological reference (Cooke 1991; Colson and Cooke 2018) is external. The acknowledged limitations — verbalised probabilities may be miscalibrated and seed sets may not reflect the target distribution — are validity concerns about the input elicitation, not cases where the claimed prediction reduces by construction to a fitted parameter or to a self-citation. No circular step can be exhibited from the paper's equations or citations.

Assumptions & free parameters 3 free parameters · 5 assumptions · 1 invented entities

Stripped of machinery, the method is: elicit probability distributions, score each expert per subject on a seed set with the log rule, softmax with temperature tau, and mix. Everything else is inherited from Cooke (1991). The ledger records the hand-chosen constants the results depend on (tau, epsilon, gamma), the domain assumptions about verbalised confidence fidelity and seed-target stationarity that the paper itself flags in Limitations, and the under-specified synthetic experts that generate the most striking context-aware gains.

free parameters (3)
  • temperature tau = 1 (fixed); ablation explores 0-8
    Eq. 5 softmax temperature converting seed log scores into weights. Set to 1 by hand; the ablation in §B.4 shows accuracy, NLL and Brier vary substantially with tau, so the default is consequential, though fixing it before seeing target data avoids target-set fitting.
  • log-score clipping constant epsilon = not reported
    §3.3 states probabilities are 'clipped with a small constant epsilon' before computing log scores. Without its value, exact log scores — and therefore the weights — cannot be reproduced; if epsilon is comparable to the smallest emitted probabilities it would dominate the weighting.
  • overconfidence threshold gamma = 0.7
    The overconfident-error metric (Eq. 7) counts wrong answers whose max probability exceeds gamma. All OE comparisons in Tables 1-2, 5-6 and Figure 1 depend on this threshold, and no sensitivity analysis around it is reported.
assumptions (5)
  • domain assumption Elicited verbalised probabilities faithfully represent each model's predictive uncertainty, at least in relative ranking across experts.
    §3.2 and §4.1: weights, NLL, Brier and OE all rest on the JSON-elicited distributions; the paper's own citations (Xiong et al. 2023) document miscalibration of verbalised confidence, and the method assumes relative log scores still rank experts correctly.
  • domain assumption Seed-target exchangeability within each subject: per-subject seed questions are representative of the target questions.
    Weights estimated on D_seed are applied to the target split (§3.3, §4.1). The authors acknowledge in the Limitations section that seed-target mismatch would break generalization.
  • standard math The log scoring rule is the correct criterion for expert trust, inheriting Cooke's classical model.
    Cooke (1991); Colson & Cooke (2018). The paper uses only the log-score weighting component and drops the statistical-accuracy/informativeness decomposition of the classical model, implicitly asserting that seed log score alone captures reliability.
  • domain assumption The subject label is an observable, correct-granularity context variable c(x).
    §3.3 assumes c(x) is observable (subject metadata). In deployments without such labels, the context-aware advantage demonstrated in Tables 3-4 disappears by construction.
  • domain assumption Noisy experts are seed-detectable and do not collude strategically in ways that mask their unreliability during calibration.
    The contaminated settings assume log scores on seeds reveal unreliability. The Discussion notes that adversarial experts can 'coincide with incorrect consensus' for disagreement-based methods, but this failure mode is not tested against Cooke weighting.
invented entities (1)
  • Synthetic contaminated experts (random, overconfident-wrong, biased, specialist, corrupted)
    purpose: Controlled stress tests of aggregation robustness (§5.3); the specialist/corrupted variants exist specifically to demonstrate the context-aware trust mechanism.
    These are experimental constructs, not theoretical entities, but they carry the headline results. Their construction is described only qualitatively ('highly accurate on a designated subject but unreliable elsewhere', 'generally reliable but confidently wrong on a designated subject'), no sampling protocol is given, and they are configured (subjects biology/business/economics, 62.5% contamination ratio) to showcase the mechanism, so the magnitude of the claimed advantage is partly an artifact of the test design.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Uncertainty-Aware Trust Estimation for Multi-LLM Systems via Structured Expert Judgement." pith.science (2026). https://pith.science/paper/EZZQM7JX

@misc{pith2026260720529,
  author       = {Pith},
  title        = {Pith review of: Uncertainty-Aware Trust Estimation for Multi-LLM Systems via Structured Expert Judgement},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EZZQM7JX}},
  note         = {Machine review of arXiv:2607.20529}
}
read the original abstract

Large Language Model (LLM) ensembles are increasingly used to improve reliability by combining predictions from multiple LLMs. However, existing aggregation methods typically assume that all models are equally trustworthy, overlooking differences in uncertainty quality. This assumption is poorly suited to heterogeneous LLMs, whose reliability and capability vary significantly, making naive aggregation vulnerable to unreliable or adversarial experts. In this work, we formulate multi-LLM aggregation as a problem of uncertainty-aware trust estimation. We adapt structured expert judgment from decision theory, using context-aware calibration questions to estimate expert reliability based on the quality of its probabilistic predictions. Specifically, we employ Cooke-style log weighting, which penalises overconfident incorrect predictions and favours well-calibrated experts. We evaluate our approach on MMLU and MMLU-Pro across homogeneous, heterogeneous, and contaminated expert panels. Results show that while aggregation methods perform similarly in homogeneous settings, Cooke weighting becomes critical under heterogeneity and contamination. It achieves a superior accuracy-reliability balance and remains robust when unreliable experts are introduced. These findings suggest that Multi-LLM aggregation requires not just combining predictions, but calibrating trust under uncertainty.

Figures

Figures reproduced from arXiv: 2607.20529 by the authors.

Figure 2
Figure 2. Robustness curve as contamination ratio in [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Weight distribution over different models as [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Weight distribution across experts by subject [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Seed question size ablation analysis on hetero [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Robustness curves under different noisy expert types on MMLU. [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Weight distribution in heterogeneous panel on [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Weight distribution in heterogeneous panel on [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 12
Figure 12. Figure 12: Ablation analysis of τ on Brier score in subject-level contaminated panels. 0 2 4 6 8 0.3 0.4 0.5 0.6 0.7 0.8 0.9 NLL accuracy_weighting cooke_log_weighting global_weighting [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]
Figure 13
Figure 13. Figure 13: Ablation analysis of τ on NLL in subject￾level contaminated panels. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

36 extracted references · 4 canonical work pages

  1. [1]

    Complementing

    Hamidieh, Kimia and Thost, Veronika and Gerych, Walter and Yurochkin, Mikhail and Ghassemi, Marzyeh , year = 2025, month = oct, urldate =. Complementing. The

    2025

  2. [2]

    Kruse, Maya and Afshar, Majid and Khatwani, Saksham and Mayampurath, Anoop and Chen, Guanhua and Gao, Yanjun , year = 2025, month = sep, number =. Simple. doi:10.48550/arXiv.2507.07236 , urldate =. arXiv , keywords =:2507.07236 , primaryclass =

    arXiv doi:10.48550/arxiv.2507.07236 2025

  3. [4]

    Leng, Jixuan and Huang, Chengsong and Zhu, Banghua and Huang, Jiaxin , year = 2024, month = oct, urldate =. Taming. The

    2024

  4. [9]

    , year = 2025, month = feb, journal =

    Chen, Zhijun and Lu, Xiaodong and Li, Jingzheng and Chen, Pengpeng and Li, Zhuoran and Sun, Kai and Luo, Yuankai and Mao, Qianren and Li, Ming and Xiao, Likang and Yang, Dingqi and Huang, Xiao and Ban, Yikun and Sun, Hailong and Yu, Philip S. , year = 2025, month = feb, journal =. Harnessing

    2025

  5. [10]

    and Chi, Ed H

    Wang, Xuezhi and Wei, Jason and Schuurmans, Dale and Le, Quoc V. and Chi, Ed H. and Narang, Sharan and Chowdhery, Aakanksha and Zhou, Denny , year = 2022, month = sep, urldate =. Self-. The

    2022

  6. [13]

    Proceedings of the International Conference on Learning Representations (ICLR) , year=

    Measuring Massive Multitask Language Understanding , author=. Proceedings of the International Conference on Learning Representations (ICLR) , year=

  7. [14]

    Wang, Yubo and Ma, Xueguang and Zhang, Ge and Ni, Yuansheng and Chandra, Abhranil and Guo, Shiguang and Ren, Weiming and Arulraj, Aaran and He, Xuan and Jiang, Ziyan and Li, Tianle and Ku, Max and Wang, Kai and Zhuang, Alex and Fan, Rongqi and Yue, Xiang and Chen, Wenhu , year = 2024, month = jun, journal =

    2024

  8. [17]

    Xiong, Miao and Hu, Zhiyuan and Lu, Xinyang and Li, Yifei and Fu, Jie and He, Junxian and Hooi, Bryan , year = 2023, month = oct, urldate =. Can. The

    2023

Show all 36 references
  1. [19]

    Zhang, Wenqi and Shen, Yongliang and Wu, Linjuan and Peng, Qiuying and Wang, Jun and Zhuang, Yueting and Lu, Weiming , editor =. Self-. Proceedings of the 62nd. doi:10.18653/v1/2024.acl-long.197 , urldate =

    2024 doi

  2. [21]

    and Santoro, Adam and Lampinen, Andrew K

    Chan, Stephanie C.Y. and Santoro, Adam and Lampinen, Andrew K. and Wang, Jane X. and Singh, Aaditya K. and Richemond, Pierre H. and McClelland, James L. and Hill, Felix , year = 2022, month = nov, series =. Data Distributional Properties Drive Emergent In-Context Learning in T...

    2022

  3. [22]

    Ensemble

    Huang, Yichong and Feng, Xiaocheng and Li, Baohang and Xiang, Yang and Wang, Hui and Liu, Ting and Qin, Bing , year = 2024, month = nov, urldate =. Ensemble. The

    2024

  4. [23]

    Ai, Rui and Pan, Yuqi and. Beyond. Proceedings of the Forty-Third International Conference on Machine Learning (ICML) , abstract =

  5. [24]

    Debate or

    Choi, Hyeong Kyu and Zhu, Jerry and Li, Sharon , year = 2025, month = oct, urldate =. Debate or. The

    2025

  6. [25]

    2012 , publisher=

    Machine Learning: A Probabilistic Perspective , author=. 2012 , publisher=

    2012

  7. [26]

    Rui Ai, Yuqi Pan, David Simchi-Levi , Milind Tambe, and Haifeng Xu. 2026. Beyond Majority Voting : LLM Aggregation by Leveraging Higher-Order Information . In Proceedings of the Forty-Third International Conference on Machine Learning (ICML)

    2026

  8. [27]

    Chan, Adam Santoro, Andrew K

    Stephanie C.Y. Chan, Adam Santoro, Andrew K. Lampinen, Jane X. Wang, Aaditya K. Singh, Pierre H. Richemond, James L. McClelland, and Felix Hill. 2022. Data distributional properties drive emergent in-context learning in transformers. In Proceedings of the 36th International Co...

    2022

  9. [28]

    Zhijun Chen, Xiaodong Lu, Jingzheng Li, Pengpeng Chen, Zhuoran Li, Kai Sun, Yuankai Luo, Qianren Mao, Ming Li, Likang Xiao, Dingqi Yang, Xiao Huang, Yikun Ban, Hailong Sun, and Philip S. Yu. 2025. Harnessing Multiple Large Language Models : A Survey on LLM Ensemble . https://a...

    2025 arXiv

  10. [29]

    Hyeong Kyu Choi, Jerry Zhu, and Sharon Li. 2025. Debate or Vote : Which Yields Better Decisions in Multi-Agent Large Language Models ? In The Thirty-ninth Annual Conference on Neural Information Processing Systems

    2025

  11. [30]

    Colson and Roger M

    Abigail R. Colson and Roger M. Cooke. 2018. https://doi.org/10.1093/reep/rex022 Expert Elicitation : Using the Classical Model to Validate Experts ' Judgments . Review of Environmental Economics and Policy, 12(1):113--132

    2018 doi

  12. [31]

    Roger M Cooke. 1991. https://doi.org/10.1093/oso/9780195064650.001.0001 Experts in Uncertainty: Opinion and Subjective Probability in Science . Oxford University Press

    1991

  13. [32]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR)

    2021

  14. [33]

    Hsiu-Yuan Huang, Yutong Yang, Zhaoxi Zhang, Sanwoo Lee, and Yunfang Wu. 2024 a . https://doi.org/10.48550/arXiv.2410.15326 A Survey of Uncertainty Estimation in LLMs : Theory Meets Practice . Preprint, arXiv:2410.15326

    2024 doi arXiv

  15. [34]

    Yichong Huang, Xiaocheng Feng, Baohang Li, Yang Xiang, Hui Wang, Ting Liu, and Bing Qin. 2024 b . Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaboration . In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  16. [35]

    Maya Kruse, Majid Afshar, Saksham Khatwani, Anoop Mayampurath, Guanhua Chen, and Yanjun Gao. 2025. https://doi.org/10.18653/v1/2025.emnlp-main.1551 Simple yet effective: An information-theoretic approach to multi- LLM uncertainty quantification . In Proceedings of the 2025 Con...

    2025 doi

  17. [36]

    Fleming, Larisa Markeeva, Joe Heyward, Andrea Banino, Mrinal Mathur, Razvan Pascanu, Simon Osindero, Benedetto De Martino, Petar Veli c kovi \'c , and Viorica Patraucean

    Dharshan Kumaran, Stephen M. Fleming, Larisa Markeeva, Joe Heyward, Andrea Banino, Mrinal Mathur, Razvan Pascanu, Simon Osindero, Benedetto De Martino, Petar Veli c kovi \'c , and Viorica Patraucean. 2026. https://doi.org/10.1038/s42256-026-01217-9 Competing Biases underlie Ov...

    2026 doi

  18. [37]

    Jixuan Leng, Chengsong Huang, Banghua Zhu, and Jiaxin Huang. 2024. Taming Overconfidence in LLMs : Reward Calibration in RLHF . In The Thirteenth International Conference on Learning Representations

    2024

  19. [38]

    Nicole Lincoln, Nick Whitehouse, Jaron Mar, and Rivindu Perera. 2026. https://doi.org/10.48550/arXiv.2605.05532 A Few Good Clauses : Comparing LLMs vs Domain-Trained Small Language Models on Structured Contract Extraction . Preprint, arXiv:2605.05532

    2026 doi arXiv

  20. [39]

    Linyu Liu, Yu Pan, Xiaocheng Li, and Guanting Chen. 2024. https://doi.org/10.48550/arXiv.2404.15993 Uncertainty Estimation and Quantification for LLMs : A Simple Supervised Approach . Preprint, arXiv:2404.15993

    2024 doi arXiv

  21. [40]

    Kevin P. Murphy. 2012. Machine Learning: A Probabilistic Perspective. MIT Press

    2012

  22. [41]

    Jeremy Qin, Bang Liu, and Quoc Dinh Nguyen. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.142 Enhancing Healthcare LLM Trust with Atypical Presentations Recalibration . In Findings of the Association for Computational Linguistics : EMNLP 2024 , pages 2520--2537, Miami,...

    2024 doi

  23. [42]

    Mauricio Rivera, Jean-Fran c ois Godbout, Reihaneh Rabbany, and Kellin Pelrine. 2024. https://doi.org/10.18653/v1/2024.uncertainlp-1.12 Combining Confidence Elicitation and Sample-based Methods for Uncertainty Quantification in Misinformation Mitigation . In Proceedings of the...

    2024 doi

  24. [43]

    Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.330 Just Ask for Calibration : Strategies for Eliciting Calibrated Confidence Scores from Language Mo...

    2023 doi

  25. [44]

    Le, Ed H

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self- Consistency Improves Chain of Thought Reasoning in Language Models . In The Eleventh International Conference on Learning Representations

    2022

  26. [45]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024. MMLU-Pro : A More Robust and Challenging Multi-Task Language...

    2024 arXiv

  27. [46]

    Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2023. Can LLMs Express Their Uncertainty ? An Empirical Evaluation of Confidence Elicitation in LLMs . In The Twelfth International Conference on Learning Representations

    2023

  28. [47]

    Melton, and Rui Zhang

    Shuang Zhou, Jiashuo Wang, Zidu Xu, Song Wang, David Brauer, Lindsay Welton, Jacob Cogan, Yuen-Hei Chung, Lei Tian, Zaifu Zhan, Yu Hou, Mingquan Lin, Genevieve B. Melton, and Rui Zhang. 2025. https://doi.org/10.1038/s41746-025-02071-6 Uncertainty-aware large language models fo...

    2025 doi

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.