Pith. sign in

REVIEW 2 major objections 2 minor 129 references

The Origins of Stochasticity: Comprehensive Investigations on Uncertainty Quantification for Large Language Models

T0 review · 2 major / 2 minor · reviewed 2026-06-26 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Consensus-based uncertainty quantification methods outperform other approaches for large language models, with larger models showing lower uncertainty.

desk verdict The four-source taxonomy and 21-method comparison are the actual contribution, but the 'consensus-based win' claim rests on an unshown disjoint partition. read the letter →

arxiv 2606.22792 v1 pith:PXMELEJA submitted 2026-06-22 cs.AI

Xiang-Jun Ou , Shuang Liang , Xin-Yu Hu , Rong-Hao Huang , Jing Wang , Shao-Qun Zhang This is my paper · ORCID
classification cs.AI
keywords uncertaintyquantificationlargelanguagemodelsstochasticityconsensus-basedmethodsscalinglawtaxonomyLLMevaluationmodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a granular taxonomy that attributes LLM uncertainty to four sources: input-level, parameter-level, token-level, and decoding-process. It categorizes UQ methods into Bayesian, ensemble, consensus-based, and single-pass approaches based on this taxonomy. Empirical evaluations of 21 methods across multiple LLMs and benchmarks reveal that consensus-based methods consistently perform best and that uncertainty decreases with model scale. This provides a practical way to assess and manage uncertainty in LLM applications.

What carries the argument

The four-source uncertainty taxonomy (input, parameter, token, decoding-process) that supports categorizing and evaluating UQ methods.

What would settle it

An observation that the uncertainty sources overlap significantly or that consensus-based methods fail to outperform on new tasks would falsify the main results.

Watch

Extended reading notes

Core claim

The paper claims that its four-source uncertainty taxonomy allows for a systematic categorization of UQ methods, and that experiments demonstrate consensus-based methods outperform others while larger model scales correlate with lower uncertainty estimates, suggesting an empirical scaling law.

Load-bearing premise

The proposed four-source taxonomy systematically and non-overlappingly attributes uncertainty sources in LLM generation.

Editorial extensions

If this is right

  • Effectiveness of UQ methods is sensitive to task types and generation settings.
  • Consensus-based methods like Deg and EigV consistently outperform other UQ approaches.
  • Larger model scales correlate with lower uncertainty estimates.
  • This indicates an empirical scaling law for LLM uncertainty.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The taxonomy might enable targeted improvements in UQ by addressing specific sources separately.
  • The scaling observation could imply that uncertainty issues diminish naturally with model advancement.
  • Consensus methods may be preferred in applications where reliability is critical.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper proposes a four-source taxonomy for uncertainty in LLMs (input-level, parameter-level, token-level, decoding-process sources) and correspondingly categorizes 21 UQ methods into Bayesian, ensemble, consensus-based, and single-pass approaches. It presents an evaluation framework and reports empirical results on Qwen3, Llama 3.2, and DeepSeek-V3 across TriviaQA, GSM8K, and HumanEval, claiming that (i) UQ effectiveness is sensitive to task type and generation settings, (ii) consensus-based methods (Deg, EigV) consistently outperform the other categories, and (iii) larger model scales correlate with lower uncertainty estimates, suggesting an empirical scaling law.

Significance. If the taxonomy can be shown to provide a non-overlapping partition and the empirical comparisons are placed on a statistically sound footing, the work would supply a structured diagnostic lens for LLM uncertainty that could guide method selection and connect theoretical sources to practical UQ performance. The reported scaling observation would also be of interest if replicated.

major comments (2)
  1. [Abstract and taxonomy section] Abstract and taxonomy section: the claim that the four sources 'systematically attribute' uncertainty without overlap is load-bearing for the subsequent categorization of the 21 methods and for the interpretation that consensus-based methods outperform because they target a distinct source; no formal disjointness argument, exhaustive mapping, or check for re-interpretability (e.g., a single-pass method also being parameter-level) is supplied.
  2. [Experimental results section] Experimental results section: the statement that Deg and EigV 'consistently outperform' other approaches requires, at minimum, per-benchmark tables with means, standard deviations or error bars, and a statistical test across the three model families; the abstract supplies none of these, leaving open whether observed gaps are significant or artifacts of implementation details within each category.
minor comments (2)
  1. [Notation and tables] Ensure every abbreviation (Deg, EigV, etc.) is defined on first use and that the exact assignment of each of the 21 methods to one of the four categories is tabulated for reproducibility.
  2. [Evaluation framework] Clarify the precise generation settings (temperature, top-p, etc.) and the exact metrics used for each benchmark so that the sensitivity claim in (i) can be verified.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments, which help clarify the presentation of our taxonomy and strengthen the empirical claims. We address each major comment below.

read point-by-point responses

  1. Referee: [Abstract and taxonomy section] Abstract and taxonomy section: the claim that the four sources 'systematically attribute' uncertainty without overlap is load-bearing for the subsequent categorization of the 21 methods and for the interpretation that consensus-based methods outperform because they target a distinct source; no formal disjointness argument, exhaustive mapping, or check for re-interpretability (e.g., a single-pass method also being parameter-level) is supplied.

    Authors: We agree that the manuscript does not supply a formal proof of disjointness or an exhaustive re-interpretability check. The taxonomy is motivated by the sequential stages of LLM generation (input encoding, parameter sampling during inference, per-token distribution, and decoding strategy), which we treat as primary attribution sources. While conceptual overlaps are possible in edge cases, the categorization of the 21 methods follows these primary attributions. We will add a dedicated subsection in the taxonomy section that discusses potential overlaps, provides an explicit mapping table, and acknowledges limitations in strict disjointness. revision: yes

  2. Referee: [Experimental results section] Experimental results section: the statement that Deg and EigV 'consistently outperform' other approaches requires, at minimum, per-benchmark tables with means, standard deviations or error bars, and a statistical test across the three model families; the abstract supplies none of these, leaving open whether observed gaps are significant or artifacts of implementation details within each category.

    Authors: We acknowledge that the current presentation of results does not include the requested statistical rigor in the reported tables or abstract. The full experimental section contains per-benchmark scores, but we will revise it to include (i) expanded tables with means and standard deviations computed over multiple runs, (ii) error bars in figures, and (iii) paired statistical tests (e.g., Wilcoxon signed-rank) across the three model families to assess whether performance gaps are significant. These additions will be reflected in both the results section and a revised abstract. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; empirical claims rest on external benchmarks and standard models

full rationale

The paper proposes a four-source taxonomy and corresponding four-way categorization of UQ methods, then reports empirical performance on public benchmarks (TriviaQA, GSM8K, HumanEval) and standard model families (Qwen3, Llama 3.2, DeepSeek-V3). No equations, fitted parameters, or self-citations are shown to reduce the central claims (consensus-based methods outperform; scaling law) to the taxonomy by construction. The taxonomy is presented as a proposed attribution scheme rather than a self-definitional mapping, and the evaluation framework uses independent data and metrics. This satisfies the default expectation of a self-contained empirical study against external references.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

The paper rests on the domain assumption that LLM generation can be decomposed into the four listed uncertainty sources; no free parameters or new physical entities are introduced.

assumptions (1)
  • domain assumption LLM generation involves distinct and attributable uncertainty sources at input, parameter, token, and decoding stages
    Invoked to motivate the granular taxonomy and method categorization.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Origins of Stochasticity: Comprehensive Investigations on Uncertainty Quantification for Large Language Models." pith.science (2026). https://pith.science/paper/PXMELEJA

@misc{pith2026260622792,
  author       = {Pith},
  title        = {Pith review of: The Origins of Stochasticity: Comprehensive Investigations on Uncertainty Quantification for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PXMELEJA}},
  note         = {Machine review of arXiv:2606.22792}
}
read the original abstract

Recent advancements in Large Language Models (LLMs) have enabled sophisticated reasoning and content generation, yet their inherent stochasticity poses significant challenges for ensuring predictive credibility. While traditional uncertainty taxonomy paradigms, such as the dichotomy of aleatoric and epistemic uncertainties, provide conceptual foundations, they often fail to capture the multi-component and multi-stage nature of LLM generation and struggle to evaluate the effectiveness of various Uncertainty Quantification (UQ) methods. In this paper, we propose a granular uncertainty taxonomy that systematically attributes LLM uncertainty into input-level, parameter-level, token-level, and decoding-process sources. Correspondingly, we categorize existing UQ methods into Bayesian, ensemble, consensus-based, and single-pass approaches. Furthermore, we introduce a comprehensive evaluation framework covering diverse generation settings and metrics. We empirically evaluate 21 typical UQ methods across three prominent LLM families, including Qwen3, Llama 3.2, and DeepSeek-V3, on benchmarks such as TriviaQA, GSM8K, and HumanEval. Our experimental results demonstrate that (i) the effectiveness of UQ methods is sensitive to task types and generation settings; (ii) consensus-based methods, typed Deg and EigV, consistently outperform other UQ approaches; and (iii) larger model scales correlate with lower uncertainty estimates, suggesting an empirical scaling law for LLM uncertainty. This work bridges the gap between theoretical origins and practical deployment, providing a versatile diagnostic tool for systematically quantifying uncertainty in LLM applications.

Figures

Figures reproduced from arXiv: 2606.22792 by the authors.

Figure 1
Figure 1. Timeline of remarkable uncertainty quantification methods, including Bayesian, ensemble, [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Illustration of four uncertainty sources that originate from the stochasticity of inputs, model [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Overall evaluation ranking of the investigated LLM UQ methods in the answer-only generation [PITH_FULL_IMAGE:figures/full_fig_p021_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Overall evaluation ranking of the investigated LLM UQ methods in the reasoning-augmented [PITH_FULL_IMAGE:figures/full_fig_p032_4.png]
Figure 5
Figure 5. Figure 5: Plots of output uncertainty versus accuracy across parameter scales. [PITH_FULL_IMAGE:figures/full_fig_p034_5.png]
Figure 6
Figure 6. Figure 6: Evaluation ranking of the investigated LLM UQ methods under the answer-only generation [PITH_FULL_IMAGE:figures/full_fig_p038_6.png]
Figure 7
Figure 7. Figure 7: Evaluation ranking of the investigated LLM UQ methods under the reasoning-augmented [PITH_FULL_IMAGE:figures/full_fig_p039_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

129 extracted references · 31 canonical work pages

  1. [1]

    Abbasi-Yadkori, Y., Kuzborskij, I., György, A., and Szepesvári, C. (2024). To believe or not to believe your LLM: Iterative prompting for estimating epistemic uncertainty. InAdvances in Neural Information Processing Systems 37, pages 58077–58117

    2024

  2. [2]

    K., Pleiss, G., Zemel, R

    Abe, T., Buchanan, E. K., Pleiss, G., Zemel, R. S., and Cunningham, J. P. (2022). Deep ensembles work, but are they necessary? InAdvances in Neural Information Processing Systems 35, pages 33646–33660

    2022

  3. [3]

    Ao, S., Rueger, S., and Siddharthan, A. (2024). CSS: Contrastive semantic similarity for uncertainty quantification of LLMs.arXiv preprint arXiv:2406.03158

    work page arXiv 2024

  4. [4]

    Baba, K., Liu, C., Kurita, S., and Sannai, A. (2025). Prover agent: An agent-based framework for formal mathematical proofs.arXiv preprint arXiv:2506.19923

    work page arXiv 2025

  5. [5]

    F., Kang, S., Huang, Z., Yaldiz, D

    Bakman, Y. F., Kang, S., Huang, Z., Yaldiz, D. N., Belém, C. G., Zhu, C., Kumar, A., Samuel, A., Avestimehr, S., Liu, D., and Karimireddy, S. P. (2025). Uncertainty as feature gaps: Epistemic uncertainty quantification of LLMs in contextual question-answering.arXiv preprint arXiv:2510.02671

    work page arXiv 2025

  6. [6]

    F., Yaldiz, D

    Bakman, Y. F., Yaldiz, D. N., Buyukates, B., Tao, C., Dimitriadis, D., and Avestimehr, S. (2024). MARS: Meaning-aware response scoring for uncertainty estimation in generative LLMs. InProceed- ings of the 20th Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 7752–7767

    2024

  7. [7]

    and Linander, H

    Balabanov, O. and Linander, H. (2024). Uncertainty quantification in fine-tuned LLMs using LoRA ensembles.arXiv preprint arXiv:2402.12264

    work page arXiv 2024

  8. [8]

    Band, N., Li, X., Ma, T., and Hashimoto, T. (2024). Linguistic calibration of long-form generations. InProceedings of the 41st International Conference on Machine Learning, pages 2732–2778

    2024

Show all 129 references
  1. [9]

    and Soatto, S

    Becker, E. and Soatto, S. (2024). Cycles of thought: Measuring LLM confidence through stable explanations.arXiv preprint arXiv:2406.03441

    2024

  2. [10]

    Blundell, C., Cornebise, J., Kavukcuoglu, K., and Wierstra, D. (2015). Weight uncertainty in neural network. InProceedings of the 32nd International Conference on Machine Learning, pages 1613–1622

    2015

  3. [11]

    Brier, W. G. (1950). Verification of forecasts expressed in terms of probability.Monthly weather review, 78(1):1–3

    1950

  4. [12]

    Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, 40 R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E....

    2020

  5. [13]

    Chen, C., Liu, K., Chen, Z., Gu, Y., Wu, Y., Tao, M., Fu, Z., and Ye, J. (2024). INSIDE: LLMs’ internal states retain the power of hallucination detection. InProceedings of the 12th International Conference on Learning Representations

    2024

  6. [14]

    and Mueller, J

    Chen, J. and Mueller, J. (2024). Quantifying uncertainty in answers from any language model and enhancing their trustworthiness. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 5186–5200

    2024

  7. [15]

    Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., B...

    2021 arXiv

  8. [16]

    Chen, Q., Qin, L., Liu, J., Peng, D., Guan, J., Wang, P., Hu, M., Zhou, Y., Gao, T., and Che, W. (2025a). Towards reasoning era: A survey of long chain-of-thought for reasoning large language models.arXiv preprint arXiv:2503.09567

    arXiv

  9. [17]

    Chen, T., Liu, X., Da, L., Chen, J., Papalexakis, V., and Wei, H. (2025b). Uncertainty quantification of large language models through multi-dimensional responses.arXiv preprint arXiv:2502.16820

  10. [18]

    and Li, Y

    Chen, W. and Li, Y. (2023). Calibrating transformers via sparse gaussian processes. InProceedings of the 11th International Conference on Learning Representations

    2023

  11. [19]

    J., Gibbs, I., and Candès, E

    Cherian, J. J., Gibbs, I., and Candès, E. J. (2024). Large language model validity via enhanced conformal prediction methods. InAdvances in Neural Information Processing Systems 37, pages 114812–114842

    2024

  12. [20]

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. (2021). Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168. 41

    2021 arXiv

  13. [21]

    Da, L., Chen, T., Cheng, L., and Wei, H. (2024). LLM uncertainty quantification through directional entailment graph and claim level response augmentation.arXiv preprint arXiv:2407.00994

    2024

  14. [22]

    Da, L., Liu, X., Dai, J., Cheng, L., Wang, Y., and Wei, H. (2025). Understanding the uncertainty of LLM explanations: A perspective based on reasoning topology. InProceedings of the 2nd Conference on Language Modeling

    2025

  15. [23]

    CanlinearprobesmeasureLLMuncertainty? arXiv preprint arXiv:2510.04108

    Dakhmouche, R., Letellier, A., andGorji, M.H.(2025). CanlinearprobesmeasureLLMuncertainty? arXiv preprint arXiv:2510.04108

    2025

  16. [24]

    Darrin, M., Piantanida, P., and Colombo, P. (2023). RainProof: An umbrella to shield text generator from out-of-distribution data. InProceedings of the 28th Conference on Empirical Methods in Natural Language Processing, pages 5831–5857

    2023

  17. [25]

    and Goadrich, M

    Davis, J. and Goadrich, M. H. (2006). The relationship between precision-recall and ROC curves. InProceedings of the 23rd International Conference on Machine Learning, pages 233–240

    2006

  18. [26]

    DeepSeek-V3 technical report.arXiv preprint arXiv:2412.19437

    DeepSeek-AI (2024). DeepSeek-V3 technical report.arXiv preprint arXiv:2412.19437

    2024 arXiv

  19. [27]

    Dinh, T. A. and Niehues, J. (2025). Are generative models underconfident? better quality estimation with boosted model probability. InProceedings of the 30th Conference on Empirical Methods in Natural Language Processing, pages 3364–3382

    2025

  20. [28]

    Du, X., Xiao, C., and Li, S. (2024). HaloScope: Harnessing unlabeled LLM generations for hallucination detection. InAdvances in Neural Information Processing Systems 37, pages 102948– 102972

    2024

  21. [29]

    Duan, J., Cheng, H., Wang, S., Zavalny, A., Wang, C., Xu, R., Kailkhura, B., and Xu, K. (2024). Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models. InProceedings of the 62nd Annual Meeting of the Association fo...

    2024

  22. [30]

    Fadeeva, E., Rubashevskii, A., Shelmanov, A., Petrakov, S., Li, H., Mubarak, H., Tsymbalov, E., Kuzmin, G., Panchenko, A., Baldwin, T., Nakov, P., and Panov, M. (2024). Fact-checking the output of large language models via token-level uncertainty quantification. InProceedings ...

    2024

  23. [31]

    Fan, A., Lewis, M., and Dauphin, Y. N. (2018). Hierarchical neural story generation. InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 889–898

    2018

  24. [32]

    Farquhar, S., Kossen, J., Kuhn, L., and Gal, Y. (2024). Detecting hallucinations in large language models using semantic entropy.Nature, 630(8017):625–630. 42

    2024

  25. [33]

    Unsupervisedqualityestimationforneuralmachinetranslation.Transactions of the Association for Computational Linguistics, 8:539–555

    Fomicheva, M., Sun, S., Yankovskaya, L., Blain, F., Guzmán, F., Fishel, M., Aletras, N., Chaudhary, V., andSpecia, L.(2020). Unsupervisedqualityestimationforneuralmachinetranslation.Transactions of the Association for Computational Linguistics, 8:539–555

    2020

  26. [34]

    and Ghahramani, Z

    Gal, Y. and Ghahramani, Z. (2016). Dropout as a bayesian approximation: Representing model uncertainty in deep learning. InProceedings of The 33rd International Conference on Machine Learning, pages 1050–1059

    2016

  27. [35]

    Gao, X., Zhang, J., Mouatadid, L., and Das, K. (2024). SPUQ: Perturbation-based uncertainty quantification for large language models. InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, pages 2336–2346

    2024

  28. [36]

    Geifman, Y., Uziel, G., and El-Yaniv, R. (2019). Bias-reduced uncertainty estimation for deep neural classifiers. InProceedings of the 7th International Conference on Learning Representations

    2019

  29. [37]

    Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. (2017). On calibration of modern neural networks. InProceedings of the 34th International Conference on Machine Learning, pages 1321–1330

    2017

  30. [38]

    He, J., Gong, Y., Chen, K., Lin, Z., Wei, C., and Zhao, Y. (2023). LLM factoscope: Uncovering LLMs’ factual discernment through inner states analysis.arXiv preprint arXiv:2312.16374

    2023

  31. [39]

    He, P., Liu, X., Gao, J., and Chen, W. (2021). DeBERTa: Decoding-enhanced BERT with disen- tangled attention. InProceedings of the 9th International Conference on Learning Representations

    2021

  32. [40]

    Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. (2020). The curious case of neural text degeneration. InProceedings of the 8th International Conference on Learning Representations

    2020

  33. [41]

    Hou, B., Liu, Y., Qian, K., Andreas, J., Chang, S., and Zhang, Y. (2024). Decomposing uncer- tainty for large language models through input clarification ensembling. InProceedings of the 35th International Conference on Machine Learning, pages 19023–19042

    2024

  34. [42]

    and Waegeman, W

    Hüllermeier, E. and Waegeman, W. (2021). Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods.Machine Learning, 110(3):457–506

    2021

  35. [43]

    S., and Zettlemoyer, L

    Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. (2017). TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1601–1611

    2017

  36. [44]

    V., Laga, H., Boussaïd, F., Buntine, W

    Jospin, L. V., Laga, H., Boussaïd, F., Buntine, W. L., and Bennamoun, M. (2022). Hands-on bayesian neural networks - A tutorial for deep learning users.IEEE Computational Intelligence Magazine, 17(2):29–48. 43

    2022

  37. [45]

    and Yuki, A

    Junya, T. and Yuki, A. (2019). Relevant and informative response generation using pointwise mutual information. InProceedings of the 1st Workshop on NLP for Conversational AI, pages 133–138

    2019

  38. [46]

    Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield- Dodds, Z., DasSarma, N., Tran-Johnson, E., Johnston, S., Showk, S. E., Jones, A., Elhage, N., Hume, T., Chen, A., Bai, Y., Bowman, S., Fort, S., Ganguli, D., Hernandez, D., Jacob...

    2022 arXiv

  39. [47]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020). Scaling laws for neural language models.arXiv preprint arXiv:2001.08361

    2020 arXiv

  40. [48]

    and Gal, Y

    Kendall, A. and Gal, Y. (2017). What uncertainties do we need in bayesian deep learning for computer vision? InAdvances in Neural Information Processing Systems 30, pages 5574–5584

    2017

  41. [49]

    and Chandorkar, A

    Kharbanda, A. and Chandorkar, A. (2024). Divergent ensemble networks: Enhancing uncertainty estimation with shared representations and independent branching.arXiv preprint arXiv:2412.01193

    2024

  42. [50]

    A., and Gal, Y

    Kossen, J., Han, J., Razzak, M., Schut, L., Malik, S. A., and Gal, Y. (2024). Semantic entropy probes: Robust and cheap hallucination detection in LLMs.arXiv preprint arXiv:2406.15927

    2024 arXiv

  43. [51]

    R., Raskar, R., and Beam, A

    Kumar, B., Lu, C., Gupta, G., Palepu, A., Bellamy, D. R., Raskar, R., and Beam, A. (2023). Conformal prediction with large language models for multi-choice question answering.arXiv preprint arXiv:2305.18404

    2023

  44. [52]

    Lakshminarayanan, B., Pritzel, A., and Blundell, C. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. InAdvances in Neural Information Processing Systems 30, pages 6402–6413

    2017

  45. [53]

    Laurent, O., Lafage, A., Tartaglione, E., Daniel, G., Martinez, J., Bursuc, A., and Franchi, G. (2023). Packed ensembles for efficient uncertainty estimation. InProceedings of the 11th International Conference on Learning Representations

    2023

  46. [54]

    Lee, K., Lee, K., Lee, H., and Shin, J. (2018). A simple unified framework for detecting out-of- distribution samples and adversarial attacks. InAdvances in Neural Information Processing Systems 31, pages 7167–7177

    2018

  47. [55]

    Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., and Kiela, D. (2020). Retrieval-augmented generation for 44 knowledge-intensive NLP tasks. InAdvances in Neural Information Processing Systems...

    2020

  48. [56]

    Li, Y., Qiang, R., Moukheiber, L., and Zhang, C. (2025). Language model uncertainty quantification with attention chain.arXiv preprint arXiv:2503.19168

    2025

  49. [57]

    Liang, S., Lu, X., Liu, Z., Wang, M., Lyu, Y., and Zhang, S. (2026). On the impact of weight quantization on deep neural network uncertainty. InProceedings of the 40th AAAI Conference on Artificial Intelligence, pages 23425–23432

    2026

  50. [58]

    Lin, S., Hilton, J., and Evans, O. (2022). Teaching models to express their uncertainty in words. Transactions on Machine Learning Research

    2022

  51. [59]

    Lin, Z., Trivedi, S., and Sun, J. (2024a). Contextualized sequence likelihood: Enhanced confidence scores for natural language generation. InProceedings of the 29th Conference on Empirical Methods in Natural Language Processing, pages 10351–10368

  52. [60]

    Lin, Z., Trivedi, S., and Sun, J. (2024b). Generating with confidence: Uncertainty quantification for black-box large language models.Transactions on Machine Learning Research

  53. [61]

    Ling, C., Zhao, X., Zhang, X., Cheng, W., Liu, Y., Sun, Y., Oishi, M., Osaki, T., Matsuda, K., Ji, J., Bai, G., Zhao, L., and Chen, H. (2024). Uncertainty quantification for in-context learning of large language models. InProceedings of the 20th Conference of the North America...

    2024

  54. [62]

    Liu, L., Pan, Y., Li, X., and Chen, G. (2024a). Uncertainty estimation and quantification for LLMs: A simple supervised approach.arXiv preprint arXiv:2404.15993

  55. [63]

    F., Chao, L

    Liu, S., Li, Z., Liu, X., Zhan, R., Wong, D. F., Chao, L. S., and Zhang, M. (2024b). Can LLMs learn uncertainty on their own? expressing uncertainty effectively in a self-training manner. In Proceedings of the 29th Conference on Empirical Methods in Natural Language Processing...

  56. [64]

    Liu, X., Chen, T., Da, L., Chen, C., Lin, Z., and Wei, H. (2025). Uncertainty quantification and confidence calibration in large language models: A survey. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, pages 6107–6117

    2025

  57. [65]

    Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach.arXiv preprint arXiv:1907.11692

    2019 arXiv

  58. [66]

    The Llama 3 herd of models.arXiv preprint arXiv:2407.21783

    Llama (2024). The Llama 3 herd of models.arXiv preprint arXiv:2407.21783. 45

    2024 arXiv

  59. [67]

    T., Yamada, Y., Hu, S., Foerster, J., Ha, D., and Clune, J

    Lu, C., Lu, C., Lange, R. T., Yamada, Y., Hu, S., Foerster, J., Ha, D., and Clune, J. (2026). Towards end-to-end automation of AI research.Nature, 651(8107):914–919

    2026

  60. [68]

    J., Izmailov, P., Garipov, T., Vetrov, D

    Maddox, W. J., Izmailov, P., Garipov, T., Vetrov, D. P., and Wilson, A. G. (2019). A simple baseline for bayesian uncertainty in deep learning. InAdvances in Neural Information Processing Systems 32, pages 13132–13143

    2019

  61. [69]

    and Gales, M

    Malinin, A. and Gales, M. J. F. (2021). Uncertainty estimation in autoregressive structured prediction. InProceedings of the 9th International Conference on Learning Representations

    2021

  62. [70]

    Manakul, P., Liusie, A., and Gales, M. J. F. (2023). SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models. InProceedings of the 28th Conference on Empirical Methods in Natural Language Processing, pages 9004–9017

    2023

  63. [71]

    Michelmore, R., Kwiatkowska, M., and Gal, Y. (2018). Evaluating uncertainty quantification in end-to-end autonomous driving control.arXiv preprint arXiv:1811.06817

    2018 arXiv

  64. [72]

    Min, S., Michael, J., Hajishirzi, H., and Zettlemoyer, L. (2020). AmbigQA: Answering ambiguous open-domain questions. InProceedings of the 25th Conference on Empirical Methods in Natural Language Processing, pages 5783–5797

    2020

  65. [73]

    and Xin, M

    Mo, S. and Xin, M. (2024). Tree of uncertain thoughts reasoning for large language models. In Proceedings of the 51st International Conference on Acoustics, Speech, and Signal Processing, pages 12742–12746

    2024

  66. [74]

    Nadeem, M. S. A., Zucker, J.-D., and Hanczar, B. (2009). Accuracy-rejection curves (ARCs) for comparing classification methods with a reject option. InProceedings of the 3rd International Workshop on Machine Learning in Systems Biology, pages 65–81

    2009

  67. [75]

    Nemani, V., Biggio, L., Huan, X., Hu, Z., Fink, O., Tran, A., Wang, Y., Zhang, X., and Hu, C. (2023). Uncertainty quantification in machine learning for engineering design and health prognostics: A tutorial.Mechanical Systems and Signal Processing, 205:110796

    2023

  68. [76]

    Nikitin, A., Kossen, J., Gal, Y., and Marttinen, P. (2024). Kernel language entropy: Fine-grained uncertainty quantification for LLMs from semantic similarities. InAdvances in Neural Information Processing Systems 37, pages 8901–8929

    2024

  69. [77]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774

    OpenAI (2023). Gpt-4 technical report.arXiv preprint arXiv:2303.08774

    2023 arXiv

  70. [78]

    Piray, P. (2026). Not all uncertainty is alike: volatility, stochasticity, and exploration.arXiv preprint arXiv:2605.19215. 46

    2026 arXiv

  71. [79]

    Semanticdensity: Uncertaintyquantificationforlargelanguage models through confidence measurement in semantic space

    Qiu, X.andMiikkulainen, R.(2024). Semanticdensity: Uncertaintyquantificationforlargelanguage models through confidence measurement in semantic space. InAdvances in Neural Information Processing Systems 37, pages 134507–134533

    2024

  72. [80]

    H., Jaakkola, T

    Quach, V., Fisch, A., Schuster, T., Yala, A., Sohn, J. H., Jaakkola, T. S., and Barzilay, R. (2024). Conformal language modeling. InProceedings of the 12th International Conference on Learning Representations

    2024

  73. [81]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388

    Qwen (2025). Qwen3 technical report.arXiv preprint arXiv:2505.09388

    2025 arXiv

  74. [82]

    Ren, J., Luo, J., Zhao, Y., Krishna, K., Saleh, M., Lakshminarayanan, B., and Liu, P. J. (2023). Out-of-distribution detection and selective generation for conditional language models. InProceedings of the 11th International Conference on Learning Representations

    2023

  75. [83]

    Ritter, H., Botev, A., and Barber, D. (2018). A scalable laplace approximation for neural networks. InProceedings of the 6th International Conference on Learning Representations

    2018

  76. [84]

    Sclar, M., Choi, Y., Tsvetkov, Y., and Suhr, A. (2024). Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting. In Proceedings of the 12th International Conference on Learning Representations

    2024

  77. [85]

    S., Taglicht, N., Shabtay, T., Ambus, A., Alon, N., Oron, S., Gordon-Tapiero, A., Kaplan, Y., Shwartz, V., Shaham, T

    Shapira, N., Wendler, C., Yen, A., Sarti, G., Pal, K., Floody, O., Belfki, A., Loftus, A., Jannali, A.R., Prakash, N., Cui, J., Rogers, G., Brinkmann, J., Rager, C., Zur, A., Ripa, M., Sankaranarayanan, A., Atkinson, D., Gandikota, R., Fiotto-Kaufman, J., Hwang, E., Orgad, H.,...

    2026 arXiv

  78. [86]

    Shi, H., Wang, Y., Han, L., Zhang, H., and Wang, H. (2025). Training-free bayesianization for low-rank adapters of large language models. InAdvances in Neural Information Processing Systems 38, pages 41663–41700

    2025

  79. [87]

    Shin, M., Cho, H., Min, H., and Lim, S. (2021). Neural bootstrapper. InAdvances in Neural Information Processing Systems 34, pages 16596–16609

    2021

  80. [88]

    Z., and Majumdar, A

    Shorinwa, O., Mei, Z., Lidard, J., Ren, A. Z., and Majumdar, A. (2026). A survey on uncertainty quantification of large language models: Taxonomy, open research challenges, and future directions. ACM Computing Surveys, 58(3):63:1–63:38

    2026

  81. [89]

    S., Saha, S., Kattakinda, P., and Feizi, S

    Sriramanan, G., Bharti, S., Sadasivan, V. S., Saha, S., Kattakinda, P., and Feizi, S. (2024). LLM-Check: Investigating detection of hallucinations in large language models. InAdvances in Neural Information Processing Systems 37, pages 34188–34216. 47

    2024

  82. [90]

    Stengel-Eskin, E., Hase, P., and Bansal, M. (2024). LACIE: Listener-aware finetuning for confidence calibration in large language models.arXiv preprint arXiv:2405.21028

    2024

  83. [91]

    Su, J., Luo, J., Wang, H., and Cheng, L. (2024). API is enough: Conformal prediction for large language models without logit-access. InProceedings of the 29th Conference on Empirical Methods in Natural Language Processing, pages 979–995

    2024

  84. [92]

    Sun, S., Zhang, G., Shi, J., and Grosse, R. B. (2019). Functional variational bayesian neural networks. InProceedings of the 7th International Conference on Learning Representations

    2019

  85. [93]

    and Lopez-Paz, D

    Tagasovska, N. and Lopez-Paz, D. (2019). Single-model uncertainties for deep learning. InAdvances in Neural Information Processing Systems 32, pages 6414–6425

    2019

  86. [94]

    Tian, K., Mitchell, E., Zhou, A., Sharma, A., Rafailov, R., Yao, H., Finn, C., and Manning, C. D. (2023). Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. InProceedings of the 28th Conference o...

    2023

  87. [95]

    Tonolini, F., Aletras, N., Massiah, J., and Kazai, G. (2024). Bayesian prompt ensembles: Model uncertainty estimation for black-box large language models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 12229–12272

    2024

  88. [96]

    Valiant, L. G. (1984). A theory of the learnable.Communications of the ACM, 27(11):1134–1142

    1984

  89. [97]

    van der Poel, L., Cotterell, R., and Meister, C. (2022). Mutual information alleviates hallucinations in abstractive summarization. InProceedings of the 27th Conference on Empirical Methods in Natural Language Processing, pages 5956–5965

    2022

  90. [98]

    B., Grishchenkov, K., Panchenko, A., Baldwin, T., Nakov, P., Panov, M., and Shelmanov, A

    Vashurin, R., Fadeeva, E., Vazhentsev, A., Rvanova, L., Vasilev, D., Tsvigun, A., Petrakov, S., Xing, R., Sadallah, A. B., Grishchenkov, K., Panchenko, A., Baldwin, T., Nakov, P., Panov, M., and Shelmanov, A. (2025a). Benchmarking uncertainty quantification methods for large l...

  91. [99]

    Vashurin, R., Goloburda, M., Ilina, A., Rubashevskii, A., Nakov, P., Shelmanov, A., and Panov, M. (2025b). CoCoA: A minimum bayes risk framework bridging confidence and consistency for uncertainty quantification in LLMs. InAdvances in Neural Information Processing Systems 38, ...

  92. [100]

    Vazhentsev, A., Kuzmin, G., Tsvigun, A., Panchenko, A., Panov, M., Burtsev, M., and Shelmanov, A. (2023). Hybrid uncertainty quantification for selective text classification in ambiguous tasks. In Proceedings of the 61st Annual Meeting of the Association for Computational Ling...

    2023

  93. [101]

    Vazhentsev, A., Rvanova, L., Kuzmin, G., Fadeeva, E., Lazichny, I., Panchenko, A., Panov, M., Baldwin, T., Sachan, M., Nakov, P., and Shelmanov, A. (2025a). Uncertainty-aware attention heads: Efficient unsupervised uncertainty quantification for LLMs.arXiv preprint arXiv:2505.20045

    arXiv

  94. [102]

    Vazhentsev, A., Rvanova, L., Lazichny, I., Panchenko, A., Panov, M., Baldwin, T., and Shelmanov, A. (2025b). Token-level density-based uncertainty quantification methods for eliciting truthfulness of large language models. InProceedings of the 21st Conference of the North Amer...

  95. [103]

    (2005).Algorithmic learning in a random world

    Vovk, V., Gammerman, A., and Shafer, G. (2005).Algorithmic learning in a random world. Springer

    2005

  96. [104]

    K., Shariatmadar, K., Moens, D., and Hallez, H

    Wang, K., Cuzzolin, F., Manchingal, S. K., Shariatmadar, K., Moens, D., and Hallez, H. (2024a). Credal deep ensembles for uncertainty quantification. InAdvances in Neural Information Processing Systems 37, pages 79540–79572

  97. [105]

    Wang, K., Cuzzolin, F., Moens, D., and Hallez, H. (2026). Credal ensemble distillation for uncertainty quantification. InProceedings of the 40th AAAI Conference on Artificial Intelligence, pages 26319–26327

    2026

  98. [106]

    Wang, X., Aitchison, L., and Rudolph, M. (2023). LoRA ensembles for large language model fine-tuning.arXiv preprint arXiv:2310.00035

    2023

  99. [107]

    N., and Wang, H

    Wang, Y., Shi, H., Han, L., Metaxas, D. N., and Wang, H. (2024b). BLoB: Bayesian low-rank adaptation by backpropagation for large language models. InAdvances in Neural Information Processing Systems 37, pages 67758–67794

  100. [108]

    T., and Zhu, X

    Wang, Z., Duan, J., Cheng, L., Zhang, Y., Wang, Q., Shi, X., Xu, K., Shen, H. T., and Zhu, X. (2024c). ConU: Conformal uncertainty in large language models with correctness coverage guarantees. InProceedings of the 29th Conference on Empirical Methods in Natural Language Proce...

  101. [109]

    H., Le, Q

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E. H., Le, Q. V., and Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. InAdvances in Neural Information Processing Systems 35, pages 24824–24837

    2022

  102. [110]

    Williams, A., Nangia, N., and Bowman, S. R. (2018). A broad-coverage challenge corpus for sentence understanding through inference. InProceedings of the 16th Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,...

    2018

  103. [111]

    E., Hernández-Lobato, J

    Wu, A., Nowozin, S., Meeds, E., Turner, R. E., Hernández-Lobato, J. M., and Gaunt, A. L. (2019). Deterministic variational inference for robust bayesian neural networks. InProceedings of the 7th International Conference on Learning Representations

    2019

  104. [112]

    and Wang, W

    Xiao, Y. and Wang, W. Y. (2021). On hallucination and predictive uncertainty in conditional language generation. InProceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2734–2744

    2021

  105. [113]

    Xu, T., Wu, S., Diao, S., Liu, X., Wang, X., Chen, Y., and Gao, J. (2024). SaySelf: Teaching LLMs to express confidence with self-reflective rationales. InProceedings of the 29th Conference on Empirical Methods in Natural Language Processing, pages 5985–5998

    2024

  106. [114]

    N., Bakman, Y

    Yaldiz, D. N., Bakman, Y. F., Buyukates, B., Tao, C., Ramakrishna, A., Dimitriadis, D., Zhao, J., and Avestimehr, S. (2025). Do not design, learn: A trainable scoring function for uncertainty estimation in generative LLMs. InProceedings of the 21st Conference of the North Amer...

    2025

  107. [115]

    X., Robeyns, M., Wang, X., and Aitchison, L

    Yang, A. X., Robeyns, M., Wang, X., and Aitchison, L. (2024a). Bayesian low-rank adapta- tion for large language models. InProceedings of the 12th International Conference on Learning Representations

  108. [116]

    E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., and Press, O

    Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., and Press, O. (2024b). SWE-agent: Agent-computer interfaces enable automated software engineering. InAdvances in Neural Information Processing Systems 38, pages 50528–50652

  109. [117]

    Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., and Narasimhan, K. (2023). Tree of thoughts: Deliberate problem solving with large language models.Advances in Neural Information Processing Systems 36, pages 11809–11822

    2023

  110. [118]

    F., Yilmaz, E., Shi, S., and Tu, Z

    Ye, F., Yang, M., Pang, J., Wang, L., Wong, D. F., Yilmaz, E., Shi, S., and Tu, Z. (2024). Benchmarking LLMs via uncertainty quantification.Advances in Neural Information Processing Systems 37, pages 15356–15385

    2024

  111. [119]

    C., Kaddar, Y., Blunsom, P., Staton, S., and Gal, Y

    Ye, Z., Melo, L. C., Kaddar, Y., Blunsom, P., Staton, S., and Gal, Y. (2025). Uncertainty-aware step-wise verification with generative reward models.arXiv preprint arXiv:2502.11250

    2025

  112. [120]

    Yu, X., Xu, C., Zhang, G., He, Y., Chen, Z., Xue, Z., Zhang, J., Liao, Y., Hu, X., Jiang, Y.-G., and Yan, S. (2026). Visual Multi-Agent system: Mitigating hallucination snowballing via visual flow. InProceedings of the 14th International Conference on Learning Representations

    2026

  113. [121]

    C., Hutter, F., and Teh, Y

    Zaidi, S., Zela, A., Elsken, T., Holmes, C. C., Hutter, F., and Teh, Y. W. (2021). Neural ensemble search for uncertainty estimation and dataset shift. InAdvances in Neural Information Processing Systems 34, pages 7898–7911. 50

    2021

  114. [122]

    and Zhang, R

    Zhang, B. and Zhang, R. (2025). CoT-UQ: Improving response-wise uncertainty quantification in LLMs with chain-of-thought. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, pages 26114–26133

    2025

  115. [123]

    Zhang, C., Liu, F., Basaldella, M., and Collier, N. (2024). LUQ: Long-text uncertainty quantifica- tion for LLMs. InProceedings of the 29th Conference on Empirical Methods in Natural Language Processing, pages 5244–5262

    2024

  116. [124]

    A., and Sricharan, K

    Zhang, J., Li, Z., Das, K., Malin, B. A., and Sricharan, K. (2023a). SAC3: Reliable hallucination detection in black-box language models via semantic-aware cross-check consistency. InProceedings of the 28th Conference on Empirical Methods in Natural Language Processing, pages ...

  117. [125]

    and Wang, D.-B

    Zhang, M.-L. and Wang, D.-B. (2026). Uncertainty calibration in deep learning: Methods, emerging challenges, and LLM frontiers.Journal of Computer Science and Technology, pages 1–23

    2026

  118. [126]

    Zhang, T., Qiu, L., Guo, Q., Deng, C., Zhang, Y., Zhang, Z., Zhou, C., Wang, X., and Fu, L. (2023b). Enhancing uncertainty-based hallucination detection with stronger focus. InProceedings of the 28th Conference on Empirical Methods in Natural Language Processing, pages 915–932

  119. [127]

    Zhao, Y., Yan, L., Sun, W., Xing, G., Meng, C., Wang, S., Cheng, Z., Ren, Z., and Yin, D. (2024). Knowing what LLMs DO NOT know: A simple yet effective self-detection method. InProceedings of the 20th Conference of the North American Chapter of the Association for Computationa...

    2024

  120. [128]

    Zhou, Z.-H. (2021). Why over-parameterization of deep neural networks does not overfit?Science China. Information Sciences, 64(1):116101

    2021

  121. [129]

    (2025).Ensemble Methods: Foundations and Algorithms

    Zhou, Z.-H. (2025).Ensemble Methods: Foundations and Algorithms. Chapman and Hall/CRC. 51

    2025

Pith tools

Reviewed June 26, 2026 · model on record in the stance chip above.