Pith. sign in

REVIEW 3 major objections 6 minor 53 references

Towards Harmonized Uncertainty Estimation for Large Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A lightweight Corrector that learns which questions a target LLM answers correctly can be blended into any existing uncertainty score, and the paper reports consistent gains in ranking, balance, and calibration.

desk verdict Useful empirical framing, but the headline UE gains are not established because the judge and the weight selection are part of the evaluation loop. read the letter →

arxiv 2505.19073 v2 pith:ZKY3Y6KV submitted 2025-05-25 cs.CL

Rui Li , Jing Long , Muge Qi , Heming Xia , Lei Sha , Peiyi Wang , Zhifang Sui This is my paper
classification cs.CL
keywords uncertaintyestimationlargelanguagemodelscalibrationAUROCexpectederrorcorrectionclassifierquestionansweringtrustworthyAI
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LLM uncertainty estimators tend to be lopsided: logit-based scores can rank errors but calibrate poorly, verbal confidence is overconfident, and consistency-based scores do not dominate on every metric. The paper claims this trade-off is unnecessary. CUE trains a small classifier on questions labeled by whether the target LLM answered them correctly, then combines the classifier's probability with any existing uncertainty score through a weighted sum. On TriviaQA and SciQA, with OPT-6.7B and LLaMA-3-8B-Instruct, the corrected scores improve AUROC by up to 0.27, raise average F1 by about 39%, and reduce expected calibration error by up to 0.34. If the claim holds, a user could attach one small trained model to any existing uncertainty estimator and get better trust signals without changing the LLM.

What carries the argument

The load-bearing object is the Corrector: a lightweight encoder (RoBERTa or DeBERTa) with a sigmoid classification head on the [CLS] representation, trained with binary cross-entropy on a dataset $\mathcal{D}^{*}_{\mathrm{cor}}=\{(q_i,1-c_i)\}$, where $c_i=1$ means the target model's answer to question $q_i$ was judged correct. The labels come from a hybrid judge that accepts an answer if a ROUGE-L score against the gold answer exceeds 0.7 or if an instruction-following language model says the two are equivalent. The Corrector's output $C(x)$ is a question-level estimate of unreliability, and its convex combination with $U_{\mathrm{norm}}(x)$ is what pulls the raw score toward the model's actual per-question accuracy. The same Corrector is trained once per dataset--model pair and then applied to every baseline, which is why the paper describes it as orthogonal to existing uncertainty estimators.

What would settle it

Randomly sample a few hundred target-model responses from TriviaQA and SciQA, have human annotators label correctness, and recompute AUROC, F1, and ECE for vanilla and corrected scores against the human labels. If the corrected scores do not consistently beat the vanilla baselines on human labels, the reported improvements are an artifact of the automatic judge.

Watch

Extended reading notes

Core claim

The paper's central claim is that uncertainty estimation improves when a score derived from the target model's own generation—entropy, semantic entropy, verbal confidence, or lexical consistency—is corrected by a supervised estimate of that model's reliability on the question. Concretely, the corrected score is $$U_{\mathrm{cor}}(x)=$w^{{*}}$U_{\mathrm{norm}}(x)+(1-$w^{{*}}$)C(x),$$ where $U_{\mathrm{norm}}(x)$ is a min--max normalized uncertainty score and $C(x)$ is the Corrector's predicted probability that the target model will answer $x$ incorrectly. The weight $w^{*}$ is grid-searched on a development set. Across two open models and two QA benchmarks, the corrected score beats each vanilla baseline on AUROC, F1, and ECE, with average AUROC gains of 0.27 on TriviaQA and 0.09 on SciQA, average F1 gains near 39%, and ECE reductions as large as 0.34. The paper reads this as evidence that indication, precision--recall balance, and calibration can be achieved together rather than traded off.

Load-bearing premise

The load-bearing premise is that the hybrid rule-plus-LLM judge that labels each target-model answer as correct or incorrect is itself accurate; if the judge is biased, the Corrector learns those biases and the reported gains show agreement with the judge rather than genuine calibration.

Editorial extensions

If this is right

  • Every family of uncertainty scores—logit-based, verbalized, consistency-based, internal-state—can be upgraded by the same Corrector, so the marginal value of designing a new base score shrinks.
  • Calibration improves even though the Corrector's training objective is binary classification, not ECE; reported ECE reductions range up to 0.34 depending on model and dataset.
  • One Corrector trained for a dataset--model pair serves all baselines on that pair, and cross-domain experiments show absolute AUROC gains of 7–11% when training and testing on different but related medical datasets.
  • Because the Corrector never inspects the target model's hidden states or token probabilities, the method is claimed to be applicable to black-box models even though the experiments use white-box models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The Corrector may be learning question answerability more than model confidence, and the paper's own cross-model results—strong transfer within the OPT family, weak transfer to LLaMA—are consistent with that reading.
  • The automatic judge's OR rule and ROUGE-L threshold could systematically count near-miss or paraphrase answers as correct, so re-evaluating the same corrections against human labels would show whether the gains reflect true correctness or agreement with the judge.
  • Part of the ECE reduction may be arithmetic rather than learned, since averaging any score with a well-calibrated binary probability compresses the confidence distribution; the paper does not isolate this contribution.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper claims that existing LLM uncertainty estimation methods each excel in only one of indication, precision-recall balance, or calibration, and that simple weighted combinations of their scores do not help. It proposes CUE, a framework that trains a lightweight RoBERTa/DeBERTa classifier on question–correctness pairs to produce a correction score C(x), and combines it with a min–max normalized uncertainty score U_norm(x) through U_cor(x) = w* U_norm(x) + (1 - w*) C(x). The weight w* is chosen by grid search, and the authors report consistent AUROC, F1, and ECE improvements across TriviaQA and SciQA for OPT-6.7B and LLaMA-3-8B-Instruct, with F1 gains averaging 38.97% and ECE reductions up to 0.34. The central mechanism is that correctness labels are produced by a hybrid rule-based (ROUGE-L > 0.7) and GPT-3.5-turbo judge, and the same label source is used for both Corrector training and the evaluation metrics.

Significance. If the results held under a fully independent evaluation, the paper would make a useful practical contribution: a model-agnostic post-hoc correction layer that improves multiple uncertainty estimators and calibration could be deployed with modest cost, and the breadth of baselines and two target models is appropriate. The paper also provides an honest conceptual framing (indication/balance/calibration) and makes code available. However, the two main experimental decisions—unvalidated judge labels and tuning w* on half of the test set—directly affect the central claim, so the contribution is conditional on a revised evaluation protocol rather than established by the current experiments.

major comments (3)
  1. [Sec. 4.1, Eq. (1)] The correctness labels c_i are generated entirely by the hybrid judge M(r_i, a_i) = M_Rule(r_i, a_i) OR M_LLM(r_i, a_i), and no alternative labeling process is described for the test split used in Sec. 5.2. Since the same labels are used to train the Corrector and, by the paper's description, to compute AUROC/F1/ECE, the reported gains (e.g., LS AUROC 19.57 to 69.82 on TriviaQA/LLaMA-3 in Table 1) demonstrate improved agreement with M rather than with the actual correctness of the model's responses. The paper reports no human validation, no agreement statistics, and no error analysis for M, and the OR logic is intentionally liberal in labeling positives; if M is biased, the Corrector learns that bias and the evaluation inherits it. This is load-bearing because the conclusion states that CUE aligns uncertainty scores with 'the actual performance of the model.'
  2. [Sec. 5.1.5 and Eq. (2)] The optimal weight w* is selected by grid search on half of the test set, while the other half is used to report results. This means Eq. (2)'s corrected score contains a hyperparameter fitted to the same distribution and the same label source as the evaluation. The reported improvements in Table 1 therefore partly measure the benefit of fitting w* to the test distribution, not an out-of-sample property of the method. A proper three-way split (train/dev/test) or nested cross-validation with the dev set used only for w* selection is needed before the headline improvements can be interpreted.
  3. [Table 1 and Appendix A.5] No error bars, multiple seeds, or repeated evaluation runs are reported for AUROC/F1/ECE, and the t-test description in Appendix A.5 omits the number of samples, whether the tests are paired, and the variance structure. For strong baselines such as SE and SAR where the reported AUROC gains are only 0.3 to 1.2 points (Table 1), the claim of consistent and statistically significant improvement is not yet supported. The sensitivity analysis in A.6.1 reports stable ranges for w*, but does not compensate for the absence of uncertainty estimates on the main metrics.
minor comments (6)
  1. [Sec. 5.1.4] The word 'donated' should be 'denoted' in the description of the SAR variants.
  2. [Sec. 4.2] The notation in the binary cross-entropy loss is inconsistent: y_i is introduced as the true label while \\hat{y}_i is the prediction, but the sentence says \\hat{y}_i is used to compute the likelihood that a data point belongs to label c1; please clarify the roles of y_i and \\hat{y}_i. Also, since D*_cor uses labels 1 - c_i, state explicitly that the positive class in Corrector training corresponds to 'incorrect'.
  3. [Abstract] The abstract claims improvements 'of up to 60%' but no table directly defines the denominator for this percentage; Table 1 reports percentage-point differences, so clarify the basis for the 60% figure.
  4. [Figure 7] The numeric labels on the bars are not defined (are they absolute AUROC gains in percentage points?), and the ordering of base models on the x-axis is unclear; adding error bars would help interpret the comparison across base models.
  5. [Appendix A.6.2] This section contains reviewer-response language ('addressing R1', 'mentioned by the reviewer') that should not appear in a published manuscript; please remove or rewrite it in the paper's own voice.
  6. [Table 5] The table heading contains a typo ('Correcter'), and the table mixes pure Corrector AUROC values with combined results; label the rows and columns more explicitly so that 'pure Corrector' and 'w/ Corrector' rows are unambiguous.

Circularity Check

1 steps flagged · score 6.0 of 10

The central 'alignment with actual performance' claim reduces to agreement with an unvalidated hybrid judge used to generate both training and evaluation labels.

  1. self definitional [Section 4.1, Eq. (1); Section 5.2; Conclusion]
    "After that, a binary label ci is assigned to each sample, defined as ci = M(ri, ai)(1) ... which serves as a representation of the target model's performance in generating correct responses across a particular knowledge domain. ... With the Corrector, we can improve the reliability of uncertainty scores and alignment with the actual performance of the model."

    The only correctness-labeling procedure described is the hybrid judge M, and the paper reports no human validation of M. The Corrector is trained on D*_cor whose labels are 1-c_i with c_i = M(r_i, a_i), and the AUROC/F1/ECE metrics in Table 1 are computed against correctness labels produced by the same data-crafting pipeline. Therefore the claimed alignment with 'actual performance' is by construction alignment with M: the trained corrector is a fitted predictor of M's labels, and the evaluation measures agreement with that same label source. If M is biased (e.g., the OR rule accepts plausible but wrong answers), the reported gains reflect reproducing that bias, not independent evidence of true correctness.

full rationale

The paper is not purely circular in the sense of deriving Eq. 2 from Eq. 1: it does train on a separate split, reports held-out results, and the weighted combination U_cor = w* U_norm + (1-w*) C(x) is a legitimate supervised approach if the label source is valid. However, the target construct ('actual performance') is operationalized entirely by the hybrid judge M in Eq. 1. Since the Corrector is trained to predict M's labels and all reported metrics use M-derived labels, the headline improvements are relative to the judge, not to an externally validated ground truth. The Limitations section concedes dependence on labeled data but does not validate the labels, and the t-tests in Appendix A.5 compare methods on the same M-derived labels. This makes the central claim of alignment with actual model performance partially self-definitional: the evaluation cannot distinguish genuine calibration from faithful reproduction of the judge's biases. The grid search for w* is standard hyperparameter selection and is not itself the main circular step; the entanglement of training and evaluation labels through the same unvalidated judge is the load-bearing issue.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

All free parameters and assumptions are tied to the correction framework. The main tunable is the blend weight w*, selected on the development half of the test set. The labeling rule and its threshold are manual choices. The correctness labels are not validated against human judgments, which is the key assumption.

free parameters (3)
  • w* (combination weight) = varies per method, dataset, model (e.g., stable ranges in Table 6)
    In Eq. (2), the corrected score is a weighted average of U_norm and C(x), with w* selected by grid search on the development half of the test set for each method/dataset/model combination (Sec 5.1.5, A.6).
  • ROUGE-L threshold 0.7 = 0.7
    In Sec 4.1, the rule-based labeler marks a response correct if ROUGE-L(ri, ai) > 0.7; this threshold is chosen without reported sensitivity analysis.
  • GPT-3.5-turbo judge prompt settings = unspecified
    The LLM judge is used in Sec 4.1 with 'strict prompting rules' but the exact prompt is not given; this is a manually chosen component affecting all labels.
assumptions (3)
  • domain assumption The hybrid judge M(ri, ai) = M_Rule OR M_LLM produces correct binary correctness labels for target model responses.
    Invoked in Sec 4.1 to create the correction dataset and implicitly in evaluation; if the judge is biased, the Corrector learns those biases.
  • domain assumption The development set (half of the test set) is representative enough for selecting w* that generalizes to the evaluation half.
    Sec 5.1.5 splits the test set into a dev half for w* search and an eval half; this assumes no overfitting to the test distribution.
  • domain assumption The target LLM's correctness on training questions is a stable predictor of its correctness on test questions in the same domain.
    The Corrector is trained on a curated dataset and applied to test questions (Secs 4.1 and 4.2); this is a standard supervised learning assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Harmonized Uncertainty Estimation for Large Language Models." pith.science (2026). https://pith.science/paper/ZKY3Y6KV

@misc{pith2026250519073,
  author       = {Pith},
  title        = {Pith review of: Towards Harmonized Uncertainty Estimation for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZKY3Y6KV}},
  note         = {Machine review of arXiv:2505.19073}
}
read the original abstract

To facilitate robust and trustworthy deployment of large language models (LLMs), it is essential to quantify the reliability of their generations through uncertainty estimation. While recent efforts have made significant advancements by leveraging the internal logic and linguistic features of LLMs to estimate uncertainty scores, our empirical analysis highlights the pitfalls of these methods to strike a harmonized estimation between indication, balance, and calibration, which hinders their broader capability for accurate uncertainty estimation. To address this challenge, we propose CUE (Corrector for Uncertainty Estimation): A straightforward yet effective method that employs a lightweight model trained on data aligned with the target LLM's performance to adjust uncertainty scores. Comprehensive experiments across diverse models and tasks demonstrate its effectiveness, which achieves consistent improvements of up to 60% over existing methods.

Figures

Figures reproduced from arXiv: 2505.19073 by the authors.

Figure 1
Figure 1. A concise overview figure of various uncer [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The performance of existing uncertainty estimation methods, evaluated on the SciQA dataset with the [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. AUROC improvement across uncertainty scores combination from different existing methods. sion of related work is provided in the Appendix A.1. 3 Preliminary Study 3.1 Limitation of Existing UE Methods We evaluate existing UE methods from both clas￾sification and calibration views, focusing on three key aspects of uncertainty scores: indication, bal￾ance, and calibration. From the classification view, uncertainty sco… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: An overview of uncertainty score correction framework. Firstly, we construct a dataset that closely aligns [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: The performance gains of using the Corrector to adjust the uncertainty scores for various methods on the datasets of TriviaQA and SciQA, and the target models of LLaMA-3-8B-Instruct and OPT-6.7B, are evaluated in terms of F1 score. facilitated by the Corrector, we prov…
Figure 6
Figure 6. Figure 6: Calibration Plots. These plots depict the relationship between predicted confidence and observed [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: The overall AUROC score gains achieved by Correctors trained on different base models across various UE methods on the SciQA dataset and Llama-3- 8B-Instruct target model. Acquisition We compare correction scores from a lightweight classifier with those estimated using…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 8 canonical work pages

  1. [1]

    Mari-Liis Allikivi, Joonas J \""a rve, and Meelis Kull. 2024. Cautious calibration in binary classification. arXiv preprint arXiv:2408.05120

    work page Pith review arXiv 2024

  2. [2]

    S \"o ren Auer, Dante A. C. Barone, Cassiano Bartz, Eduardo G. Cortes, Mohamad Yaser Jaradeh, Oliver Karras, Manolis Koubarakis, Dmitry Mouromtsev, Dmitrii Pliukhin, Daniil Radyush, Ivan Shilin, Markus Stocker, and Eleni Tsalapati. 2023. https://doi.org/10.1038/s41598-023-33607-z The sciqa scientific question answering benchmark for scholarly knowledge . ...

    doi:10.1038/s41598-023-33607-z 2023

  3. [3]

    Amos Azaria and Tom Mitchell. 2023. The internal state of an llm knows when it's lying. arXiv preprint arXiv:2304.13734

    arXiv 2023

  4. [4]

    Yavuz Faruk Bakman, Duygu Nur Yaldiz, Baturalp Buyukates, Chenyang Tao, Dimitrios Dimitriadis, and Salman Avestimehr. 2024. Mars: Meaning-aware response scoring for uncertainty estimation in generative llms. arXiv preprint arXiv:2402.11756

    arXiv 2024

  5. [5]

    Evan Becker and Stefano Soatto. 2024. Cycles of thought: Measuring llm confidence through stable explanations. arXiv preprint arXiv:2406.03441

    arXiv 2024

  6. [6]

    Arslan Chaudhry, Sridhar Thiagarajan, and Dilan Gorur. 2024. https://arxiv.org/abs/2409.12180 Finetuning language models to emit linguistic expressions of uncertainty . Preprint, arXiv:2409.12180

    arXiv 2024

  7. [7]

    Jinhao Duan, Hao Cheng, Shiqi Wang, Chenan Wang, Alex Zavalny, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. 2023. Shifting attention to relevance: Towards the uncertainty estimation of large language models. arXiv preprint arXiv:2307.01379

    arXiv 2023

  8. [8]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

Show all 53 references
  1. [9]

    Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Fr \'e d \'e ric Blain, Francisco Guzm \'a n, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. 2020. Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computation...

    2020

  2. [10]

    Karol Gregor, Ivo Danihelka, Andriy Mnih, Charles Blundell, and Daan Wierstra. 2014. Deep autoregressive networks. In International Conference on Machine Learning, pages 1242--1250. PMLR

    2014

  3. [11]

    Tobias Groot and Matias Valdenegro-Toro. 2024. Overconfidence is key: Verbalized uncertainty evaluation in large language and vision-language models. arXiv preprint arXiv:2405.02917

    2024 arXiv

  4. [12]

    Haixia Han, Tingyun Li, Shisong Chen, Jie Shi, Chengyu Du, Yanghua Xiao, Jiaqing Liang, and Xin Lin. 2024. Enhancing confidence expression in large language models through learning from past experience. arXiv preprint arXiv:2404.10315

    2024 arXiv

  5. [13]

    Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021 a . https://arxiv.org/abs/2111.09543 Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing . Preprint, arXiv:2111.09543

    2021 arXiv

  6. [14]

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021 b . https://openreview.net/forum?id=XPZIaotutsD Deberta: Decoding-enhanced bert with disentangled attention . In International Conference on Learning Representations

    2021

  7. [15]

    Ziwei Ji, Delong Chen, Etsuko Ishii, Samuel Cahyawijaya, Yejin Bang, Bryan Wilie, and Pascale Fung. 2024. Llm internal states reveal hallucination risk faced with a query. arXiv preprint arXiv:2407.03282

    2024 arXiv

  8. [16]

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2020. https://arxiv.org/abs/2009.13081 What disease does this patient have? a large-scale open domain question answering dataset from medical exams . Preprint, arXiv:2009.13081

    2020 arXiv

  9. [17]

    Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551

    2017 arXiv

  10. [18]

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221

    2022 arXiv

  11. [19]

    Amita Kamath, Robin Jia, and Percy Liang. 2020. Selective question answering under domain shift. arXiv preprint arXiv:2006.09462

    2020 arXiv

  12. [20]

    Sanyam Kapoor, Nate Gruver, Manley Roberts, Arka Pal, Samuel Dooley, Micah Goldblum, and Andrew Wilson. 2024. https://aclanthology.org/2024.uncertainlp-1.1/ Calibration-tuning: Teaching large language models to know what they don`t know . In Proceedings of the 1st Workshop on ...

    2024

  13. [21]

    Jannik Kossen, Jiatong Han, Muhammed Razzak, Lisa Schut, Shreshth Malik, and Yarin Gal. 2024. Semantic entropy probes: Robust and cheap hallucination detection in llms. arXiv preprint arXiv:2406.15927

    2024 arXiv

  14. [22]

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664

    2023 arXiv

  15. [23]

    Kenneth Li, Oam Patel, Fernanda Vi \'e gas, Hanspeter Pfister, and Martin Wattenberg. 2024 a . Inference-time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36

    2024

  16. [24]

    Moxin Li, Wenjie Wang, Fuli Feng, Fengbin Zhu, Qifan Wang, and Tat-Seng Chua. 2024 b . Think twice before assure: Confidence estimation for large language models through reflection on multiple answers. arXiv preprint arXiv:2403.09972

    2024 arXiv

  17. [25]

    Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2023. Generating with confidence: Uncertainty quantification for black-box large language models. arXiv preprint arXiv:2305.19187

    2023 arXiv

  18. [26]

    Linyu Liu, Yu Pan, Xiaocheng Li, and Guanting Chen. 2024. Uncertainty estimation and quantification for llms: A simple supervised approach. arXiv preprint arXiv:2404.15993

    2024 arXiv

  19. [27]

    Yinhan Liu. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692

    2019 arXiv

  20. [28]

    Antonio Loquercio, Mattia Segu, and Davide Scaramuzza. 2020. A general framework for uncertainty estimation in deep learning. IEEE Robotics and Automation Letters, 5(2):3153--3160

    2020

  21. [29]

    Nishanth Madhusudhan, Sathwik Tejaswi Madhusudhan, Vikas Yadav, and Masoud Hashemi. 2024. Do llms know when to not answer? investigating abstention abilities of large language models. arXiv preprint arXiv:2407.16221

    2024 arXiv

  22. [30]

    Andrey Malinin and Mark Gales. 2020. Uncertainty estimation in autoregressive structured prediction. arXiv preprint arXiv:2002.07650

    2020 arXiv

  23. [31]

    Shiyu Ni, Keping Bi, Lulu Yu, and Jiafeng Guo. 2024. Are large language models more honest in their probabilistic or verbalized confidence? arXiv preprint arXiv:2408.09773

    2024 arXiv

  24. [32]

    Alexander Nikitin, Jannik Kossen, Yarin Gal, and Pekka Marttinen. 2024. Kernel language entropy: Fine-grained uncertainty quantification for llms from semantic similarities. arXiv preprint arXiv:2405.20003

    2024 arXiv

  25. [33]

    OpenAI. 2023. https://doi.org/10.48550/ARXIV.2303.08774 GPT-4 technical report . CoRR, abs/2303.08774

    2023 doi arXiv

  26. [34]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leik...

    2022

  27. [35]

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, pages 248--260. PMLR

    2022

  28. [36]

    Christos E Papadopoulos and Hoi Yeung. 2001. Uncertainty estimation and monte carlo simulation method. Flow Measurement and Instrumentation, 12(4):291--298

    2001

  29. [37]

    Tejaswini Pedapati, Amit Dhurandhar, Soumya Ghosh, Soham Dan, and Prasanna Sattigeri. 2024. Large language model confidence estimation via black-box access. arXiv preprint arXiv:2406.04370

    2024 arXiv

  30. [38]

    Maohao Shen, Subhro Das, Kristjan Greenewald, Prasanna Sattigeri, Gregory Wornell, and Soumya Ghosh. 2024. https://arxiv.org/abs/2403.08819 Thermometer: Towards universal calibration for large language models . Preprint, arXiv:2403.08819

    2024 arXiv

  31. [39]

    Linwei Tao, Minjing Dong, and Chang Xu. 2023. Dual focal loss for calibration. In International Conference on Machine Learning, pages 33833--33849. PMLR

    2023

  32. [40]

    Shuchang Tao, Liuyi Yao, Hanxing Ding, Yuexiang Xie, Qi Cao, Fei Sun, Jinyang Gao, Huawei Shen, and Bolin Ding. 2024. When to trust llms: Aligning confidence with response quality. arXiv preprint arXiv:2404.17287

    2024 arXiv

  33. [41]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie - Anne Lachaux, Timoth \' e e Lacroix, Baptiste Rozi \` e re, Naman Goyal, Eric Hambro, Faisal Azhar, Aur \' e lien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023 a . https://doi.org/10....

    2023 doi arXiv

  34. [42]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, et al. 2023 b . https://doi.org/10.48550/ARXIV.2307.09288 Llama 2: Open foundation and fine-tuned chat models . CoRR, abs/2307.09288

    2023 doi arXiv

  35. [43]

    Dennis Ulmer, Martin Gubri, Hwaran Lee, Sangdoo Yun, and Seong Joon Oh. 2024. https://arxiv.org/abs/2403.05973 Calibrating large language models using their generations only . Preprint, arXiv:2403.05973

    2024 arXiv

  36. [44]

    Artem Vazhentsev, Gleb Kuzmin, Akim Tsvigun, Alexander Panchenko, Maxim Panov, Mikhail Burtsev, and Artem Shelmanov. 2023. Hybrid uncertainty quantification for selective text classification in ambiguous tasks. In Proceedings of the 61st Annual Meeting of the Association for C...

    2023

  37. [45]

    Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2024. Do large language models have a legal duty to tell the truth? Royal Society Open Science, 11(8):240197

    2024

  38. [46]

    Zhiyuan Wang, Jinhao Duan, Chenxi Yuan, Qingyu Chen, Tianlong Chen, Huaxiu Yao, Yue Zhang, Ren Wang, Kaidi Xu, and Xiaoshuang Shi. 2024. Word-sequence entropy: Towards uncertainty estimation in free-form medical question answering applications and beyond. arXiv preprint arXiv:...

    2024 arXiv

  39. [47]

    Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2023. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. arXiv preprint arXiv:2306.13063

    2023 arXiv

  40. [48]

    Duygu Nur Yaldiz, Yavuz Faruk Bakman, Baturalp Buyukates, Chenyang Tao, Anil Ramakrishna, Dimitrios Dimitriadis, and Salman Avestimehr. 2024. Do not design, learn: A trainable scoring function for uncertainty estimation in generative llms. arXiv preprint arXiv:2406.11278

    2024 arXiv

  41. [49]

    Adam Yang, Chen Chen, and Konstantinos Pitas. 2024. Just rephrase it! uncertainty estimation in closed-source language models via multiple rephrased queries. arXiv preprint arXiv:2405.13907

    2024 arXiv

  42. [50]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068

    2022 arXiv

  43. [51]

    Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. 2023. Siren's song in the ai ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.01219

    2023 arXiv

  44. [52]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  45. [53]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.