Pith. sign in

REVIEW 3 major objections 5 minor 74 references

Variability Need Not Imply Error: The Case of Adequate but Semantically Distinct Responses

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Judge language-model responses for adequacy, not semantic variation, to know when to trust them.

desk verdict A simple, well-tested idea—adequacy-based confidence (PROBAR) beats semantic entropy in open-ended settings—with one honest, fixable gap in the AmbigQA half. read the letter →

arxiv 2412.15683 v1 pith:H22SXQZ2 submitted 2024-12-20 cs.CL

Evgenia Ilia , Wilker Aziz This is my paper · ORCID
classification cs.CL
keywords selectivepredictionuncertaintyquantificationsemanticentropyadequacyopen-endedlanguagegenerationPROBARmodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a language model's uncertainty should be measured by how often it produces adequate responses, not by how varied its responses are. Semantic variation is a poor error signal in ambiguous or open-ended settings, where a model may give many different yet valid answers. The paper introduces PROBAR, an estimate of the probability a model assigns to responses judged adequate to the prompt, computed as the fraction of sampled responses a classifier labels adequate. In selective prediction experiments with OPT models across two QA datasets and next-word prediction, PROBAR reports higher AUROC than entropy, semantic entropy, and a P(True) variant, for both ambiguous and non-ambiguous prompts. If the finding holds, the practical lesson is that confidence should be read as the rate of adequate answers, not as semantic homogeneity.

What carries the argument

The central object is PROBAR, an MC estimate of the probability that the model, given a prompt, produces a response judged adequate: $\mathrm{PROBAR} = \frac{1}{N}\sum_{n=1}^{N} [a(x, y_n) = A_1]$, where $a$ is a judge mapping prompt-response pairs to adequate or inadequate and $y_1,\dots,y_N$ are sampled responses. The judge can be a human, a trained classifier, or a large language model; the paper implements it with an instruction-tuned language model for adequacy and uses a separate LLM judge for the correctness labels in QA. The measure works by interpreting the relative frequency of adequate samples as instance-level confidence, so that high semantic variability is not penalised when the model's varied responses are all adequate. It is this adequacy judge that carries the argument: PROBAR is only as informative as the classifier's ability to recognise plausible answers to open-ended prompts.

What would settle it

Recompute the QA AUROC comparisons with correctness labels produced by a second independent judge, such as two human annotators or an LLM from a different model family than the adequacy classifier, and check whether PROBAR still beats semantic entropy by the same margin; if the advantage collapses or reverses under the alternative judge, the central claim is not robust.

Watch

Extended reading notes

Core claim

The paper's claim is that an uncertainty score should quantify the probability that a sampled response is adequate to the prompt, rather than the variability of the response set. Semantic entropy treats dispersion over meaning clusters as evidence of error; PROBAR instead groups responses by adequacy and accumulates the model's probability mass over adequate responses. Using OPT models of four sizes and prompts of varying open-endedness, the paper reports that PROBAR achieves higher AUROC and better selective precision versus coverage than semantic entropy and other baselines, in both ambiguous and non-ambiguous settings. The authors further show that the advantage is not an artifact of automation: replacing PROBAR's adequacy classifier and semantic entropy's clustering with human judgements still leaves PROBAR ahead, and even the practical PROBAR implementation beats semantic entropy's human-labelled upper bound.

Load-bearing premise

The load-bearing premise is that the automated judge used to label responses as correct, and the human labels used for upper bounds, are accurate enough and independent of the errors made by PROBAR's adequacy classifier; if the same bias corrupts both the correctness labels and the adequacy labels, PROBAR's measured advantage over semantic entropy would be inflated.

Editorial extensions

If this is right

  • In selective prediction, choosing responses by PROBAR instead of semantic entropy should raise precision at fixed coverage for ambiguous and open-ended prompts.
  • PROBAR can flag confident errors, where a model commits strongly to a single inadequate response, a case that variation-based scores miss.
  • Because PROBAR relies only on a per-response adequacy judgement, the same recipe transfers across tasks, including next-word prediction over passage prefixes.
  • Fewer samples are sufficient for a usable PROBAR estimate, with the paper reporting that five samples behave similarly to ten, reducing the computational cost.
  • PROBAR can seed confidence-aware decoding: choosing an adequate sampled response instead of the greedy decoding improved precision in most settings tested.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One extension the authors leave implicit is making adequacy multi-dimensional, so a response counts only if it is plausible, grammatical, and non-toxic, which would carry the method toward safety-sensitive generation.
  • A testable extension is to recompute the QA comparisons with correctness labels from a judge drawn from a different model family than the adequacy classifier, isolating whether shared judge biases inflate PROBAR's margin.
  • Because PROBAR separates what the model knows (adequacy mass) from how it hedges (semantic spread), it points toward a two-dimensional reliability profile rather than a single uncertainty score.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that semantic variability among sampled LM responses is not necessarily evidence of error, especially for ambiguous or open-ended prompts. It defines PROBAR (Eq. 3) as a Monte Carlo estimate of the probability that sampled responses are judged adequate by a classifier or LLM judge, and evaluates PROBAR as a confidence measure for selective prediction with OPT models (2.7B-30B) on Abg-COQA, AmbigQA, and Provo next-word prediction. Compared with entropy, semantic entropy, and a P(True) variant, the paper reports that PROBAR mostly achieves higher AUROC and better selective precision-coverage trade-offs, supplemented by human annotation on Abg-COQA and Provo, upper-bound analyses on 50 manually labeled contexts, and ablations on sample size and bootstrap variance.

Significance. If the empirical claims hold, PROBAR is a conceptually useful alternative to consistency-based uncertainty: it formalizes confidence as the rate of adequate, rather than semantically homogeneous, responses, and it is simple to compute from samples and a judge. The study is thorough in coverage (three datasets, four OPT sizes, human annotation, classifier ablations, sample-size analysis, and bootstrap variance), and the core estimator in Eq. (3) is transparent and free of fitted parameters. The paper also releases code and data, which supports reproducibility. The main reservation is that one of the two QA legs (AmbigQA) rests on an adequacy classifier that is never validated against human labels, while the correctness judge may share a model-specific bias with that classifier; this makes the headline claim stronger than the current evidence supports.

major comments (3)
  1. [Section 5 (AmbigQA) and Limitations] The AmbigQA leg is load-bearing for the headline claim, but its adequacy classifier (Mistral-Small-Instruct-2409, Appendix A.4) is never validated against human labels; the Limitations section explicitly concedes that the KBQA classifier was 'not explicitly evaluated' and only 'implicitly' validated through improved AUROC. Because the correctness labels for AmbigQA are produced by gpt3.5-turbo asked to judge plausibility 'given your training data' (Appendix B.2) and the adequacy classifier is prompted to judge plausibility 'with respect to your training data' (Appendix A.4), a shared model-specific bias could inflate PROBAR's AUROC without any human ground truth to detect it. This needs human validation of the adequacy classifier on AmbigQA, or at least a sensitivity analysis with an independently sourced judge, before the cross-dataset claim is supported.
  2. [Section 5 (AmbigQA), Figure 4 right] The abstract and Section 1 state that PROBAR outperforms semantic entropy 'across prompts with varying degrees of ambiguity/open-endedness,' but the AmbigQA results show OPT-13B, where PROBAR does not outperform Norm.SE and the paper itself notes 'except for OPT13B.' The headline claim should either be qualified to exclude this exception or the exception should be explained; as written, the claim is broader than the reported evidence.
  3. [Figures 3 and 4] The main AUROC comparisons are presented as point estimates without confidence intervals or error bars. Appendix C.3 provides bootstrap distributions only for Abg-COQA (and an appendix figure for AmbigQA), not for the headline figures, and several reported advantages are small, for example in the Abg-COQA ambiguous test set in Figure 3b. Without intervals or paired significance tests, the reader cannot tell whether the reported advantage of PROBAR over Norm.SE is statistically distinguishable from noise.
minor comments (5)
  1. [Figure 3 caption] The dataset name is spelled inconsistently: 'AbgCOQA', 'Abc-COQA', and 'Abg-COQA' all appear within one caption; the rest of the paper should use a single spelling.
  2. [Appendix A.2] The text refers to 'Section 2.2' to explain the NLI-based clustering algorithm, but the section numbering in the paper does not include a Section 2.2; this cross-reference should be updated.
  3. [Section 7 (Related Work)] There are typos in the related work paragraph, including 'diffeent methods' and the garbled phrase 'rather than repeatedly sampling, responses’ some work perturbs inputs,' which should be rewritten.
  4. [Section 4 and Figure 4 caption] Section 4 says 100 contexts were sampled from Provo, while the Figure 4 caption says '200 prompts'; clarify that the count includes the 100 corrupted counterpart contexts.
  5. [Appendix C.2] The claim that 5 samples give 'reasonably similar performance' to 10 samples in Figure 17 is not accompanied by error bars or a statistical comparison; please state whether the difference is within sampling variability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PROBAR is an empirical frequency from a judge, evaluated against independently defined correctness.

full rationale

The paper's central quantity is defined by Eq. (3) as the MC estimate of Pr(a(x,Y)=A1|x), i.e. the relative frequency with which an external judge labels sampled responses adequate. No parameter is fitted to the correctness labels, and the adequacy judge used by PROBAR is explicitly denied access to reference answers: 'No adequacy classifier internal to PROBAR has access to reference answers.' The comparison with semantic entropy is therefore an empirical comparison of two estimators, not a reduction of one to the other. The closest concern is the AmbigQA leg, where the adequacy classifier (Mis22B) and the correctness judge (gpt3.5-turbo) are both LLMs asked for 'plausible' judgments from parametric knowledge; the authors themselves flag that the KBQA classifier 'was not explicitly evaluated for their performance, and we only "implicitly" validate their performance through the improved AUROC performance of PROBAR.' This is a validation gap and a potential shared-method confound, but it is not an identity or a fit: one model scores sampled responses without references, while the other scores the greedy response against reference answers. The 50-context upper-bound analysis uses the authors' own single-annotator adequacy labels for both the predictor and the correctness criterion, but the predicted quantity (correctness of the greedy response) is not the same as the predictor (proportion of adequate sampled responses), so no equation-level circularity is present. Self-citations (Baan et al. 2023; Ilia and Aziz 2024) are background or implementation details and are not load-bearing for the central claim.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central performance claim rests on the empirical validity of the LLM correctness judge and the task-specific adequacy classifiers; it does not introduce fitted parameters or new theoretical entities. The main risk is shared bias between the adequacy classifier (Mistral) and the correctness judge (gpt3.5) rather than overfitting.

assumptions (4)
  • domain assumption Correctness of the greedy response can be validly assessed by gpt3.5-turbo prompted with passage, question, and reference answers.
    Used as gold labels for AUROC in QA tasks (Section 4, Metrics; Appendix B.2). If this judge is biased, all AUROC comparisons rest on that bias.
  • standard math Sampling with temperature=1 and top_p=1 yields an unbiased sample of the model's response distribution, making Equation (3) an unbiased MC estimate of the adequacy probability.
    Section A.1 and Section 2 state unbiased sampling; used to justify PROBAR as a probability estimate.
  • domain assumption Adequacy is a binary notion: a response is adequate if plausible given the passage/question (RCQA), training data (KBQA), or continuation plausibility (NWP).
    PROBAR depends on this task-specific operationalization of adequacy (Section 3, Limitations).
  • domain assumption Human annotations by the authors (single annotator) are reliable enough to serve as gold standard for evaluating classifiers and computing upper bounds.
    Used for the 50-context upper-bound analysis and classifier selection (Section A.5, B.1). Single-annotator labels by the authors may carry bias.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Variability Need Not Imply Error: The Case of Adequate but Semantically Distinct Responses." pith.science (2026). https://pith.science/paper/H22SXQZ2

@misc{pith2026241215683,
  author       = {Pith},
  title        = {Pith review of: Variability Need Not Imply Error: The Case of Adequate but Semantically Distinct Responses},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H22SXQZ2}},
  note         = {Machine review of arXiv:2412.15683}
}
read the original abstract

With the broader use of language models (LMs) comes the need to estimate their ability to respond reliably to prompts (e.g., are generated responses likely to be correct?). Uncertainty quantification tools (notions of confidence and entropy, i.a.) can be used to that end (e.g., to reject a response when the model is `uncertain'). For example, Kuhn et al. (semantic entropy; 2022b) regard semantic variation amongst sampled responses as evidence that the model `struggles' with the prompt and that the LM is likely to err. We argue that semantic variability need not imply error--this being especially intuitive in open-ended settings, where prompts elicit multiple adequate but semantically distinct responses. Hence, we propose to annotate sampled responses for their adequacy to the prompt (e.g., using a classifier) and estimate the Probability the model assigns to Adequate Responses (PROBAR), which we then regard as an indicator of the model's reliability at the instance level. We evaluate PROBAR as a measure of confidence in selective prediction with OPT models (in two QA datasets and in next-word prediction, for English) and find PROBAR to outperform semantic entropy across prompts with varying degrees of ambiguity/open-endedness.

Figures

Figures reproduced from arXiv: 2412.15683 by the authors.

Figure 1
Figure 1. Bottom: a sampled-based approximation of [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. For each question, we show adequate responses (wrt a plausible interpretation of the prompt) in green [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. AUROC values for various quantifiers for Abg-COQA. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: AUROCs for Provo Corpus (left) and AmbigQA (right) tasks. For Provo Corpus (200 prompts), correctness [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Example of an ambiguous instance from Abg [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 7
Figure 7. Figure 7: Coverage vs ‘Precision’ for different uncer [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Coverage vs ‘Precision’ for different un [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 11
Figure 11. Figure 11: Coverage vs ‘Precision’ for different un [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: Coverage vs ‘Precision’ for different uncer [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 13
Figure 13. Figure 13: Entropy (E) and Semantic Entropy (SE), non-normalised for two datasets of ambiguous prompts. [PITH_FULL_IMAGE:figures/full_fig_p023_13.png]
Figure 14
Figure 14. Figure 14: Entropy and Semantic Entropy histograms across Provo Corpus prompts, for different models. tiveness of the metric overall. We speculate that a higher number of samples, will make PROBAR’s performance more robust to the misclassifications of the adequacy classifier it …
Figure 15
Figure 15. Figure 15: AUROC values for AbgCOQA’s development set on ambiguous prompts (left), unambiguous prompts [PITH_FULL_IMAGE:figures/full_fig_p024_15.png]
Figure 16
Figure 16. Figure 16: Average AUROC values (over 5 sampled responses) for various uncertainty indicators for the models analysed. They regard the 50 contexts that were manually annotated, and for the correctness of the sam￾ple, the manual annotation is used. we have manually annotated). Fo…
Figure 19
Figure 19. Figure 19: AUROC values (and their mean) for 50 sub [PITH_FULL_IMAGE:figures/full_fig_p025_19.png]
Figure 20
Figure 20. Figure 20: AUROC values (and their mean) for 50 sub [PITH_FULL_IMAGE:figures/full_fig_p026_20.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

74 extracted references · 12 canonical work pages

  1. [1]

    Lukas Aichberger, Kajetan Schweighofer, Mykyta Ielanskyi, and Sepp Hochreiter. 2024. Semantically diverse language generation for uncertainty estimation in language models. arXiv preprint arXiv:2406.04306

    arXiv 2024

  2. [2]

    Laura Aina and Tal Linzen. 2021. https://doi.org/10.18653/v1/2021.blackboxnlp-1.4 The language model understood the prompt was ambiguous: Probing syntactic uncertainty through generation . In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 42--57, Punta Cana, Dominican Republic. Association for C...

    doi:10.18653/v1/2021.blackboxnlp-1.4 2021

  3. [3]

    Lora Aroyo and Chris Welty. 2015. Truth is a lie: Crowd truth and the seven myths of human annotation. AI Magazine, 36(1):15--24

    2015

  4. [4]

    Joris Baan, Nico Daheim, Evgenia Ilia, Dennis Ulmer, Haau-Sing Li, Raquel Fern \'a ndez, Barbara Plank, Rico Sennrich, Chrysoula Zerva, and Wilker Aziz. 2023. Uncertainty in natural language generation: From theory to applications. arXiv preprint arXiv:2307.15703

    arXiv 2023

  5. [5]

    Yavuz Faruk Bakman, Duygu Nur Yaldiz, Baturalp Buyukates, Chenyang Tao, Dimitrios Dimitriadis, and Salman Avestimehr. 2024. https://doi.org/10.18653/v1/2024.acl-long.419 MARS : Meaning-aware response scoring for uncertainty estimation in generative LLM s . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1...

    doi:10.18653/v1/2024.acl-long.419 2024

  6. [6]

    Amanda Bertsch, Alex Xie, Graham Neubig, and Matthew Gormley. 2023. https://doi.org/10.18653/v1/2023.bigpicture-1.9 It ' s MBR all the way down: Modern generation techniques through the lens of minimum B ayes risk . In Proceedings of the Big Picture Workshop, pages 108--122, Singapore. Association for Computational Linguistics

    doi:10.18653/v1/2023.bigpicture-1.9 2023

  7. [7]

    Christopher M Bishop and Nasser M Nasrabadi. 2006. Pattern recognition and machine learning, volume 4. Springer

    2006

  8. [8]

    Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. 2024. Inside: Llms' internal states retain the power of hallucination detection. arXiv preprint arXiv:2402.03744

    arXiv 2024

Show all 74 references
  1. [9]

    Jiuhai Chen and Jonas Mueller. 2024. https://doi.org/10.18653/v1/2024.acl-long.283 Quantifying uncertainty in answers from any language model and enhancing their trustworthiness . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volum...

    2024 doi

  2. [10]

    Julius Cheng and Andreas Vlachos. 2024. https://aclanthology.org/2024.eacl-long.129 Measuring uncertainty in neural machine translation with similarity-sensitive entropy . In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Lingui...

    2024

  3. [11]

    Jeremy Cole, Michael Zhang, Dan Gillick, Julian Eisenschlos, Bhuwan Dhingra, and Jacob Eisenstein. 2023. Selectively answering ambiguous questions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 530--543

    2023

  4. [12]

    Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. 2024. https://doi.org/10.18653/v1/2024.acl-long.276 Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models ...

    2024 doi

  5. [13]

    Bryan Eikema and Wilker Aziz. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.754 Sampling-based approximations to minimum B ayes risk decoding for neural machine translation . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1...

    2022 doi

  6. [14]

    Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Sch \"u tze, and Yoav Goldberg. 2021. https://doi.org/10.1162/tacl_a_00410 Measuring and improving consistency in pretrained language models . Transactions of the Association for Computati...

    2021 doi

  7. [15]

    Angela Fan, Mike Lewis, and Yann Dauphin. 2018. https://doi.org/10.18653/v1/P18-1082 Hierarchical neural story generation . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889--898, Melbourne, Australia....

    2018 doi

  8. [16]

    Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625--630

    2024

  9. [17]

    Constanza Fierro and Anders S gaard. 2022. https://doi.org/10.18653/v1/2022.findings-acl.240 Factual consistency of multilingual pretrained language models . In Findings of the Association for Computational Linguistics: ACL 2022, pages 3046--3052, Dublin, Ireland. Association ...

    2022 doi

  10. [18]

    Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Fr \'e d \'e ric Blain, Francisco Guzm \'a n, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. 2020. https://doi.org/10.1162/tacl_a_00330 Unsupervised quality estimation for neural machine translation . Transacti...

    2020 doi

  11. [19]

    Xiang Gao, Jiaxin Zhang, Lalla Mouatadid, and Kamalika Das. 2024. https://aclanthology.org/2024.eacl-long.143 SPUQ : Perturbation-based uncertainty quantification for large language models . In Proceedings of the 18th Conference of the European Chapter of the Association for C...

    2024

  12. [20]

    Alex Graves. 2012. Sequence transduction with recurrent neural networks. In ICML Workshop on Representation Learning, volume abs/1211.3711

    2012 arXiv

  13. [21]

    Sebastian Gregor Gruber and Florian Buettner. 2024. A bias-variance-covariance decomposition of kernel scores for generative models. In Forty-first International Conference on Machine Learning

    2024

  14. [22]

    Meiqi Guo, Mingda Zhang, Siva Reddy, and Malihe Alikhani. 2021. Abg-coqa: Clarifying ambiguity in conversational question answering. In 3rd Conference on Automated Knowledge Base Construction

    2021

  15. [23]

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654

    2020 arXiv

  16. [24]

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. https://openreview.net/forum?id=rygGQyrFvH The curious case of neural text degeneration . In International Conference on Learning Representations

    2020

  17. [25]

    Bairu Hou, Yujian Liu, Kaizhi Qian, Jacob Andreas, Shiyu Chang, and Yang Zhang. 2023. Decomposing uncertainty for large language models through input clarification ensembling. arXiv preprint arXiv:2311.08718

    2023 arXiv

  18. [26]

    Evgenia Ilia and Wilker Aziz. 2024. https://aclanthology.org/2024.eacl-short.22 Predict the next word: humans exhibit uncertainty in this task and language models \_ \_ \_ \_ \_ . In Proceedings of the 18th Conference of the European Chapter of the Association for Computationa...

    2024

  19. [27]

    Myeongjun Jang, Deuk Sin Kwon, and Thomas Lukasiewicz. 2022. https://aclanthology.org/2022.coling-1.324 BECEL : Benchmark for consistency evaluation of language models . In Proceedings of the 29th International Conference on Computational Linguistics, pages 3680--3696, Gyeongj...

    2022

  20. [28]

    Nan-Jiang Jiang and Marie-Catherine de Marneffe. 2022. https://doi.org/10.1162/tacl_a_00523 Investigating reasons for disagreement in natural language inference . Transactions of the Association for Computational Linguistics, 10:1357--1374

    2022 doi

  21. [29]

    Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. https://doi.org/10.1162/tacl_a_00407 How can we know when language models know? on the calibration of language models for question answering . Transactions of the Association for Computational Linguistics, 9:962--977

    2021 doi

  22. [30]

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221

    2022 arXiv

  23. [31]

    Amita Kamath, Robin Jia, and Percy Liang. 2020. https://doi.org/10.18653/v1/2020.acl-main.503 Selective question answering under domain shift . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5684--5696, Online. Association for...

    2020 doi

  24. [32]

    Jannik Kossen, Jiatong Han, Muhammed Razzak, Lisa Schut, Shreshth Malik, and Yarin Gal. 2024. Semantic entropy probes: Robust and cheap hallucination detection in llms. arXiv preprint arXiv:2406.15927

    2024 arXiv

  25. [33]

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2022 a . Clam: Selective clarification for ambiguous questions with generative language models. arXiv preprint arXiv:2212.07769

    2022 arXiv

  26. [34]

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2022 b . Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In NeurIPS ML Safety Workshop

    2022

  27. [35]

    Aviral Kumar and Sunita Sarawagi. 2019. Calibration of encoder decoder models for neural machine translation. arXiv preprint arXiv:1903.00802

    2019 arXiv

  28. [36]

    Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. 2024 a . From generation to judgment: Opportunities and challenges of llm-as-a-judge. arXiv preprint arXiv:2411.16594

    2024

  29. [37]

    Xiang Lisa Li, Vaishnavi Shrivastava, Siyan Li, Tatsunori Hashimoto, and Percy Liang. 2024 b . Benchmarking and improving generator-validator consistency of language models. In The Twelfth International Conference on Learning Representations

    2024

  30. [38]

    Chin-Yew Lin and Franz Josef Och. 2004. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the 42nd annual meeting of the association for computational linguistics (ACL-04), pages 605--612

    2004

  31. [39]

    Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2024 a . https://doi.org/10.18653/v1/2024.emnlp-main.578 Contextualized sequence likelihood: Enhanced confidence scores for natural language generation . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language...

    2024 doi

  32. [40]

    Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2024 b . Generating with confidence: Uncertainty quantification for black-box large language models. Transactions on Machine Learning Research

    2024

  33. [41]

    Chen Ling, Xujiang Zhao, Xuchao Zhang, Wei Cheng, Yanchi Liu, Yiyou Sun, Mika Oishi, Takao Osaki, Katsushi Matsuda, Jie Ji, Guangji Bai, Liang Zhao, and Haifeng Chen. 2024. https://doi.org/10.18653/v1/2024.naacl-long.184 Uncertainty quantification for in-context learning of la...

    2024 doi

  34. [42]

    Steven G Luke and Kiel Christianson. 2018. The provo corpus: A large eye-tracking corpus with predictability norms. Behavior research methods, 50:826--833

    2018

  35. [43]

    Andrey Malinin and Mark Gales. 2021. Uncertainty estimation in autoregressive structured prediction. In International Conference on Learning Representations

    2021

  36. [44]

    Potsawee Manakul, Adian Liusie, and Mark Gales. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.557 S elf C heck GPT : Zero-resource black-box hallucination detection for generative large language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural...

    2023 doi

  37. [45]

    Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau

    Sabrina J. Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. 2022. https://doi.org/10.1162/tacl_a_00494 Reducing conversational agents ' overconfidence through linguistic calibration . Transactions of the Association for Computational Linguistics, 10:857--872

    2022 doi

  38. [46]

    Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.466 A mbig QA : Answering ambiguous open-domain questions . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)...

    2020 doi

  39. [47]

    AI Mistral. 2024. https://mistral.ai/news/mistral-nemo/ Mistral nemo

    2024

  40. [48]

    Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. https://doi.org/10.18653/v1/2020.acl-main.441 Adversarial NLI : A new benchmark for natural language understanding . In Proceedings of the 58th Annual Meeting of the Association for Comp...

    2020 doi

  41. [49]

    Alexander Nikitin, Jannik Kossen, Yarin Gal, and Pekka Marttinen. 2024. Kernel language entropy: Fine-grained uncertainty quantification for llms from semantic similarities. arXiv preprint arXiv:2405.20003

    2024 arXiv

  42. [50]

    Barbara Plank. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.731 The `` problem '' of human label variation: On ground truth in data, modeling and evaluation . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671--10682, Ab...

    2022 doi

  43. [51]

    Ella Rabinovich, Samuel Ackerman, Orna Raz, Eitan Farchi, and Ateret Anaby Tavor. 2023. https://aclanthology.org/2023.gem-1.12 Predicting question-answering performance of large language models through semantic consistency . In Proceedings of the Third Workshop on Natural Lang...

    2023

  44. [52]

    Harsh Raj, Vipul Gupta, Domenic Rosati, and Subhabrata Majumdar. 2023. Semantic consistency for assuring reliability of large language models. arXiv preprint arXiv:2308.09138

    2023 arXiv

  45. [53]

    Vipula Rawte, Swagata Chakraborty, Agnibh Pathak, Anubhav Sarkar, S.M Towhidul Islam Tonmoy, Aman Chadha, Amit Sheth, and Amitava Das. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.155 The troubling emergence of hallucination in large language models - an extensive definit...

    2023 doi

  46. [54]

    Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshminarayanan, and Peter J Liu. 2022. Out-of-distribution detection and selective generation for conditional language models. In The Eleventh International Conference on Learning Representations

    2022

  47. [55]

    Pritish Sahu, Michael Cogswell, Yunye Gong, and Ajay Divakaran. 2022. Unpacking large language models with conceptual consistency. arXiv preprint arXiv:2209.15093

    2022 arXiv

  48. [56]

    Nino Scherrer, Claudia Shi, Amir Feder, and David Blei. 2024. Evaluating the moral beliefs encoded in llms. Advances in Neural Information Processing Systems, 36

    2024

  49. [57]

    Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. 2024. Judging the judges: Evaluating alignment and vulnerabilities in llms-as-judges. arXiv preprint arXiv:2406.12624

    2024 arXiv

  50. [58]

    Francesco Tonolini, Nikolaos Aletras, Jordan Massiah, and Gabriella Kazai. 2024. https://doi.org/10.18653/v1/2024.findings-acl.728 B ayesian prompt ensembles: Model uncertainty estimation for black-box large language models . In Findings of the Association for Computational Li...

    2024 doi

  51. [59]

    Liam van der Poel, Ryan Cotterell, and Clara Meister. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.399 Mutual information alleviates hallucinations in abstractive summarization . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pa...

    2022 doi

  52. [60]

    Neeraj Varshney, Swaroop Mishra, and Chitta Baral. 2022. https://doi.org/10.18653/v1/2022.findings-acl.158 Investigating selective prediction approaches across several tasks in IID , OOD , and adversarial settings . In Findings of the Association for Computational Linguistics:...

    2022 doi

  53. [61]

    Neeraj Varshney, Wenlin Yao, Hongming Zhang, Jianshu Chen, and Dong Yu. 2023. A stitch in time saves nine: Detecting and mitigating hallucinations of llms by validating low-confidence generation. arXiv preprint arXiv:2307.03987

    2023 arXiv

  54. [62]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations

    2023

  55. [63]

    Yijun Xiao and William Yang Wang. 2021. https://doi.org/10.18653/v1/2021.eacl-main.236 On hallucination and predictive uncertainty in conditional language generation . In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistic...

    2021 doi

  56. [64]

    Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2023. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. arXiv preprint arXiv:2306.13063

    2023 arXiv

  57. [65]

    Jiacheng Xu, Shrey Desai, and Greg Durrett. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.508 Understanding neural abstractive summarization models via uncertainty . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 62...

    2020 doi

  58. [66]

    Yasin Abbasi Yadkori, Ilja Kuzborskij, Andr \'a s Gy \"o rgy, and Csaba Szepesv \'a ri. 2024. To believe or not to believe your llm. arXiv preprint arXiv:2406.02543

    2024 arXiv

  59. [67]

    Adam Yang, Chen Chen, and Konstantinos Pitas. 2024. Just rephrase it! uncertainty estimation in closed-source language models via multiple rephrased queries. arXiv preprint arXiv:2405.13907

    2024 arXiv

  60. [68]

    Caiqi Zhang, Fangyu Liu, Marco Basaldella, and Nigel Collier. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.299 LUQ : Long-text uncertainty quantification for LLM s . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5244--526...

    2024 doi

  61. [69]

    Jiaxin Zhang, Zhuohang Li, Kamalika Das, Bradley Malin, and Sricharan Kumar. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.1032 SAC ^3 : Reliable hallucination detection in black-box language models via semantic-aware cross-check consistency . In Findings of the Associ...

    2023 doi

  62. [70]

    Michael JQ Zhang and Eunsol Choi. 2023. Clarify when necessary: Resolving ambiguity through interaction with lms. arXiv preprint arXiv:2311.09469

    2023 arXiv

  63. [71]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068

    2022 arXiv

  64. [72]

    Yukun Zhao, Lingyong Yan, Weiwei Sun, Guoliang Xing, Chong Meng, Shuaiqiang Wang, Zhicong Cheng, Zhaochun Ren, and Dawei Yin. 2024. https://doi.org/10.18653/v1/2024.naacl-long.390 Knowing what LLM s DO NOT know: A simple yet effective self-detection method . In Proceedings of ...

    2024 doi

  65. [73]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  66. [74]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.