Pith. sign in

REVIEW 4 major objections 4 minor 39 references

The Consistency Hypothesis in Uncertainty Quantification for Large Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Similar LLM answers are likelier to be correct, new tests show.

desk verdict Useful formalization and broad empirical map of the consistency hypothesis, but the statistical verification test is anti-conservative because it treats dependent pairwise similarities as independent. read the letter →

arxiv 2506.21849 v1 pith:H5QZ7NT2 submitted 2025-06-27 cs.CL cs.AIcs.LG

Quan Xiao , Debarun Bhattacharjya , Balaji Ganesan , Radu Marinescu , Katsiaryna Mirylenka , Nhan H Pham , Michael Glass , Junkyu Lee This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords consistencyhypothesisuncertaintyquantificationlargelanguagemodelsblack-boxconfidenceestimationgenerationsimilaritystatisticaltestingsemanticselective
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to establish that a long-standing informal assumption in uncertainty quantification actually holds: when a language model is asked the same question several times, a correct answer resembles the other answers more than an incorrect answer does. It formalizes this as three testable statements (Sim-Correct, Sim-Separate, and Sim-Any), each comparing the mean pairwise similarity of correct and incorrect generations, and validates them across eight datasets in question answering, summarization, and text-to-SQL. The authors single out Sim-Any because it needs no labels: a generation's confidence can be read directly from its average similarity to all other generations. This matters because it turns a black-box, API-only procedure into a principled confidence signal, and the paper's geometric- and harmonic-mean aggregators beat several baselines on AUROC and AUARC.

What carries the argument

The load-bearing object is the pair $(s(\cdot), f(\cdot))$: a similarity metric on $[0,1]$ between two generations, such as Jaccard, Rouge-L, or SBERT cosine, and an aggregation function $f$ that combines a generation's similarities to others into a scalar. For each instance the paper forms two sets, $S^C$ (similarities involving correct generations) and $S^I$ (similarities involving incorrect ones), and tests whether their means satisfy $\mu^C > \mu^I$. Algorithm 1 splits the dataset into groups and runs one-sided $t$-tests to compute the fraction of groups that reject the null, producing the metrics $\Delta\mu$, $\rho(n_g)$, and $\theta^*$. The Sim-Any variant makes the machinery directly usable for UQ, since $f$ over all other generations is the confidence score.

What would settle it

Take any dataset where a single wrong pattern dominates (for example a text-to-SQL benchmark with one recurring SQL error), so incorrect generations are highly similar to each other; computing the Sim-Any mean difference and the group-level fraction $\rho(n_g)$ on that dataset would reveal whether the hypothesis still holds when incorrect answers cluster. A second check is to recompute the p-values in Algorithm 1 with a permutation test or paired bootstrap that respects shared generations; if the fraction of verified groups drops below the 80% threshold across datasets, the statistical verification is unsupported.

Watch

Extended reading notes

Core claim

The central claim is the consistency hypothesis: for a given query, the mean pairwise similarity among correct generations exceeds the mean pairwise similarity involving incorrect generations, under three formal variants. In the most actionable variant, Sim-Any, each generation is compared with all other generations regardless of correctness, so confidence can be estimated without ground-truth labels. The paper reports that across eight benchmark datasets and three tasks the mean difference is positive for all three hypotheses, that group-level statistical verification holds to varying degrees, and that aggregation of similarities with geometric or harmonic means yields black-box confidence scores that outperform the baselines considered.

Load-bearing premise

Algorithm 1 treats every pairwise similarity as an independent sample in the t-test, but the same generation appears in many pairs, so the samples are not independent; if this inflation of sample size is not justified, the reported significance levels and verified-group fractions are optimistic.

Editorial extensions

If this is right

  • If the consistency hypothesis holds, a black-box API that only sees generated text can rank answers by correctness without any model internals.
  • Geometric and harmonic mean aggregators provide a direct improvement over arithmetic-mean aggregation, which the paper identifies with the degree-based method of Lin et al.
  • The verification metrics give a standard way to compare how much a dataset/model combination conforms to consistency, and to choose similarity and aggregation functions accordingly.
  • Because Sim-Any holds across QA, summarization, and text-to-SQL, consistency-based selective generation (rejecting low-confidence outputs) is justified on these tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same verification procedure could be rerun as a paired test that accounts for overlapping generations; if the positive mean differences persist, the practical conclusion strengthens, and if not, the reported p-values should be read with caution.
  • The paper leaves open whether the consistency hypothesis transfers to longer, multi-step reasoning or tool-use generations, where a single correct answer may legitimately look different from other correct answers.
  • A testable extension is to use the Sim-Any score as a training signal or as a feature alongside verbalized confidence, rather than as a standalone score.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper formalizes the 'consistency hypothesis' underlying black-box uncertainty quantification methods for LLMs, proposing three concrete statements (Sim-Any, Sim-Correct, Sim-Separate) and a statistical verification framework based on pairwise or aggregated similarities between sampled generations. It reports empirical results across eight datasets and three tasks, showing that the mean similarity of correct generations exceeds that of incorrect ones, and uses the Sim-Any hypothesis to design simple aggregation-based confidence scores (arithmetic, geometric, harmonic means) that outperform several baselines. The paper also introduces metrics (Δμ, ρ(n_g), θ*) for quantifying the extent to which a dataset conforms to each hypothesis.

Significance. If the statistical claims are properly supported, this paper would make a useful contribution by turning an implicit and widely used assumption into testable formal statements, and by demonstrating that simple similarity aggregation can yield competitive black-box confidence signals. The strengths include the breadth of the empirical study (8 datasets, 3 tasks, multiple models), the clarity of the formal framework, the inclusion of ablations over similarity metrics, aggregation functions, sampling methods, and Rouge-L thresholds, and the fact that the proposed UQ methods are simple and parameter-free apart from the correctness thresholds. However, the main formal verification procedure has a load-bearing statistical flaw that undermines the 'rigorous statistical procedure' claim as currently stated.

major comments (4)
  1. [Section 3, Algorithm 1] The t-test in Algorithm 1 assumes that the entries of the pooled sets SC_g and SI_g are independent. These entries are pairwise similarities that share generations: for m generations per instance, each generation appears in m-1 pairs, and for symmetric similarity metrics both (j,k) and (k,j) are included, doubling each unordered pair. This violation of the independence assumption leads to systematically underestimated standard errors and anti-conservative p-values. Because the fraction of verified groups ρ(n_g) and the derived quantity θ* are used to conclude that hypotheses are 'verified to some extent' and to compare aggregation functions (e.g., Table 1 and Figures 4-5), the claimed statistical verification is not supported as stated. I recommend using a permutation test that reshuffles correctness labels within each instance, or a clustered test that treats the instance (or generation) as the unit of analysis, and reporting the resulting p-values and group verification fractions.
  2. [Section 3, metric ρ(n_g)] The metric ρ(n_g) is the fraction of groups for which the null hypothesis is rejected at p≤0.05, but the paper never compares this quantity to its expected value under the null. Even with truly independent data, ρ(n_g) would be approximately 0.05 by chance; with the dependency issue described above it may be substantially higher. Without a null calibration, such as permuting correctness labels within instances, a reader cannot interpret an absolute value like ρ=0.8 as evidence that the hypothesis holds rather than as an artifact of inflated false-positive rates. This affects the interpretability of the 'verification extent' and the conclusions drawn from the trade-off curves.
  3. [Section 5, Table 1 vs Table 2] The text states that the geometric and harmonic aggregation functions were 'inspired by their superior empirical performance on verifying consistency hypothesis in Table 1,' where Table 1 is computed on the Natural Questions dataset. The main UQ evaluation in Table 2 is then reported only on CoQA, TriviaQA, and Spider. Choosing the methods based on NQ and evaluating on other datasets is a mild but real selection effect, and it is not acknowledged in the evaluation section. The authors should either include NQ in the main UQ comparison or explicitly describe this as a model-selection step and discuss its implications for the generality of the UQ gains.
  4. [Section 2.2.1 and Section 2.2.3] The aggregated version of the consistency hypotheses is under-specified. In Section 2.2.1, the definition of the aggregated similarity contains a type error: "¯s^j_i = f(s^{j,i_1}_i, ..., s^{j,i_m}_i) with {s^{j,i_1}_i, ..., s^{j,i_m}_i} ∈ Y*_i" is not meaningful (a set of similarities cannot be a member of the set of correct responses). More importantly, for Sim-Any, the text says aggregation is over "all generations" but does not state whether the self-similarity s^{j,j}_i (typically equal to 1) is included. For arithmetic mean the inclusion of a constant 1 does not affect ranking, but for geometric and harmonic means it changes the aggregated values and can alter the relative ordering of generations. The paper should define these sets precisely, explicitly excluding the self-pair, and state whether the reported UQ results include or exclude the self-similarity.
minor comments (4)
  1. [Algorithm 1] The algorithm's input is described as 'similarity matrix of generations,' but constructing SC_g and SI_g also requires the correctness labels of the generations; the algorithm should list the labels as an input or explicitly state that they are available.
  2. [Table 1 and Table 3] The header 'Mean difference when n_g=10' is not defined in Section 3, where Δμ is introduced as a dataset-level metric. Please clarify whether this is the mean of per-group mean differences and describe how the error bars are computed.
  3. [Section 2.2.3 / Section 4] The mapping between 'Hypo #1, #2, #3' and 'H1, H2, H3' is introduced only in Section 2.2.3 but the figures and text use H1-H3 without a pointer back to the mapping; adding a sentence at first use of H1-H3 would improve readability.
  4. [Section 5] The text refers to the 'degree' approach used after spectral clustering, but Table 2 labels the baseline 'spec-ecc.' Please make the terminology consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the consistency hypotheses are tested against external benchmark data and the UQ method is an application, not a restatement, of those tests.

full rationale

The paper's three consistency hypotheses are formalized as mean comparisons between label-defined similarity sets (SC and SI) and are verified empirically on held-out benchmark data; the Sim-Any UQ method subsequently applies the same aggregated similarity as a confidence score without using labels, so the method is an application of the tested hypothesis rather than an assumption of it. The geometric/harmonic aggregation choices are motivated by a verification table on NQ and then evaluated on different datasets (CoQA, TriviaQA, Spider), which is mild selection but not circular derivation. No load-bearing self-citation or imported uniqueness claim appears; the statistical concern about dependent t-test samples affects validity of significance claims, not circularity of the derivation.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the choice of similarity metric, the operational definition of correctness, and the validity of the t-test. None of these are fitted to make the hypothesis true, but they shape the empirical measurement. The study does not introduce new physical or conceptual entities.

free parameters (2)
  • Rouge-L correctness thresholds = 0.5 for QA, 0.2 for summarization
    Defines whether a generation is correct for QA and summarization, directly determining the SC and SI sets. The authors ablate the threshold in Appendix A.5, so the qualitative conclusion is stable, but the exact numbers depend on this hand-chosen value.
  • Verification thresholds = p-value <= 0.05; rho* = 0.8
    These thresholds determine the reported fraction of verified groups and the maximum relative group number. The authors state conclusions are based on curve comparison rather than the specific 0.8 value, but the headline numbers in tables depend on these choices.
assumptions (3)
  • domain assumption Similarity metrics, Jaccard, ROUGE, and sbert, capture the semantic relation relevant to correctness.
    The formal statements compare similarity sets, and the validity of the hypotheses is measured only through the chosen s(). If s() misses task-relevant equivalence, the verification results may not transfer to other metrics. This enters in Section 2.1.
  • domain assumption Pairwise similarity entries are treated as independent observations in the t-tests.
    Algorithm 1 checks the null hypothesis of equal means with a t-test. Because pairs share generations, this independence assumption is violated in practice, which means the reported p-values are likely anti-conservative. This enters in Algorithm 1 and Section 3.
  • domain assumption Rouge-L thresholds define ground-truth correctness for QA and summarization.
    Correctness is not intrinsic; the study defines a generation as correct if Rouge-L against a reference passes a threshold. The authors ablate thresholds, but the hypothesis statements are only about this operational correctness. This enters in Section 4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Consistency Hypothesis in Uncertainty Quantification for Large Language Models." pith.science (2026). https://pith.science/paper/H5QZ7NT2

@misc{pith2026250621849,
  author       = {Pith},
  title        = {Pith review of: The Consistency Hypothesis in Uncertainty Quantification for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H5QZ7NT2}},
  note         = {Machine review of arXiv:2506.21849}
}
read the original abstract

Estimating the confidence of large language model (LLM) outputs is essential for real-world applications requiring high user trust. Black-box uncertainty quantification (UQ) methods, relying solely on model API access, have gained popularity due to their practical benefits. In this paper, we examine the implicit assumption behind several UQ methods, which use generation consistency as a proxy for confidence, an idea we formalize as the consistency hypothesis. We introduce three mathematical statements with corresponding statistical tests to capture variations of this hypothesis and metrics to evaluate LLM output conformity across tasks. Our empirical investigation, spanning 8 benchmark datasets and 3 tasks (question answering, text summarization, and text-to-SQL), highlights the prevalence of the hypothesis under different settings. Among the statements, we highlight the `Sim-Any' hypothesis as the most actionable, and demonstrate how it can be leveraged by proposing data-free black-box UQ methods that aggregate similarities between generations for confidence estimation. These approaches can outperform the closest baselines, showcasing the practical value of the empirically observed consistency hypothesis.

Figures

Figures reproduced from arXiv: 2506.21849 by the authors.

Figure 1
Figure 1. T-SNE projections of 30 generations represented by sentence embeddings, for 2 instances each from the CoQA and [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Visualization of the distributions of the Jaccard [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Verification of hypotheses using mean difference [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Verification of hypotheses (pairwise consistency statements) using the fraction of verified groups [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Verification of hypotheses using the fraction of verified groups [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Impact of different datasets and models on verifying Hypothesis 1–3 for the text-to-SQL task. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Effects of sampling approaches to the verification [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: We also provide the validation of the pairwise consistency hypotheses using Rouge-L similarity in Figure 9. All of [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 8
Figure 8. Figure 8: Verification of three hypotheses using Jaccard pairwise similarity on various datasets for QA, summarization, and [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Verification of different hypotheses with Rouge-L pairwise similarity on QA, summarization, and text-to-SQL [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Verification of different hypotheses using Jaccard similarity and arithmetic mean as aggregation function on QA, [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Verification of hypotheses for non-default models using mean difference [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Impact of aggregation functions for verifying aggregation version of Hypothesis 1 on different datasets in QA, [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 13
Figure 13. Figure 13: Verfication of hypotheses under varying Rouge-L score thresholds for the correctness of generations using mean [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]
Figure 14
Figure 14. Figure 14: Visualization of the distributions of the Jaccard pairwise similarities of correct cluster [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 30 canonical work pages

  1. [1]

    Adaptation with self-evaluation to improve selective prediction in LLM s

    Jiefeng Chen, Jinsung Yoon, Sayna Ebrahimi, Sercan O Arik, Tomas Pfister, and Somesh Jha. Adaptation with self-evaluation to improve selective prediction in LLM s. In Conference on Empirical Methods in Natural Language Processing, pages 5190--5213, 2023

    work page 2023

  2. [2]

    Quantifying uncertainty in answers from any language model via intrinsic and extrinsic confidence assessment

    Jiuhai Chen and Jonas Mueller. Quantifying uncertainty in answers from any language model via intrinsic and extrinsic confidence assessment. arXiv preprint arXiv:2308.16175, 2023

    arXiv 2023

  3. [3]

    Selectively answering ambiguous questions

    Jeremy Cole, Michael Zhang, Dan Gillick, Julian Eisenschlos, Bhuwan Dhingra, and Jacob Eisenstein. Selectively answering ambiguous questions. In Conference on Empirical Methods in Natural Language Processing, pages 530--543, 2023

    work page 2023

  4. [4]

    The well-calibrated B ayesian

    A Philip Dawid. The well-calibrated B ayesian. Journal of the American Statistical Association, 77 0 (379): 0 605--610, 1982

    work page 1982

  5. [5]

    Structure-grounded pretraining for text-to-sql

    Xiang Deng, Ahmed Hassan Awadallah, Christopher Meek, Oleksandr Polozov, Huan Sun, and Matthew Richardson. Structure-grounded pretraining for text-to-sql. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 2021

    work page 2021

  6. [6]

    On the foundations of noise-free selective classification

    Ran El-Yaniv et al. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11 0 (5), 2010

    2010

  7. [7]

    Dropout as a bayesian approximation: Representing model uncertainty in deep learning

    Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Proc. International Conference on Machine Learning, pages 1050--1059, 2016

    work page 2016

  8. [8]

    SPUQ : P erturbation-based uncertainty quantification for large language models

    Xiang Gao, Jiaxin Zhang, Lalla Mouatadid, and Kamalika Das. SPUQ : P erturbation-based uncertainty quantification for large language models. In Proceedings of the European Chapter of the Association for Computational Linguistics, pages 2336--2346, 2024

    work page 2024

Show all 39 references
  1. [9]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming -- the rise of code intelligence, 2024

    2024

  2. [10]

    Teaching machines to read and comprehend

    Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Proc. Advances in Neural Information Processing Systems, pages 1693--1701, 2015

    2015

  3. [11]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In Proc. International Conference on Learning Representations, 2022

    2022

  4. [12]

    Prompting is not a substitute for probability measurements in large language models

    Jennifer Hu and Roger Levy. Prompting is not a substitute for probability measurements in large language models. In Conference on Empirical Methods in Natural Language Processing, pages 5040--5060, 2023

    2023

  5. [13]

    Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension

    Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page 1601...

    2017

  6. [14]

    Language models (mostly) know what they know

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022

    2022 arXiv

  7. [15]

    Calibration-tuning: Teaching large language models to know what they don ' t know

    Sanyam Kapoor, Nate Gruver, Manley Roberts, Arka Pal, Samuel Dooley, Micah Goldblum, and Andrew Wilson. Calibration-tuning: Teaching large language models to know what they don ' t know. In Proceedings of the 1st Workshop on Uncertainty-Aware NLP (UncertaiNLP 2024), pages 1--14, 2024

    2024

  8. [16]

    Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In Proc. International Conference on Learning Representations, 2022

    2022

  9. [17]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav...

    2019

  10. [18]

    Can LLM already serve as a database interface? A big bench for large-scale database grounded text-to- SQL s

    Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al. Can LLM already serve as a database interface? A big bench for large-scale database grounded text-to- SQL s. In Proc. Advances in Neural Information Processin...

    2024

  11. [19]

    Teaching models to express their uncertainty in words

    Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. Transactions on Machine Learning Research, 2022

    2022

  12. [20]

    Generating with confidence: Uncertainty quantification for black-box large language models

    Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Generating with confidence: Uncertainty quantification for black-box large language models. Transactions on Machine Learning Research, 2023

    2023

  13. [21]

    Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. In Conference on Empirical Methods in Natural Language Processing, 2023

    2023

  14. [22]

    Introducing meta llama 3: The most capable openly available LLM to date

    Meta. Introducing meta llama 3: The most capable openly available LLM to date. https://ai.meta.com/blog/meta-llama-3/, 2024. Accessed: 2024-06-14

    2024

  15. [23]

    Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau

    Sabrina J. Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. Reducing conversational agents ' overconfidence through linguistic calibration. Transactions of the Association for Computational Linguistics, 10: 0 857--872, 2022. doi:10.1162/tacl_a_00494

    2022 doi

  16. [24]

    Cox, Ruchir Puri, and Rameswar Panda

    Mayank Mishra, Matt Stallone, Gaoyuan Zhang, Yikang Shen, Aditya Prasad, Adriana Meza Soria, Michele Merler, Parameswaran Selvam, Saptha Surendran, Shivdeep Singh, Manish Sethi, Xuan-Hong Dang, Pengyuan Li, Kun-Lung Wu, Syed Zawad, Andrew Coleman, Matthew White, Mark Lewis, Ra...

    2024

  17. [25]

    Mixtral of experts

    MistralAI. Mixtral of experts. https://mistral.ai/news/mixtral-of-experts/, 2023. Accessed: 2024-06-14

    2023

  18. [26]

    Verification of probabilistic predictions: A brief review

    Allan H Murphy and Edward S Epstein. Verification of probabilistic predictions: A brief review. Journal of Applied Meteorology and Climatology, 6 0 (5): 0 748--755, 1967

    1967

  19. [27]

    Cohen, and Mirella Lapata

    Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! T opic-aware convolutional neural networks for extreme summarization. In Conference on Empirical Methods in Natural Language Processing, 2018

    2018

  20. [28]

    DIN-SQL : Decomposed in-context learning of text-to- SQL with self-correction

    Mohammadreza Pourreza and Davood Rafiei. DIN-SQL : Decomposed in-context learning of text-to- SQL with self-correction. In Proc. Advances in Neural Information Processing Systems, volume 36, 2024

    2024

  21. [29]

    Siva Reddy, Danqi Chen, and Christopher D. Manning. CoQA : A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7: 0 249--266, 2019

    2019

  22. [30]

    Out-of-distribution detection and selective generation for conditional language models

    Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshminarayanan, and Peter J Liu. Out-of-distribution detection and selective generation for conditional language models. In Proc. International Conference on Learning Representations, 2023

    2023

  23. [31]

    Code llama: Open foundation models for code, 2024

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong...

    2024

  24. [32]

    Liu, and Christopher D

    Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073--1083, July 2017

    2017

  25. [33]

    Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback

    Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Conference on...

    2023

  26. [34]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  27. [35]

    Calibrating large language models using their generations only

    Dennis Ulmer, Martin Gubri, Hwaran Lee, Sangdoo Yun, and Seong Oh. Calibrating large language models using their generations only. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15440--15459, 2024

    2024

  28. [36]

    Quantifying uncertainties in natural language processing tasks

    Yijun Xiao and William Yang Wang. Quantifying uncertainties in natural language processing tasks. In Proc. Association for the Advancement of Artificial Intelligence, volume 33, pages 7322--7329, 2019

    2019

  29. [37]

    Can LLM s express their uncertainty? A n empirical evaluation of confidence elicitation in LLM s

    Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can LLM s express their uncertainty? A n empirical evaluation of confidence elicitation in LLM s. 2024

    2024

  30. [38]

    Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task

    Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Conference on Empirical Methods i...

    2018

  31. [39]

    Hot or cold? A daptive temperature sampling for code generation with large language models

    Yuqi Zhu, Jia Li, Ge Li, YunFei Zhao, Zhi Jin, and Hong Mei. Hot or cold? A daptive temperature sampling for code generation with large language models. In Proc. Association for the Advancement of Artificial Intelligence, volume 38, pages 437--445, 2024

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.