Pith. sign in

REVIEW 3 major objections 6 minor 45 references

Evaluation Pitfalls and Sparsity Limitations in LLM-based Confidence Estimates for Classification

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper shows that LLM self-reported confidence scores are so concentrated on a few values that the choice of interpolation in AUARC flips method rankings, and that weighting digit logprobs restores fine-grained scores at no extra…

desk verdict The sparsity and interpolation findings are solid and worth a serious look; the proposed verbalization logprobs is clever but the +2.3 AUARC claim rests on one model and no error bars. read the letter →

arxiv 2608.04899 v1 pith:CH7A2ZXY submitted 2026-08-05 cs.CL

Elena Merdjanovska , Omar Zaidan , Andreas Rücklé This is my paper · ORCID
classification cs.CL
keywords confidenceestimationselectiveclassificationverbalizationAUARCinterpolationtokenlogprobssparsitylargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that confidence scores produced by asking an LLM to verbalize its own probability are not continuous: one model produced only eight distinct values on SST-2, with more than half of all answers stating exactly 95%. That sparsity is not just a practical nuisance, because the standard way of computing the area under the accuracy-rejection curve connects the few observed points with straight lines, which artificially inflates the scores of sparse methods; switching to stepwise interpolation drops those methods by up to 4.5 points and flips the supposed best method into the worst. The paper then introduces verbalization logprobs, which replaces the sampled digit string with an expectation over the model's token probabilities for each digit, turning 8–12 distinct values into 630–2,227. It reports that this method beats vanilla verbalization by 2.3 AUARC points at no additional inference cost, and matters because selective classification depends on being able to set fine-grained rejection thresholds.

What carries the argument

The load-bearing object is the verbalization logprobs estimator: for a verbalized confidence whose tens digit is token $x_i$ and whose units digit is token $x_{i+1}$, the confidence is $$\sum_{d=0}^{9} 10\,d\,P(x_i=d) + \sum_{d=0}^{9} d\,P(x_{i+1}=d),$$ computed from the model's per-token probabilities rather than from the sampled digit string. The paper pairs this with the evaluation mechanism: AUARC computed with stepwise rather than linear interpolation, which is what makes sparse scores comparable. The estimator's job is to re-introduce the information discarded when a model says '95' and the confidence is recorded as exactly 0.95; the stepwise AUARC rule stops that sparsity from silently inflating the scores of weak methods.

What would settle it

Run verbalization logprobs on a model whose tokenizer emits '95' as one token rather than '9' and '5', and compare AUARC rankings with a model that has single-digit tokens: if the advantage disappears or cannot be calculated, the single-token assumption is load-bearing. A second test is to recompute the method with exact conditional digit probabilities—one forward pass for each possible tens digit—on the same four datasets; if the independence approximation changes the rankings or the gains, the approximation rather than the underlying signal is driving the reported result.

Watch

Extended reading notes

Core claim

The discovery is that sparsity in verbalized confidence is large, consequential, and fixable with information already present in the API response. Across four datasets and two LLMs, 45–93% of verbalized confidences fall in the top five values, and the digit 95 dominates. In the AUARC evaluation, the interpolation step—whether the accuracy-rejection curve is joined by straight lines or by steps—changes absolute scores by 0.7–4.5 points and reverses rankings, so consistency sampling, best under linear interpolation, becomes worst under stepwise interpolation. The paper claims that stepwise interpolation is the correct choice for sparse confidence scores, and that under it the proposed verbalization logprobs method—weighting each digit of the verbalized number by its token probability to form an expected value—achieves the best AUARC, +2.3 points over vanilla verbalization, reduces concentration in top-five values from 92% to 1%, and, unlike sampling-based methods, does so with a single inference call.

Load-bearing premise

The method's benefits depend on the API returning faithful token probabilities for all ten digits at both the tens and units positions and on each digit being a separate single token; if the provider only exposes top-k logprobs, or the tokenizer merges digits, or the model outputs a single- or three-digit number, the estimator cannot be computed as defined.

Editorial extensions

If this is right

  • Switching published AUARC evaluations from linear to stepwise interpolation will reorder the rankings of verbalization and sampling methods; consistency sampling, the winner under linear interpolation, becomes the loser.
  • Sparse confidence scores force large rejection jumps to gain small accuracy: on SST-2, moving from 97.9% to 98.7% accuracy requires rejecting 39.7% of predictions instead of 23.6%.
  • Verbalization logprobs gives fine-grained thresholds, with at least 630 unique confidence values per dataset versus 8–12, so a user can reach at least 98% accuracy by rejecting 28.1% of examples instead of 39.7%.
  • The method matches the strongest sampling baseline on AUARC (0.736 vs. 0.734) at 1.7× inference cost instead of 6.8×, since it needs only one call.
  • Verbalization logprobs outperforms both vanilla verbalization and raw token logprobs on the metrics reported, indicating that the digit distribution carries signal beyond the class-token probability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's experiments, the interpolation critique should apply to other threshold-based metrics such as AUPRC, AUROC, and PRR whenever the score distribution is discrete; the paper notes this but does not test it.
  • The independent-position approximation is not intrinsic: with full model access one could compute the exact conditional expectation of the units digit given each tens digit, and the formula could be generalized to one-digit or three-digit verbalizations, removing the currently undefined cases.
  • The strong preference for values like 95, 90, and 98 suggests that sparsity is partly a prompt-scale artifact, so alternative elicitation formats—open-ended numbers, natural-language quantifiers, or verbalized distributions—might be a complementary fix that the paper does not explore.
  • If API providers increasingly expose full logprobs, verbalization logprobs becomes a near-free add-on to existing verbalization prompts, which could make sampling-based confidence estimation less attractive in cost-sensitive deployment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper studies the sparsity of confidence estimates produced by prompted LLMs for classification. It reports that vanilla verbalization yields very few distinct confidence values, e.g., Qwen3-32B produces only eight unique values on SST-2 with more than half at 95%, and that 45-93% of confidence values concentrate in the top five values across four datasets and two models. It shows that the choice of linear versus stepwise interpolation when computing AUARC changes both scores and rankings, in one case moving consistency sampling from first to last. To address sparsity, the paper proposes verbalization logprobs, which computes an expected confidence from the token probabilities of the tens and units digits of the verbalized score, and reports a +2.3 AUARC improvement over vanilla verbalization at no additional inference cost. The main claims are supported by experiments with Qwen3-32B and, for the sparsity and interpolation findings, Claude 3.7 Sonnet.

Significance. If the descriptive findings hold, they are a useful and practically important contribution: they identify a measurement artifact in a widely used evaluation metric and quantify its effect. The paper ships no code, but the histograms, concentration statistics, and the replication across two models, four datasets, two confidence ranges, and reasoning settings make the sparsity and interpolation-reversal findings convincing. The proposed verbalization logprobs method is elegant and cost-effective, but its empirical validation is currently too thin to support the advertised gains; this is the main weakness.

major comments (3)
  1. [Section 4, Table 3] The headline improvement of verbalization logprobs (+2.3 AUARC over vanilla verbalization, and 'best' AUARC) is based on one model (Qwen3-32B) and on dataset-averaged scores, with no per-dataset breakdown, confidence intervals, or significance tests in the main text or appendix. The gap to the strongest baseline, verbalization sampling, is 0.736 vs 0.734, which is much smaller than the headline gain and could easily flip under bootstrap resampling. Appendix B reports Claude 3.7 Sonnet for the other methods but does not evaluate verbalization logprobs, so there is no second-model confirmation. Please add per-dataset tables, uncertainty estimates, and at least one additional model, or rephrase the contribution as 'comparable to verbalization sampling at lower cost' rather than 'best.'
  2. [Section 4, unnumbered equation and footnote 5] The proposed expectation treats the two digit positions as independent, using P(x_{i+1}=d) from the single actually generated continuation. The paper acknowledges this in footnote 5 as an approximation, but the magnitude of the resulting error is not quantified. Because the tens digit contributes ten times the units digit, even a small bias in the units term could shift the confidence value enough to affect threshold-based metrics at the scale of the reported 2.3-point difference. More fundamentally, the formula is defined only for exactly two digit tokens; confidence outputs such as '5' or '100' are not covered, and the paper does not state how such outputs were parsed in the experiments. Please specify the parsing rule and provide an empirical check of the approximation, for example by using the full next-token distribution available through vLLM to compute the exact conditional expectation for a subset of examples.
  3. [Section 4, Table 4 and surrounding paragraph] The correlation analysis is descriptive, not a theoretical justification for the method. A Spearman correlation of 0.89 between verbalization logprobs and vanilla verbalization shows that the two scores are related but does not explain why the weighted expectation is better calibrated or better for selective classification; the AUARC gap could be due to the finer resolution alone. The statement that 'verbalized digit distribution carries useful confidence signals beyond what token logprobs provides' needs direct evidence, such as per-dataset coverage-risk curves or calibration breakdowns comparing verbalization logprobs with token logprobs and verbalization sampling.
minor comments (6)
  1. [Section 3, Table 2] The notation '0.8081→50.683' is hard to parse; please add a table note explaining that the superscripts are ranks before and after switching interpolation, or split the columns into linear score, stepwise score, and ranks.
  2. [Section 4] The equation should be numbered and the notation x_i and x_{i+1} defined immediately before the equation; currently the variables are introduced only in the following prose.
  3. [Figure 1] The two shaded rejection regions are described in the caption, but the caption does not state which curve (stepwise or linear) the shaded area corresponds to; please clarify.
  4. [Appendix B] The tables for Claude 3.7 Sonnet do not include verbalization logprobs; this absence should be mentioned explicitly in Section 4 when the paper discusses generality, not left to the reader to infer.
  5. [Limitations] The Limitations section should mention the variable-length output parsing issue in addition to the single-token-per-digit tokenization assumption, since both constrain when the method is applicable.
  6. [Section 3] The paper recommends 'stepwise interpolation' but does not define it formally; please define it as the integral over the step function that keeps accuracy constant between consecutive distinct confidence thresholds.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are empirical and the proposed method is a fixed, parameter-free transformation evaluated against external baselines.

full rationale

The paper does not fit free parameters and then relabel them as predictions; verbalization logprobs is a fixed formula that weights verbalized digit tokens by their token probabilities. The main claims—sparsity of verbalized confidences, the effect of interpolation on AUARC rankings, and the relative performance of methods—are empirical observations on standard datasets with external baselines. There are no self-citations that carry argumentative weight, and no uniqueness theorem or prior-work premise is invoked to force the proposed method. The stated limitations (tokenization assumptions and the independence approximation in computing the expectation) are explicitly acknowledged in the manuscript and do not make the evaluation circular. The correlation analysis between verbalization logprobs and vanilla verbalization is descriptive rather than a derivation of the method's advantage. Therefore, no circular step can be identified, and the derivation chain is self-contained with respect to the paper's claims.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters are fitted; the method is a fixed transformation using digit weights 10 and 1. The assumptions are the tokenization of digits, the availability of logprobs for all digits, the independence approximation, and the normative choice of stepwise interpolation.

assumptions (3)
  • domain assumption Each digit of the verbalized confidence is a single token, so the tens and units digits can be identified and averaged over.
    Stated in Section 4 and Limitations: 'assumes that each digit is consistently tokenized as a single token'. If tokenizers split differently, the expectation formula does not apply.
  • domain assumption The marginal token probabilities at the tens and units positions, as returned by standard logprobs APIs, are sufficient to approximate the expected confidence value (treating the two positions as independent).
    The paper's equation in Section 4 uses P(x_i=d) and P(x_{i+1}=d) without conditioning; the independence approximation is acknowledged in a footnote. If dependence is strong, the estimate may be biased.
  • domain assumption Stepwise interpolation is the correct way to compute AUARC for sparse confidence outputs.
    Section 3 argues that linear interpolation artificially inflates scores because intermediate thresholds are unachievable. This is a methodological stance, not a formal theorem, and is load-bearing for the ranking reversal finding.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluation Pitfalls and Sparsity Limitations in LLM-based Confidence Estimates for Classification." pith.science (2026). https://pith.science/paper/CH7A2ZXY

@misc{pith2026260804899,
  author       = {Pith},
  title        = {Pith review of: Evaluation Pitfalls and Sparsity Limitations in LLM-based Confidence Estimates for Classification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CH7A2ZXY}},
  note         = {Machine review of arXiv:2608.04899}
}
read the original abstract

Confidence estimation is essential when LLMs are used for classification, indicating when predictions can be trusted. However, common approaches such as verbalization produce extremely sparse outputs. For instance, Qwen3-32B verbalizes only eight unique confidence values on SST-2, with over half being exactly 95%, a pattern we observe consistently across four datasets and two LLMs. Besides limiting practical utility, we show that this sparsity critically affects evaluation: the choice of interpolation in area under the accuracy-rejection curve (AUARC) dramatically alters rankings, with consistency sampling dropping from best to worst under stepwise versus linear interpolation. We advocate for standardizing stepwise interpolation for a fairer comparison. Under such a fair evaluation, we find that weighting verbalized digits by token probabilities, a method we term verbalization logprobs, addresses sparsity and achieves the best AUARC (+2.3 points over vanilla verbalization) without incurring additional inference cost.

Figures

Figures reproduced from arXiv: 2608.04899 by the authors.

Figure 1
Figure 1. Sparse confidences limit practical utility. (a) Histogram of vanilla verbalized confidences on SST-2 with Qwen3-32B, showing extreme sparsity with only eight unique values (0.6, 0.65, 0.7, 0.75, 0.85, 0.9, 0.95, 0.98). The shaded regions highlight two rejection strategies: rejecting 205 samples (lighter gray) achieves 97.9% accuracy, while rejecting 345 samples (darker gray) achieves 98.7% accuracy. (b) Accuracy-rej… view at source ↗
Figure 2
Figure 2. Sparse confidences for confidence scoring range 0–9. (a) Histogram of vanilla verbalized confidences on SST-2 with Qwen3-32B using the confidence range 0–9 (normalized to 0–1). Only seven unique values appear, with the majority concentrated at 8 9 and 9 9 . (b) Accuracy-rejection curve demonstrating limited threshold choices due to sparsity. • Consistency sampling (Xiong et al., 2024): The same prompt is executed mu… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 13 canonical work pages

  1. [1]

    Benchmarking Uncertainty Quantification Methods for Large Language Models with LM -Polygraph

    Vashurin, Roman and Fadeeva, Ekaterina and Vazhentsev, Artem and Rvanova, Lyudmila and Vasilev, Daniil and Tsvigun, Akim and Petrakov, Sergey and Xing, Rui and Sadallah, Abdelrahman and Grishchenkov, Kirill and Panchenko, Alexander and Baldwin, Timothy and Nakov, Preslav and Panov, Maxim and Shelmanov, Artem. Benchmarking Uncertainty Quantification Method...

    doi:10.1162/tacl_a_00737 2025

  2. [2]

    Beyond Semantic Entropy: Boosting LLM Uncertainty Quantification with Pairwise Semantic Similarity

    Nguyen, Dang and Payani, Ali and Mirzasoleiman, Baharan. Beyond Semantic Entropy: Boosting LLM Uncertainty Quantification with Pairwise Semantic Similarity. Findings of the Association for Computational Linguistics: ACL 2025. 2025. doi:10.18653/v1/2025.findings-acl.234

    doi:10.18653/v1/2025.findings-acl.234 2025

  3. [3]

    Thinking Out Loud: Do Reasoning Models Know When They ' re Right?

    Zeng, Qingcheng and Xuan, Weihao and Cui, Leyang and Voigt, Rob. Thinking Out Loud: Do Reasoning Models Know When They ' re Right?. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025. doi:10.18653/v1/2025.emnlp-main.73

    doi:10.18653/v1/2025.emnlp-main.73 2025

  4. [4]

    Seeing is Believing, but How Much? A Comprehensive Analysis of Verbalized Calibration in Vision-Language Models

    Xuan, Weihao and Zeng, Qingcheng and Qi, Heli and Wang, Junjue and Yokoya, Naoto. Seeing is Believing, but How Much? A Comprehensive Analysis of Verbalized Calibration in Vision-Language Models. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025. doi:10.18653/v1/2025.emnlp-main.74

    doi:10.18653/v1/2025.emnlp-main.74 2025

  5. [5]

    C heck E val: A reliable LLM -as-a-Judge framework for evaluating text generation using checklists

    Lee, Yukyung and Kim, JoongHoon and Kim, Jaehee and Cho, Hyowon and Kang, Jaewook and Kang, Pilsung and Kim, Najoung. C heck E val: A reliable LLM -as-a-Judge framework for evaluating text generation using checklists. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025. doi:10.18653/v1/2025.emnlp-main.796

    doi:10.18653/v1/2025.emnlp-main.796 2025

  6. [6]

    Liu, Gabrielle Kaili-May and Yona, Gal and Caciularu, Avi and Szpektor, Idan and Rudner, Tim G. J. and Cohan, Arman. M eta F aith: Faithful Natural Language Uncertainty Expression in LLM s. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025. doi:10.18653/v1/2025.emnlp-main.1505

    doi:10.18653/v1/2025.emnlp-main.1505 2025

  7. [7]

    Unconditional Truthfulness: Learning Unconditional Uncertainty of Large Language Models

    Vazhentsev, Artem and Fadeeva, Ekaterina and Xing, Rui and Kuzmin, Gleb and Lazichny, Ivan and Panchenko, Alexander and Nakov, Preslav and Baldwin, Timothy and Panov, Maxim and Shelmanov, Artem. Unconditional Truthfulness: Learning Unconditional Uncertainty of Large Language Models. Proceedings of the 2025 Conference on Empirical Methods in Natural Langua...

    doi:10.18653/v1/2025.emnlp-main.1807 2025

  8. [8]

    Uncertainty Quantification for In-Context Learning of Large Language Models

    Ling, Chen and Zhao, Xujiang and Zhang, Xuchao and Cheng, Wei and Liu, Yanchi and Sun, Yiyou and Oishi, Mika and Osaki, Takao and Matsuda, Katsushi and Ji, Jie and Bai, Guangji and Zhao, Liang and Chen, Haifeng. Uncertainty Quantification for In-Context Learning of Large Language Models. Proceedings of the 2024 Conference of the North American Chapter of ...

    doi:10.18653/v1/2024.naacl-long.184 2024

Show all 45 references
  1. [9]

    A Survey of Confidence Estimation and Calibration in Large Language Models

    Geng, Jiahui and Cai, Fengyu and Wang, Yuxia and Koeppl, Heinz and Nakov, Preslav and Gurevych, Iryna. A Survey of Confidence Estimation and Calibration in Large Language Models. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computatio...

    2024 doi

  2. [10]

    B ayesian Prompt Ensembles: Model Uncertainty Estimation for Black-Box Large Language Models

    Tonolini, Francesco and Aletras, Nikolaos and Massiah, Jordan and Kazai, Gabriella. B ayesian Prompt Ensembles: Model Uncertainty Estimation for Black-Box Large Language Models. Findings of the Association for Computational Linguistics: ACL 2024. 2024. doi:10.18653/v1/2024.fin...

    2024 doi

  3. [11]

    Calibrating the Confidence of Large Language Models by Eliciting Fidelity

    Zhang, Mozhi and Huang, Mianqiu and Shi, Rundong and Guo, Linsen and Peng, Chong and Yan, Peng and Zhou, Yaqian and Qiu, Xipeng. Calibrating the Confidence of Large Language Models by Eliciting Fidelity. Proceedings of the 2024 Conference on Empirical Methods in Natural Langua...

    2024 doi

  4. [12]

    Contextualized Sequence Likelihood: Enhanced Confidence Scores for Natural Language Generation

    Lin, Zhen and Trivedi, Shubhendu and Sun, Jimeng. Contextualized Sequence Likelihood: Enhanced Confidence Scores for Natural Language Generation. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024. doi:10.18653/v1/2024.emnlp-main.578

    2024 doi

  5. [13]

    Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models

    Duan, Jinhao and Cheng, Hao and Wang, Shiqi and Zavalny, Alex and Wang, Chenan and Xu, Renjing and Kailkhura, Bhavya and Xu, Kaidi. Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models. Proceedings of the 62nd An...

    2024 doi

  6. [14]

    Adaptation with Self-Evaluation to Improve Selective Prediction in LLM s

    Chen, Jiefeng and Yoon, Jinsung and Ebrahimi, Sayna and Arik, Sercan and Pfister, Tomas and Jha, Somesh. Adaptation with Self-Evaluation to Improve Selective Prediction in LLM s. Findings of the Association for Computational Linguistics: EMNLP 2023. 2023. doi:10.18653/v1/2023....

    2023 doi

  7. [15]

    Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback

    Tian, Katherine and Mitchell, Eric and Zhou, Allan and Sharma, Archit and Rafailov, Rafael and Yao, Huaxiu and Finn, Chelsea and Manning, Christopher. Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Fee...

    2023 doi

  8. [16]

    and Ng, Andrew and Potts, Christopher

    Socher, Richard and Perelygin, Alex and Wu, Jean and Chuang, Jason and Manning, Christopher D. and Ng, Andrew and Potts, Christopher. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. Proceedings of the 2013 Conference on Empirical Methods in Natur...

    2013

  9. [17]

    Xiong, Miao and Hu, Zhiyuan and Lu, Xinyang and Li, Yifei and Fu, Jie and He, Junxian and Hooi, Bryan , booktitle =. Can. 2024 , url =

    2024

  10. [18]

    Proceedings of the 36th International Conference on Neural Information Processing Systems , articleno =

    Kojima, Takeshi and Gu, Shixiang Shane and Reid, Machel and Matsuo, Yutaka and Iwasawa, Yusuke , title =. Proceedings of the 36th International Conference on Neural Information Processing Systems , articleno =. 2022 , isbn =

    2022

  11. [19]

    2025 , eprint=

    Deterministic or probabilistic? The psychology of LLMs as random number generators , author=. 2025 , eprint=

    2025

  12. [20]

    Transactions on Machine Learning Research , issn=

    Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models , author=. Transactions on Machine Learning Research , issn=. 2024 , url=

    2024

  13. [21]

    2024 , archivePrefix =

    Calibrating verbalized probabilities for large language models , author =. 2024 , archivePrefix =. 2410.06707 , url =

    2024 arXiv

  14. [22]

    IEEE Transactions on Software Engineering , year =

    Look Before You Leap: An Exploratory Study of Uncertainty Analysis for Large Language Models , author =. IEEE Transactions on Software Engineering , year =

  15. [23]

    The Eleventh International Conference on Learning Representations (ICLR 2023) , year=

    Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation , author=. The Eleventh International Conference on Learning Representations (ICLR 2023) , year=

    2023

  16. [24]

    2022 , eprint=

    Language Models (Mostly) Know What They Know , author=. 2022 , eprint=

    2022

  17. [25]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    arXiv

  18. [26]

    Journal of Machine Learning Research , year =

    Ran El-Yaniv and Yair Wiener , title =. Journal of Machine Learning Research , year =

  19. [27]

    Proceedings of the 38th International Conference on Neural Information Processing Systems , articleno =

    Nikitin, Alexander and Kossen, Jannik and Gal, Yarin and Marttinen, Pekka , title =. Proceedings of the 38th International Conference on Neural Information Processing Systems , articleno =. 2024 , isbn =

    2024

  20. [28]

    BingoGuard:

    Fan Yin and Philippe Laban and XIANGYU PENG and Yilun Zhou and Yixin Mao and Vaibhav Vats and Linnea Ross and Divyansh Agarwal and Caiming Xiong and Chien-Sheng Wu , booktitle=. BingoGuard:. 2025 , url=

    2025

  21. [29]

    arXiv preprint arXiv:2509.15174 , year=

    SMARTER: A Data-efficient Framework to Improve Toxicity Detection with Explanation via Self-augmenting Large Language Models , author=. arXiv preprint arXiv:2509.15174 , year=

    arXiv

  22. [30]

    arXiv e-prints , pages=

    Scaling Policy Compliance Assessment in Language Models with Policy Reasoning Traces , author=. arXiv e-prints , pages=

  23. [31]

    Journal of classification , volume=

    ROC and AUC with a binary predictor: a potentially misleading metric , author=. Journal of classification , volume=. 2020 , publisher=

    2020

  24. [32]

    Genome Biology , volume=

    Commonly used software tools produce conflicting and overly-optimistic AUPRC values , author=. Genome Biology , volume=. 2024 , publisher=

    2024

  25. [33]

    Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering , pages =

    Sun, Tao and Xu, Jian and Li, Yuanpeng and Yan, Zhao and Zhang, Ge and Xie, Lintao and Geng, Lu and Wang, Zheng and Chen, Yueyan and Lin, Qin and Duan, Wenbo and Sui, Kaixin and Zhu, Yuanshuo , title =. Proceedings of the 33rd ACM International Conference on the Foundations of...

    2025

  26. [34]

    and Zhang, Hao and Gonzalez, Joseph E

    Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Zhuang, Siyuan and Wu, Zhanghao and Zhuang, Yonghao and Lin, Zi and Li, Zhuohan and Li, Dacheng and Xing, Eric P. and Zhang, Hao and Gonzalez, Joseph E. and Stoica, Ion , title =. Proceedings of the 37th International Conf...

    2023

  27. [35]

    Proceedings of the third International Workshop on Machine Learning in Systems Biology , pages =

    Accuracy-Rejection Curves (ARCs) for Comparing Classification Methods with a Reject Option , author =. Proceedings of the third International Workshop on Machine Learning in Systems Biology , pages =. 2009 , editor =

    2009

  28. [36]

    Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining , year =

    Uncertainty quantification and confidence calibration in large language models: A survey , author =. Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining , year =

  29. [37]

    I Can't Believe It's Not Better: Failure Modes in the Age of Foundation Models

    Self-Evaluation Improves Selective Generation in Large Language Models , author =. Proceedings on "I Can't Believe It's Not Better: Failure Modes in the Age of Foundation Models" at NeurIPS 2023 Workshops , pages =. 2023 , editor =

    2023

  30. [38]

    arXiv preprint arXiv:2506.01734 , year=

    Benford's Curse: Tracing Digit Bias to Numerical Hallucination in LLMs , author=. arXiv preprint arXiv:2506.01734 , year=

  31. [39]

    Advances in Neural Information Processing Systems (NeurIPS 2015) , year =

    Character-level Convolutional Networks for Text Classification , author =. Advances in Neural Information Processing Systems (NeurIPS 2015) , year =

    2015

  32. [40]

    Reddy, Chandan K. and M. Shopping Queries Dataset: A Large-Scale. 2022 , archivePrefix =. 2206.06588 , url =

    2022 arXiv

  33. [41]

    , author=

    The meaning and use of the area under a receiver operating characteristic (ROC) curve. , author=. Radiology , volume=

  34. [42]

    arXiv preprint arXiv:2509.13813 , year=

    Geometric Uncertainty for Detecting and Correcting Hallucinations in LLMs , author=. arXiv preprint arXiv:2509.13813 , year=

  35. [43]

    M onte C arlo Temperature: a robust sampling strategy for LLM ' s uncertainty quantification methods

    Cecere, Nicola and Bacciu, Andrea and Fern \'a ndez-Tob \'i as, Ignacio and Mantrach, Amin. M onte C arlo Temperature: a robust sampling strategy for LLM ' s uncertainty quantification methods. Proceedings of the 5th Workshop on Trustworthy NLP (TrustNLP 2025). 2025. doi:10.18...

    2025 doi

  36. [44]

    Scikit-learn: Machine Learning in Python , year =

    Pedregosa, Fabian and Varoquaux, Ga\". Scikit-learn: Machine Learning in Python , year =. J. Mach. Learn. Res. , month = nov, pages =

  37. [45]

    Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models

    Kim, Seungone and Suk, Juyoung and Longpre, Shayne and Lin, Bill Yuchen and Shin, Jamin and Welleck, Sean and Neubig, Graham and Lee, Moontae and Lee, Kyungjae and Seo, Minjoon. Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models. Procee...

    2024 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.