REVIEW 5 cited by
Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Uncertainty quantification in Large Language Models (LLMs) is crucial for applications where safety and reliability are important. In particular, uncertainty can be used to improve the trustworthiness of LLMs by detecting factually incorrect model responses, commonly called hallucinations. Critically, one should seek to capture the model's semantic uncertainty, i.e., the uncertainty over the meanings of LLM outputs, rather than uncertainty over lexical or syntactic variations that do not affect answer correctness. To address this problem, we propose Kernel Language Entropy (KLE), a novel method for uncertainty estimation in white- and black-box LLMs. KLE defines positive semidefinite unit trace kernels to encode the semantic similarities of LLM outputs and quantifies uncertainty using the von Neumann entropy. It considers pairwise semantic dependencies between answers (or semantic clusters), providing more fine-grained uncertainty estimates than previous methods based on hard clustering of answers. We theoretically prove that KLE generalizes the previous state-of-the-art method called semantic entropy and empirically demonstrate that it improves uncertainty quantification performance across multiple natural language generation datasets and LLM architectures.
Forward citations
Cited by 5 Pith papers
-
SAVAA: Mitigating Hallucinations in LVLMs via Step-wise Adaptive Visual Attention Amplification
Adaptively scaling visual attention boosting per token, guided by a combined entropy-and-visual-grounding risk score, reduces hallucinations in LVLMs more than fixed boosting.
-
Beyond Semantic Entropy: Boosting LLM Uncertainty Quantification with Pairwise Semantic Similarity
SNNE estimates LLM uncertainty from pairwise semantic similarities of sampled answers using a log-sum-exp aggregation, and it generalizes semantic entropy as a special case.
-
Seeing is Believing, but How Much? A Comprehensive Analysis of Verbalized Calibration in Vision-Language Models
Current vision-language models are largely miscalibrated when they verbalize confidence, visual reasoning models such as o3 and o4-mini are better calibrated, and Visual Confidence-Aware Prompting reduces ECE on IsoBench.
-
Improving the Calibration of Confidence Scores in Text Generation Using the Output Distribution's Characteristics
Two probability-only confidence metrics, a top-to-kth beam ratio and a tail-thinness score, improve quality correlation for BART and Flan-T5 on several summarization, translation, and QA datasets.
-
Towards Harmonized Uncertainty Estimation for Large Language Models
CUE combines a supervised correctness classifier with existing LLM uncertainty scores to improve indication, balance, and calibration, reporting AUROC and ECE gains across models and datasets.
Discussion (0). Continue with ORCID to comment.