Pith. sign in

REVIEW 3 cited by

Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysis

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2210.04714 v2 pith:PJSA6RHB submitted 2022-10-10 cs.CL cs.LGstat.ML

Yuxin Xiao , Paul Pu Liang , Umang Bhatt , Willie Neiswanger , Ruslan Salakhutdinov , Louis-Philippe Morency This is my paper · ORCID
classification cs.CLcs.LGstat.ML
keywords pipelinechoicelanguagepredictiontasksuncertaintyanalysisconsiderations
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Pre-trained language models (PLMs) have gained increasing popularity due to their compelling prediction performance in diverse natural language processing (NLP) tasks. When formulating a PLM-based prediction pipeline for NLP tasks, it is also crucial for the pipeline to minimize the calibration error, especially in safety-critical applications. That is, the pipeline should reliably indicate when we can trust its predictions. In particular, there are various considerations behind the pipeline: (1) the choice and (2) the size of PLM, (3) the choice of uncertainty quantifier, (4) the choice of fine-tuning loss, and many more. Although prior work has looked into some of these considerations, they usually draw conclusions based on a limited scope of empirical studies. There still lacks a holistic analysis on how to compose a well-calibrated PLM-based prediction pipeline. To fill this void, we compare a wide range of popular options for each consideration based on three prevalent NLP classification tasks and the setting of domain shift. In response, we recommend the following: (1) use ELECTRA for PLM encoding, (2) use larger PLMs if possible, (3) use Temp Scaling as the uncertainty quantifier, and (4) use Focal Loss for fine-tuning.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centered

    cs.CL 2025-06 conditional novelty 5.0 of 10

    LLM uncertainty quantification should be judged by whether it improves real human decisions, not by calibration scores on trivia benchmarks.

  2. Avoid Forgetting by Preserving Global Knowledge Gradients in Federated Learning with Non-IID Data

    cs.LG 2025-05 conditional novelty 5.0 of 10

    FedProj combines client-side gradient projection onto a global-knowledge loss with server-side ensemble distillation and outperforms existing federated learning methods on non-IID image and NLP benchmarks.

  3. Evaluating Uncertainty and Quality of Visual Language Action-enabled Robots

    cs.SE 2025-07 conditional novelty 4.0 of 10

    Across three VLA models and four simulated manipulation tasks, motion-instability and goal-distance metrics correlate with expert-rated execution quality, showing that binary success rates hide large quality differences.

Pith tools