REVIEW 1 cited by
Schroedinger's Threshold: When the AUC doesn't predict Accuracy
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
The Area Under Curve measure (AUC) seems apt to evaluate and compare diverse models, possibly without calibration. An important example of AUC application is the evaluation and benchmarking of models that predict faithfulness of generated text. But we show that the AUC yields an academic and optimistic notion of accuracy that can misalign with the actual accuracy observed in application, yielding significant changes in benchmark rankings. To paint a more realistic picture of downstream model performance (and prepare a model for actual application), we explore different calibration modes, testing calibration data and method.
Forward citations
Cited by 1 Pith paper
-
How to Choose a Threshold for an Evaluation Metric for Large Language Models
A recipe that maps stakeholder risk tolerance to a confidence level and then uses statistical methods, with conformal prediction performing best, to set thresholds on LLM evaluation metrics.
Discussion (0). Continue with ORCID to comment.