REVIEW 20 cited by
Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control
read the original abstract
We introduce a framework for calibrating machine learning models so that their predictions satisfy explicit, finite-sample statistical guarantees. Our calibration algorithms work with any underlying model and (unknown) data-generating distribution and do not require model refitting. The framework addresses, among other examples, false discovery rate control in multi-label classification, intersection-over-union control in instance segmentation, and the simultaneous control of the type-1 error of outlier detection and confidence set coverage in classification or regression. Our main insight is to reframe the risk-control problem as multiple hypothesis testing, enabling techniques and mathematical arguments different from those in the previous literature. We use the framework to provide new calibration methods for several core machine learning tasks, with detailed worked examples in computer vision and tabular medical data.
Forward citations
Cited by 20 Pith papers
-
The Statistical Cost of Adaptation in Multi-Source Transfer Learning
Multi-source transfer learning incurs an intrinsic adaptation cost that can exceed one, with phase transitions separating regimes where bias-agnostic estimators match oracle performance from those where they cannot.
-
Conformal Risk-Averse Decision Making with Action Conditional Guarantee
Action-conditional conformal prediction sets provide per-action safety guarantees for risk-averse policies that optimize conditional value-at-risk through pinball-loss minimization.
-
EST-PRM: Stress-Testing Process Reward Models Before They Become Load-Bearing
EST-PRM stress-tests five PRM models on 4,687 reasoning chains from MATH-500, GSM8K, and PRMBench using three label-preserving transformations and reports model-specific vulnerability patterns.
-
BOKBO (Best of K Bad Options): Calibrated Abstention for VLA Policies
BOKBO is the first conformal abstention method for K-sample VLA policies that supplies finite-sample distribution-free guarantees on executed violation rates, with global and Mondrian per-task variants.
-
How Many Iterations to Jailbreak? Dynamic Budget Allocation for Multi-Turn LLM Evaluation
DAPRO provides the first dynamic, theoretically guaranteed way to allocate interaction budgets across test cases for bounding time-to-event in multi-turn LLM evaluations, achieving tighter coverage than static conform...
-
Risk-Controlled Post-Processing of Decision Policies
Risk-controlled post-processing yields a threshold-structured policy that follows the baseline except where an oracle fallback sharply reduces conditional violation risk, achieving O(log n/n) expected excess risk in i...
-
VLM Judges Can Rank but Cannot Score: Task-Dependent Uncertainty in Multimodal Evaluation
VLM judges exhibit task-dependent uncertainty in their scores, with conformal prediction revealing wide intervals for complex tasks and a decoupling between good ranking performance and poor absolute scoring reliability.
-
Decomposition-Based Modular Conformal Prediction for Two-Stage Modeling
A decomposition-based modular conformal prediction method for two-stage models with FWER-controlled stage-wise scaling and adaptive extension for non-stationary data.
-
How Many Iterations to Jailbreak? Dynamic Budget Allocation for Multi-Turn LLM Evaluation
DAPRO dynamically allocates multi-turn LLM evaluation budget to produce valid, tighter lower bounds on iterations-to-event without the conditional-independence assumption of prior conformal survival methods.
-
Two Axes of LLM Abstention: Answer Correctness and Question Answerability
Answer correctness and question answerability are separate axes: ordinary confidence tracks the first while hidden probes track the second, and a factorized dual-threshold policy certifies both risk budgets at higher ...
-
What Predicts Correctness in Text-to-SQL? A Selective-Prediction Study
On hard multi-table text-to-SQL, verification-based LLM judges beat self-consistency and log-probability for predicting execution correctness, and fine-tuned verifiers fail to transfer across schemas.
-
Budgeted Act-or-Defer Multi-Agent LLM Deliberation with Local Reliability Bounds
A kNN lower-confidence-bound approach for act-or-defer decisions in multi-agent LLM debates respects user-declared wrong-action budgets while achieving high automation rates on benchmarks.
-
Uncertainty Quantification for Computer-Use Agents: A Benchmark across Vision-Language Models and GUI Grounding Datasets
Argus benchmark shows UQ method rankings for GUI grounding agents are stable within models across datasets but degrade across model classes and to closed-source vendors.
-
A Joint Finite-Sample Certificate for Adaptive Selective Conformal Risk Control
A joint finite-sample certificate for adaptive selective conformal risk control that treats selected risk as a ratio and couples empirical-Bernstein, Clopper-Pearson, and closeness bounds.
-
Co-GLANCE: Uncertainty-Aware Active Perception for Heterogeneous Robot Teaming
Co-GLANCE distills vision-language models into an end-to-end onboard model for occlusion segmentation and robot allocation, using conformal prediction plus selective abstention to trigger active perception and achieve...
-
Empirical Bayes Conformal Prediction for Vision and Language Models
Empirical Bayes conformal prediction converts score variability into r-value nonconformity scores that preserve target coverage while reducing inclusion of high-variance false candidates in image classification, CLIP ...
-
BalanceRAG: Joint Risk Calibration for Cascaded Retrieval-Augmented Generation
BalanceRAG uses sequential graphical testing on a 2D lattice of threshold pairs to certify safe operating points that meet target risk levels in cascaded RAG while increasing coverage.
-
Unsupervised Confidence Calibration for Reasoning LLMs from a Single Generation
Unsupervised single-generation confidence calibration for reasoning LLMs via offline self-consistency proxy distillation outperforms baselines on math and QA tasks and improves selective prediction.
-
Trust Functions: Near-Lossless Weak-to-Strong Generalization by Learning When to Trust the Weak Teacher
Trust functions filter unreliable weak labels to enable near-lossless weak-to-strong generalization and iterative chaining.
-
A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification
Pith review generated a malformed one-line summary.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.