Optimized Conformal Selection: Powerful Selective Inference After Conformity Score Optimization
Reviewed by Pithpith:PPB6MGROopen to challenge →
read the original abstract
Model selection/optimization in conformal inference is challenging, since it may break the exchangeability between labeled and unlabeled data. We study this problem in the context of conformal selection, which uses conformal p-values to select ``interesting'' instances with large unobserved labels from a pool of unlabeled data, while controlling the FDR in finite sample. For validity, existing solutions require the model choice to be independent of the data used to construct the p-values and calibrate the selection set. However, when presented with many model choices and limited labeled data, it is desirable to (i) select the best model in a data-driven manner, and (ii) mitigate power loss due to sample splitting. This paper presents OptCS, a general framework that allows valid statistical testing (selection) after flexible data-driven model optimization. We introduce general conditions under which OptCS constructs valid conformal p-values despite substantial data reuse and handles complex p-value dependencies to maintain finite-sample FDR control via a novel multiple testing procedure. We instantiate this general recipe to propose three FDR-controlling procedures, each optimizing the models differently: (i) selecting the most powerful one among multiple pre-trained candidate models, (ii) using all data for model fitting without sample splitting, and (iii) combining full-sample model fitting and selection. We demonstrate the efficacy of our methods via simulation studies and real applications in drug discovery and alignment of large language models in radiology report generation.
This paper has not been read by Pith yet.
Forward citations
Cited by 7 Pith papers
-
Full Conformal Prediction under Stochastic Non-Conformity Measure
The relaxation to permutation invariance in distribution is shown to be insufficient for full conformal prediction validity under stochastic non-conformity measures, and Conditional Independence & Permutation Invarian...
-
Null-Calibrated Conformal Selection via Target-Membership Scores
Introduces target-membership scores and Null-Calibrated Conformal Selection (NCCS) for finite-sample valid FDR control on non-mean-monotone targets.
-
Everywhere Valid Bounds on False Discovery Proportions in Conformal Inference
Derives simultaneous high-probability upper bounds on realized FDP for conformal p-values that hold for arbitrary post-hoc thresholds via an envelope on the null empirical distribution function.
-
Beyond Fixed False Discovery Rates: Post-Hoc Conformal Selection with E-Variables
Post-hoc conformal selection creates a path of selection sets with estimated false discovery proportions, enabling data-driven adaptive FDR control with average reliability guarantees via e-variables and e-BH.
-
Everywhere Valid Bounds on False Discovery Proportions in Conformal Inference
Derives simultaneous finite-sample distribution-free upper bounds on false discovery proportions for conformal p-values that hold for every possible rejection threshold.
-
Feedback-Enhanced Online Multiple Testing with Applications to Conformal Selection
GAIF dynamically adjusts testing thresholds with feedback for finite-sample FDR control in sequential settings and extends to conformal selection via feedback-driven model selection.
-
Structure-Adaptive Conformal Inference for Large-Scale Out-of-Distribution Testing
Introduces SCQ and P-TAMS for structure-adaptive conformal inference under pairwise exchangeability, claiming finite-sample FDR control for large-scale OOD testing.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.