Recognition: unknown
Conformal prediction after data-dependent model selection
read the original abstract
Given a family of pretrained models and a hold-out set, how can we construct a valid conformal prediction set while selecting a model that minimizes the width of the set? If we use the same hold-out data set both to select a model (the model that yields the smallest conformal prediction sets) and then to construct a conformal prediction set based on that selected model, we suffer a loss of coverage due to selection bias. Alternatively, we could further split the data to perform selection and calibration separately, but this comes at a steep cost if the size of the dataset is limited. In this paper, we address the challenge of constructing a valid prediction set after data-dependent model selection -- commonly, selecting the model that minimizes the width of the resulting prediction sets. Our novel methods can be implemented efficiently and admit finite-sample validity guarantees without invoking additional sample-splitting. We show that our methods yield prediction sets with asymptotically optimal width under certain notion of regularity for the model class. The improvement in the width of the prediction sets constructed by our methods are further demonstrated through applications to synthetic datasets in various settings and a real data example.
This paper has not been read by Pith yet.
Forward citations
Cited by 7 Pith papers
-
Online Localized Conformal Prediction
OLCP combines online adaptation with covariate-based localization to achieve valid long-run coverage and narrower intervals than global baselines in non-exchangeable settings.
-
A Unified Theory of Conditional Coverage in Conformal Prediction with Applications
A unified framework derives non-asymptotic bounds on conditional miscoverage in conformal prediction via pointwise and L_p routes and gives a common view of existing methods.
-
Learning Polyhedral Conformal Sets for Robust Optimization
Learns polyhedral uncertainty sets for robust optimization using a decision-aware conformal prediction approach that minimizes robust loss while ensuring statistical coverage.
-
Learning Polyhedral Conformal Sets for Robust Optimization
A new conformal framework learns polyhedral uncertainty sets tailored to robust optimization objectives, minimizing decision loss while preserving coverage via calibration and independent re-calibration.
-
Online Localized Conformal Prediction
OLCP and OLCP-Hedge achieve long-run valid coverage in non-exchangeable online settings with narrower prediction sets by localizing conformal prediction to covariates and selecting bandwidth via online convex optimization.
-
Conformalized Percentile Interval: Finite Sample Validity and Improved Conditional Performance
A PIT-calibrated percentile interval method delivers finite-sample marginal coverage, asymptotic conditional coverage, and shorter intervals than prior conformal approaches.
-
Stable Localized Conformal Prediction via Transduction
StCP leverages transfer learning to stabilize the size of conformal prediction sets without additional target labels.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.