{"work":{"id":"7721c193-e1ff-4cac-a0b6-ced585d69d3d","openalex_id":"https://openalex.org/W3180462067","doi":"10.48550/arxiv.2107.07511","arxiv_id":"2107.07511","raw_key":null,"title":"A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification","authors":null,"authors_text":"Anastasios N. Angelopoulos, Stephen Bates","year":2021,"venue":"cs.LG","abstract":"Black-box machine learning models are now routinely used in high-risk settings, like medical diagnostics, which demand uncertainty quantification to avoid consequential model failures. Conformal prediction is a user-friendly paradigm for creating statistically rigorous uncertainty sets/intervals for the predictions of such models. Critically, the sets are valid in a distribution-free sense: they possess explicit, non-asymptotic guarantees even without distributional assumptions or model assumptions. One can use conformal prediction with any pre-trained model, such as a neural network, to produce sets that are guaranteed to contain the ground truth with a user-specified probability, such as 90%. It is easy-to-understand, easy-to-use, and general, applying naturally to problems arising in the fields of computer vision, natural language processing, deep reinforcement learning, and so on.\n  This hands-on introduction is aimed to provide the reader a working understanding of conformal prediction and related distribution-free uncertainty quantification techniques with one self-contained document. We lead the reader through practical theory for and examples of conformal prediction and describe its extensions to complex machine learning tasks involving structured outputs, distribution shift, time-series, outliers, models that abstain, and more. Throughout, there are many explanatory illustrations, examples, and code samples in Python. With each code sample comes a Jupyter notebook implementing the method on a real-data example; the notebooks can be accessed and easily run using our codebase.","external_url":"https://arxiv.org/abs/2107.07511","cited_by_count":28,"metadata_source":"pith","metadata_fetched_at":"2026-08-05T02:28:24.338817+00:00","pith_arxiv_id":"2107.07511","created_at":"2026-05-08T18:13:53.724503+00:00","updated_at":"2026-08-05T02:28:24.338817+00:00","title_quality_ok":true,"display_title":"A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification","render_title":"A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification"},"hub":{"state":{"work_id":"7721c193-e1ff-4cac-a0b6-ced585d69d3d","tier":"super_hub","tier_reason":"100+ Pith inbound or 10,000+ external citations","pith_inbound_count":110,"external_cited_by_count":28,"distinct_field_count":22,"first_pith_cited_at":"2025-09-08T18:54:56+00:00","last_pith_cited_at":"2026-07-09T11:54:53+00:00","author_build_status":"needed","summary_status":"needed","contexts_status":"needed","graph_status":"needed","ask_index_status":"needed","reader_status":"not_needed","recognition_status":"not_needed","updated_at":"2026-08-22T19:49:26.888801+00:00","tier_text":"super_hub"},"tier":"super_hub","role_counts":[{"context_role":"background","n":6},{"context_role":"method","n":1}],"polarity_counts":[{"context_polarity":"background","n":6},{"context_polarity":"use_method","n":1}],"runs":{"ask_index":{"job_type":"ask_index","status":"succeeded","result":{"title":"A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification","claims":[{"claim_text":"Black-box machine learning models are now routinely used in high-risk settings, like medical diagnostics, which demand uncertainty quantification to avoid consequential model failures. Conformal prediction is a user-friendly paradigm for creating statistically rigorous uncertainty sets/intervals for the predictions of such models. Critically, the sets are valid in a distribution-free sense: they possess explicit, non-asymptotic guarantees even without distributional assumptions or model assumptions. One can use conformal prediction with any pre-trained model, such as a neural network, to produ","claim_type":"abstract","evidence_strength":"source_metadata"},{"claim_text":"M. Keesee, C. M. Ngwira, and H. Connor,A survey of uncertainty quantification in machine learning for space weather prediction, Geosciences 12(2022) 1, . [12]B. Kompa, J. Snoek, and A. L. Beam,Second opinion needed: communicating uncertainty in medical machine learning, npj Digital Medicine4(2021) 1, 4. https://doi.org/10.1038/s41746-020-00367-3. [13]T . J. Loftus, B. Shickel, M. M. Ruppert, J. A. Balch, T . Ozrazgat-Baslanti, P . J. Tighe, P . A. Efron, W . R. Hogan, P . Rashidi, G. R. Upchurch","claim_type":"background","confidence":0.9,"evidence_strength":"citation_context"},{"claim_text":"established methods including conformal prediction [4], ∗Equal contribution. E. Dietrich, H. Krasowski, and M. Arcak are with the University of California, Berkeley, USA. Email:{eadietri, krasowski, arcak}@berkeley.edu. This work was funded in part by the National Science Foundation under grant CNS-2111688. Elizabeth Dietrich was also supported by an NSF Graduate Research Fellowship. [11], [12], scenario optimization [13]-[16], and the holdout method [17]-[19]. Given the efficiency of PAC bounds","claim_type":"background","confidence":0.9,"evidence_strength":"citation_context"},{"claim_text":"67±.030.0 1.61±.080.60±.035.9 1.93±.040.71±.0110.5 2.10±.130.76±.010.0 1.78±.030.64±.030.0 2.91±.110.63±.0362.5 3.43±.120.71±.027.7 2.10±.12 work highlights the importance of structure-aware localization for UQ on graphs. An important future direction is to extend localized conformal prediction to dynamic graphs with evolving topologies. References [1] Anastasios N Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification.arXiv","claim_type":"background","confidence":0.9,"evidence_strength":"citation_context"},{"claim_text":"components: turn-level confidence estimation, adaptive early stopping, and final prediction 4 Preprint. Under review. Reasoning stepsamplimg Stage 1. Pre-Generation Knowledge Assessment Entity Extraction'Silas Hardy' ,'Lee Mantle'Silas Hardy : 258Lee Mantle : 180Frequency Pre-training CorpusIndex Engine !\"#$!\"#<&$%&'&(Retrieved docs[1]: Silas Hardywas an..[2]: ...… !): Lee Mantle was born on December 13, 1851.!*: Silas Hardy was born on January 24, 1827. CheckCo-occurrenceCount: 3Count: 0 LLM Ge","claim_type":"background","confidence":0.85,"evidence_strength":"citation_context"},{"claim_text":"Given a generative model for estimating the conditional density, PCP defines the non-conformity score of a data point (xi,y i)as si = min 1≤k≤K ∥yi − ˆyk i ∥, where{ ˆyi},K k=1 is a sample of the output generated from the estimated conditional density. Then the prediction region atx n+1 is obtained by generating a sample{ ˆyk n+1}K k=1 and form bCPCP(xn+1) = K[ k=1 {y:∥y− ˆyk n+1∥2 ≤s 1−α}, wheres 1−α is the⌈(1−α)(n 2 + 1)⌉-th smallest member of{s i, i∈I 2}. Note that each PCP prediction region ","claim_type":"background","confidence":0.8,"evidence_strength":"citation_context"},{"claim_text":"Algorithm 1:The CONFIDE Algorithm Input:Labeled training dataset(X,Y); modelM; significance levelϵ; test inputz; layerℓ; number of neighbors k; representation mode; PCA flag; distance metric Step 1: Extract Representations from Calibration Data foreachtraining example(x i,yi)do RunM(x i)to extractH i from layerℓ ifmode ==Flattenedthen hi←reshape(Hi,[1,seq_len·hidden_dim]) // Flatten full matrix else ifmode ==Attentionthen hi←Hi[CLS] // Use [CLS] token end ifˆyi =y i then Storeh i in pool for cla","claim_type":"method","confidence":0.75,"evidence_strength":"citation_context"}],"why_cited":"Pith tracks A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification because it crossed a citation-hub threshold. Current citing contexts most often use it as background evidence (6 contexts).","role_counts":[{"n":6,"context_role":"background"},{"n":1,"context_role":"method"}]},"error":null,"updated_at":"2026-07-04T08:16:48.047822+00:00"},"author_expand":{"job_type":"author_expand","status":"succeeded","result":{"authors_linked":[{"id":"686ce4da-52cd-4f79-904f-db25edb66d6e","orcid":null,"display_name":"Anastasios N. Angelopoulos"},{"id":"e42bdfcf-3cb9-422b-83d3-fa964abcff02","orcid":null,"display_name":"Stephen Bates"}]},"error":null,"updated_at":"2026-07-04T08:16:48.042357+00:00"},"context_extract":{"job_type":"context_extract","status":"succeeded","result":{"enqueued_papers":25},"error":null,"updated_at":"2026-05-14T18:29:38.292772+00:00"},"graph_features":{"job_type":"graph_features","status":"succeeded","result":{"co_cited":[{"title":"1984 , PAGES =","work_id":"263373f9-83d0-4280-a5ff-f1dde7abe956","shared_citers":3},{"title":"2005 , publisher=","work_id":"31470f85-2f7f-4117-905b-da888e9ae129","shared_citers":3},{"title":"2010 , note =","work_id":"aa219abe-2334-40b0-94c0-c7b070cfc332","shared_citers":3},{"title":"2022 , note =","work_id":"801a2b39-ffba-4cf3-916d-aae2847dab53","shared_citers":3},{"title":"ACM Transactions on Graphics , author =","work_id":"055301d6-5966-48c7-b3d7-390a9f3cf3d2","shared_citers":3},{"title":"Alimentary Pharmacology & Therapeutics , author =","work_id":"6f9c4d6b-d533-4cb2-a8d5-c8c3cb46283d","shared_citers":3},{"title":"American Economic Review , author =","work_id":"71ac31be-a75a-4259-82b0-a44054ecff86","shared_citers":3},{"title":"American Political Science Review , author =","work_id":"f8de7511-4535-4f3f-b732-ce6283f992c6","shared_citers":3},{"title":"and Peyré, G","work_id":"c81e09e5-3637-4a6e-809f-7044fd147917","shared_citers":3},{"title":"and Zha, Hongyuan , editor =","work_id":"63d53723-ad38-41d8-8e5d-04c3af2c001f","shared_citers":3},{"title":"Annals of Operations Research , author =","work_id":"3ff4b138-bec0-478c-b619-39a4719a7cc9","shared_citers":3},{"title":"Annual Review of Statistics and Its Application , author =","work_id":"b7c9cd54-2a1d-4682-bb33-064108e269c7","shared_citers":3},{"title":"Applied Statistics , author =","work_id":"ffc828ac-e8b5-4357-9663-48037b0040fc","shared_citers":3},{"title":"arXiv:1905.11027 [cs, stat] , author =","work_id":"f430bbee-bbb1-445d-9f01-f12fb6a34785","shared_citers":3},{"title":"arXiv preprint arXiv:2506.22405 , year=","work_id":"21677228-f8df-4248-a5d4-4fd577355cdf","shared_citers":3},{"title":"Assessment , author =","work_id":"ab43cfcd-92d7-46a0-bc57-48185e0e8c8b","shared_citers":3},{"title":"Barycenters in the","work_id":"b7786cf0-535d-4d9c-815e-0f573b781c5d","shared_citers":3},{"title":"Bernoulli , author =","work_id":"7d58031f-a5b7-4e95-a0d7-543257d008fa","shared_citers":3},{"title":"Bernoulli , author =","work_id":"5c713db7-322e-4920-82e8-9491c2175ab8","shared_citers":3},{"title":"Biometrika , author =","work_id":"e665f17a-c597-47e7-870f-ad6c0229a3a2","shared_citers":3},{"title":"Biometrika , author =","work_id":"86c39f09-cd03-4fec-ac72-c54dd762844f","shared_citers":3},{"title":"Cambridge University Press","work_id":"6eef43d6-d9f7-42da-90a7-d535e797f287","shared_citers":3},{"title":"Canadian Journal of Mathematics , author =","work_id":"4e436a1f-729f-4d96-b1cc-84da003a311b","shared_citers":3},{"title":"Communications in Partial Differential Equations , author =","work_id":"cfa1382e-7d9f-4f5a-99f0-029b641a31a1","shared_citers":3}],"time_series":[{"n":34,"year":2026}],"dependency_candidates":[]},"error":null,"updated_at":"2026-05-14T18:29:29.680420+00:00"},"identity_refresh":{"job_type":"identity_refresh","status":"succeeded","result":{"items":[{"title":"Qwen3 Technical Report","outcome":"unchanged","work_id":"25a4e30c-1232-48e7-9925-02fa12ba7c9e","resolver":"local_arxiv","confidence":0.98,"old_work_id":"25a4e30c-1232-48e7-9925-02fa12ba7c9e"}],"counts":{"fixed":0,"merged":0,"unchanged":1,"quarantined":0,"needs_external_resolution":0},"errors":[],"attempted":1},"error":null,"updated_at":"2026-05-14T18:29:34.300992+00:00"},"role_polarity":{"job_type":"role_polarity","status":"succeeded","result":{"title":"A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification","claims":[{"claim_text":"Black-box machine learning models are now routinely used in high-risk settings, like medical diagnostics, which demand uncertainty quantification to avoid consequential model failures. Conformal prediction is a user-friendly paradigm for creating statistically rigorous uncertainty sets/intervals for the predictions of such models. Critically, the sets are valid in a distribution-free sense: they possess explicit, non-asymptotic guarantees even without distributional assumptions or model assumptions. One can use conformal prediction with any pre-trained model, such as a neural network, to produ","claim_type":"abstract","evidence_strength":"source_metadata"},{"claim_text":"M. Keesee, C. M. Ngwira, and H. Connor,A survey of uncertainty quantification in machine learning for space weather prediction, Geosciences 12(2022) 1, . [12]B. Kompa, J. Snoek, and A. L. Beam,Second opinion needed: communicating uncertainty in medical machine learning, npj Digital Medicine4(2021) 1, 4. https://doi.org/10.1038/s41746-020-00367-3. [13]T . J. Loftus, B. Shickel, M. M. Ruppert, J. A. Balch, T . Ozrazgat-Baslanti, P . J. Tighe, P . A. Efron, W . R. Hogan, P . Rashidi, G. R. Upchurch","claim_type":"background","confidence":0.9,"evidence_strength":"citation_context"},{"claim_text":"established methods including conformal prediction [4], ∗Equal contribution. E. Dietrich, H. Krasowski, and M. Arcak are with the University of California, Berkeley, USA. Email:{eadietri, krasowski, arcak}@berkeley.edu. This work was funded in part by the National Science Foundation under grant CNS-2111688. Elizabeth Dietrich was also supported by an NSF Graduate Research Fellowship. [11], [12], scenario optimization [13]-[16], and the holdout method [17]-[19]. Given the efficiency of PAC bounds","claim_type":"background","confidence":0.9,"evidence_strength":"citation_context"},{"claim_text":"67±.030.0 1.61±.080.60±.035.9 1.93±.040.71±.0110.5 2.10±.130.76±.010.0 1.78±.030.64±.030.0 2.91±.110.63±.0362.5 3.43±.120.71±.027.7 2.10±.12 work highlights the importance of structure-aware localization for UQ on graphs. An important future direction is to extend localized conformal prediction to dynamic graphs with evolving topologies. References [1] Anastasios N Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification.arXiv","claim_type":"background","confidence":0.9,"evidence_strength":"citation_context"},{"claim_text":"components: turn-level confidence estimation, adaptive early stopping, and final prediction 4 Preprint. Under review. Reasoning stepsamplimg Stage 1. Pre-Generation Knowledge Assessment Entity Extraction'Silas Hardy' ,'Lee Mantle'Silas Hardy : 258Lee Mantle : 180Frequency Pre-training CorpusIndex Engine !\"#$!\"#<&$%&'&(Retrieved docs[1]: Silas Hardywas an..[2]: ...… !): Lee Mantle was born on December 13, 1851.!*: Silas Hardy was born on January 24, 1827. CheckCo-occurrenceCount: 3Count: 0 LLM Ge","claim_type":"background","confidence":0.85,"evidence_strength":"citation_context"},{"claim_text":"Given a generative model for estimating the conditional density, PCP defines the non-conformity score of a data point (xi,y i)as si = min 1≤k≤K ∥yi − ˆyk i ∥, where{ ˆyi},K k=1 is a sample of the output generated from the estimated conditional density. Then the prediction region atx n+1 is obtained by generating a sample{ ˆyk n+1}K k=1 and form bCPCP(xn+1) = K[ k=1 {y:∥y− ˆyk n+1∥2 ≤s 1−α}, wheres 1−α is the⌈(1−α)(n 2 + 1)⌉-th smallest member of{s i, i∈I 2}. Note that each PCP prediction region ","claim_type":"background","confidence":0.8,"evidence_strength":"citation_context"},{"claim_text":"Algorithm 1:The CONFIDE Algorithm Input:Labeled training dataset(X,Y); modelM; significance levelϵ; test inputz; layerℓ; number of neighbors k; representation mode; PCA flag; distance metric Step 1: Extract Representations from Calibration Data foreachtraining example(x i,yi)do RunM(x i)to extractH i from layerℓ ifmode ==Flattenedthen hi←reshape(Hi,[1,seq_len·hidden_dim]) // Flatten full matrix else ifmode ==Attentionthen hi←Hi[CLS] // Use [CLS] token end ifˆyi =y i then Storeh i in pool for cla","claim_type":"method","confidence":0.75,"evidence_strength":"citation_context"}],"why_cited":"Pith tracks A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification because it crossed a citation-hub threshold. Current citing contexts most often use it as background evidence (6 contexts).","role_counts":[{"n":6,"context_role":"background"},{"n":1,"context_role":"method"}]},"error":null,"updated_at":"2026-07-04T08:16:48.045309+00:00"},"summary_claims":{"job_type":"summary_claims","status":"succeeded","result":{"title":"A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification","claims":[{"claim_text":"Black-box machine learning models are now routinely used in high-risk settings, like medical diagnostics, which demand uncertainty quantification to avoid consequential model failures. Conformal prediction is a user-friendly paradigm for creating statistically rigorous uncertainty sets/intervals for the predictions of such models. Critically, the sets are valid in a distribution-free sense: they possess explicit, non-asymptotic guarantees even without distributional assumptions or model assumptions. One can use conformal prediction with any pre-trained model, such as a neural network, to produ","claim_type":"abstract","evidence_strength":"source_metadata"}],"why_cited":"Pith tracks A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification because it crossed a citation-hub threshold.","role_counts":[]},"error":null,"updated_at":"2026-05-14T18:29:51.240978+00:00"}},"summary":{"title":"A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification","claims":[{"claim_text":"Black-box machine learning models are now routinely used in high-risk settings, like medical diagnostics, which demand uncertainty quantification to avoid consequential model failures. Conformal prediction is a user-friendly paradigm for creating statistically rigorous uncertainty sets/intervals for the predictions of such models. Critically, the sets are valid in a distribution-free sense: they possess explicit, non-asymptotic guarantees even without distributional assumptions or model assumptions. One can use conformal prediction with any pre-trained model, such as a neural network, to produ","claim_type":"abstract","evidence_strength":"source_metadata"}],"why_cited":"Pith tracks A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification because it crossed a citation-hub threshold.","role_counts":[]},"graph":{"co_cited":[{"title":"1984 , PAGES =","work_id":"263373f9-83d0-4280-a5ff-f1dde7abe956","shared_citers":3},{"title":"2005 , publisher=","work_id":"31470f85-2f7f-4117-905b-da888e9ae129","shared_citers":3},{"title":"2010 , note =","work_id":"aa219abe-2334-40b0-94c0-c7b070cfc332","shared_citers":3},{"title":"2022 , note =","work_id":"801a2b39-ffba-4cf3-916d-aae2847dab53","shared_citers":3},{"title":"ACM Transactions on Graphics , author =","work_id":"055301d6-5966-48c7-b3d7-390a9f3cf3d2","shared_citers":3},{"title":"Alimentary Pharmacology & Therapeutics , author =","work_id":"6f9c4d6b-d533-4cb2-a8d5-c8c3cb46283d","shared_citers":3},{"title":"American Economic Review , author =","work_id":"71ac31be-a75a-4259-82b0-a44054ecff86","shared_citers":3},{"title":"American Political Science Review , author =","work_id":"f8de7511-4535-4f3f-b732-ce6283f992c6","shared_citers":3},{"title":"and Peyré, G","work_id":"c81e09e5-3637-4a6e-809f-7044fd147917","shared_citers":3},{"title":"and Zha, Hongyuan , editor =","work_id":"63d53723-ad38-41d8-8e5d-04c3af2c001f","shared_citers":3},{"title":"Annals of Operations Research , author =","work_id":"3ff4b138-bec0-478c-b619-39a4719a7cc9","shared_citers":3},{"title":"Annual Review of Statistics and Its Application , author =","work_id":"b7c9cd54-2a1d-4682-bb33-064108e269c7","shared_citers":3},{"title":"Applied Statistics , author =","work_id":"ffc828ac-e8b5-4357-9663-48037b0040fc","shared_citers":3},{"title":"arXiv:1905.11027 [cs, stat] , author =","work_id":"f430bbee-bbb1-445d-9f01-f12fb6a34785","shared_citers":3},{"title":"arXiv preprint arXiv:2506.22405 , year=","work_id":"21677228-f8df-4248-a5d4-4fd577355cdf","shared_citers":3},{"title":"Assessment , author =","work_id":"ab43cfcd-92d7-46a0-bc57-48185e0e8c8b","shared_citers":3},{"title":"Barycenters in the","work_id":"b7786cf0-535d-4d9c-815e-0f573b781c5d","shared_citers":3},{"title":"Bernoulli , author =","work_id":"7d58031f-a5b7-4e95-a0d7-543257d008fa","shared_citers":3},{"title":"Bernoulli , author =","work_id":"5c713db7-322e-4920-82e8-9491c2175ab8","shared_citers":3},{"title":"Biometrika , author =","work_id":"e665f17a-c597-47e7-870f-ad6c0229a3a2","shared_citers":3},{"title":"Biometrika , author =","work_id":"86c39f09-cd03-4fec-ac72-c54dd762844f","shared_citers":3},{"title":"Cambridge University Press","work_id":"6eef43d6-d9f7-42da-90a7-d535e797f287","shared_citers":3},{"title":"Canadian Journal of Mathematics , author =","work_id":"4e436a1f-729f-4d96-b1cc-84da003a311b","shared_citers":3},{"title":"Communications in Partial Differential Equations , author =","work_id":"cfa1382e-7d9f-4f5a-99f0-029b641a31a1","shared_citers":3}],"time_series":[{"n":34,"year":2026}],"dependency_candidates":[]},"authors":[{"id":"686ce4da-52cd-4f79-904f-db25edb66d6e","orcid":null,"display_name":"Anastasios N. Angelopoulos","source":"manual","import_confidence":0.72},{"id":"e42bdfcf-3cb9-422b-83d3-fa964abcff02","orcid":null,"display_name":"Stephen Bates","source":"manual","import_confidence":0.72}]}}