Pitfalls of Conformal Predictions for Medical Image Classification

Hendrik Mehrtens; Tabea Bucher; Titus J. Brinker

Pitfalls of Conformal Predictions for Medical Image Classification

T0 review · 0 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Conformal prediction guarantees vanish under medical data shifts.

desk verdict A sound, honest guide to known conformal prediction pitfalls, illustrated on medical datasets; no new theory, but useful and deserving a proper review. read the letter →

arxiv 2506.18162 v1 pith:HFN5NQVW submitted 2025-06-22 cs.LG cs.CV

Hendrik Mehrtens , Tabea Bucher , Titus J. Brinker This is my paper · ORCID

classification cs.LGcs.CV

keywords conformalpredictionuncertaintyestimationmedicalimageclassificationcoverageguaranteesdistributionshiftselectivedermatologyhistopathology

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that conformal prediction, while it delivers its advertised coverage guarantee on in-distribution test sets, does not deliver what medical deployment requires: coverage for a specific patient, class, or subgroup, or coverage after the input or label distribution shifts. On two medical imaging benchmarks—dermatology (HAM10K, seven classes) and histopathology (CAMELYON17, binary tumor detection)—the authors show that marginal coverage holds on test data, but individual classes such as melanoma can be under-covered while frequent classes are over-covered. They show the guarantee breaks under domain shift and label shift, and that recalibration on new representative domain data restores it only if fresh calibration data can be gathered. They conclude that conformal predictions should not be used to select high-accuracy predictions, and that with few classes the prediction sets are often too uninformative to be clinically useful.

What carries the argument

The central machinery is the conformal prediction pipeline: a nonparametric quantile threshold computed on a held-out calibration set from a conformity score, here adaptive prediction sets (APS), produces a predictive set of classes for each test input. The guarantee it provides is marginal coverage under the assumption that calibration and test data are exchangeable. All of the paper's identified limitations stem from the fact that this guarantee is an average over the distribution: it quantifies over instances rather than for them, and exchangeability is exactly the assumption that deployment in a new clinic or with a shifted label distribution breaks.

What would settle it

Re-run the paper's CAMELYON17 domain-shift protocol (train on centers 1, 3, and 5, calibrate with 1000 patches from the validation set, test on centers 2 and 4) with APS, and compute coverage separately for each center and each class. If every center and every class attains at least the nominal $0.9$ coverage, then the paper's central claim that coverage breaks under domain shift would fail in that setting.

Watch

Extended reading notes

Core claim

Using the adaptive prediction sets (APS) conformity score with ResNet-34 classifiers, the paper demonstrates that conformal predictions satisfy the marginal coverage statement $P_{(x,y)\in D_{test}}(y \in C(\hat{Y}(x))) \geq 1-\alpha$ only as an average over the test distribution, and that this average hides systematic failures. The empirical calibration curves show that coverage holds on in-distribution test sets for both benchmarks; under domain shift (training on CAMELYON17 centers 1, 3, and 5, testing on centers 2 and 4) and under an artificial label-distribution shift the coverage guarantee is violated; after recalibrating with 1000 domain-shifted images the guarantee returns. On HAM10K the per-class curves show the frequent nevus class over-covered and the melanoma class under-covered even though marginal coverage is met. The selective-accuracy experiments, using a framework for selective classification, show that conformal prediction sets of size 1 are over-covered but conformal prediction cannot control their accuracy, whereas selective classification can bound accuracy with a chosen failure probability. The efficiency experiments show that when the seven HAM10K classes are collapsed into benign and malignant superclasses, an increasing fraction of prediction sets contain both classes, making them uninformative, and a binary setting has only three possible prediction sets.

Load-bearing premise

The broad prescriptive conclusions rest on the assumption that two experimental setups—ResNet-34 with the APS score on HAM10K and CAMELYON17, with 500 and 1000 calibration points and single-network uncertainty—are representative of medical image classification generally.

Editorial extensions

If this is right

A medical practitioner who calibrates once on validation data cannot quote a coverage level for any deployed subgroup, rare class, or future clinic unless the deployment distribution matches the calibration distribution.
Under input domain shift or label shift, the nominal $1-\alpha$ coverage is not a bound; re-collecting and recalibrating on each deployment domain is required to restore it.
Using conformal prediction sets of size 1 as a selection criterion for higher accuracy is not justified by the conformal framework; accuracy control requires selective classification methods.
In binary or few-class medical tasks, high desired coverage levels force conformal prediction sets to contain both classes, so the output carries no clinical decision and therefore limited practical value.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's logic implies a positive design target for medical uncertainty quantification: a method should offer conditional or class-conditional coverage, or an explicit recalibration protocol, rather than relying on a marginal average that can hide under-coverage of dangerous classes.
The experiments suggest a testable reporting standard: per-class and per-set-size coverage curves should accompany any marginal coverage claim in a medical conformal application; without them, over- and under-coverage can coexist with a perfectly calibrated average.
The paper's argument implies that monitoring the label distribution in deployment is as important as monitoring the input distribution, since exchangeability—and therefore the coverage guarantee—requires both to remain stable.
The binary-class inefficiency likely generalizes beyond medicine: any low-cardinality classification task that collapses into two decision-relevant superclasses will produce uninformative two-class prediction sets at high coverage levels.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a solid, clearly written practitioner's guide to known conformal prediction pitfalls, demonstrated on two medical datasets. There is no new theory or algorithm, but the experiments make the warnings concrete, and the authors are honest that the limitations are already established in the CP literature.

What I found valuable: the paper correctly lays out the core issues — marginal coverage is a distribution-level guarantee, not conditional; exchangeability between calibration and test is required and fails under input or label shift; CP does not control coverage for individual set sizes; and with few classes the prediction sets coarsen and can become uninformative. The figures on HAM10K and CAMELYON17 illustrate each point well, and the selective classification section properly distinguishes what CP can and cannot do, pointing to the alternative from Angelopoulos and Bates. Citations are appropriate, and the paper is not overselling itself.

Soft spots, in proportion: the averaged curves lack error bars or confidence intervals; minor, because the argument doesn't depend on precise numbers, but easy to fix. No code or data release, which limits reproducibility; also minor. The label-shift experiment is described vaguely (just 'resampling ... with a shifted label distribution'), and the protocol should be explicit. The scope claim is broader than the evidence: one architecture plus APS on two datasets doesn't establish a universal claim about medical image classification, but the authors carefully hedge with 'showcase' and conditional language, so this is a scope limitation, not a flaw. A few typos and an awkward sentence around Figure 3 don't affect substance.

I agree with the stress-test note: the central argument holds up. The paper is not aimed at CP researchers, who already know these facts; it is aimed at medical imaging practitioners who might otherwise overtrust conformal sets in deployment. For that audience it is genuinely useful. I would engage with it if I were an editor: it deserves a serious referee, but with a request for error bars, a clearer label-shift protocol, and code or instructions to reproduce the plots. Not a must-cite for my own work, but a good reading-group piece for people new to the area.

Referee Report

0 major / 6 minor

Summary. The manuscript discusses limitations of conformal prediction (CP) for medical image classification. It introduces split conformal prediction with the APS score, trains five ResNet-34 models on HAM10K and CAMELYON17, and presents calibration curves. It reports that marginal coverage holds in-distribution, but class-wise and set-size coverage vary; that coverage fails under covariate and label shift; that CP does not control the accuracy of size-1 sets and is therefore unsuitable for selective classification; and that in low-class settings predictive sets become uninformative. The paper is framed as a practical guide rather than a new methodological contribution, and it explicitly acknowledges that several of the limitations are already known in the CP literature.

Significance. The paper is a useful and largely correct synthesis of known CP properties for a medical imaging audience. Its strengths are: the theoretical statements are consistent with split-conformal theory; the experiments use two public medical datasets and a standard conformity score; the authors explicitly scope the experiments as illustrative; no fitted parameters or self-referential outputs are used; and the prescriptive claims are mostly conditional. The main limitation is that the empirical demonstrations are reported without variance or confidence intervals and some experimental protocols are under-specified, which weakens the quantitative support but not the underlying theory. If these presentation issues are fixed, the paper can serve as a cautionary guide for practitioners.

minor comments (6)

[Figures 1-3] The curves in Figures 1-3 are averages over 5 networks and 10 calibration samplings, but no error bars, confidence intervals, or per-run values are shown. Since some claims (e.g., per-class under-coverage in Fig. 1c) concern deviations from marginal coverage, the authors should report variance so that readers can judge whether these deviations are stable or within sampling noise.
[§5, Figure 1a] The label-shift experiment is not fully specified: the text only says that an artificial test set was generated by "resampling the in-distribution dataset with a shifted label distribution." Please state the original and shifted label proportions, the sample size, and whether the calibration set remained the original one; without this, the empirical label-shift curve cannot be reproduced.
[§6, Figure 2] The selective-classification curves need a clearer protocol: what threshold λ was used, how accuracy and rejection proportion were computed, and whether the calibration set size for the CP part was the same as the calibration size for the selective-classification part. This would make the comparison more interpretable.
[§8 and elsewhere] There are several typos and ungrammatical sentences: "neural nework" should be "neural network", "intend" should be "intent", "often situation" appears malformed, and "Figure 3 that with rising guarantees" is missing a verb (likely "shows that"). A careful copy-edit pass is needed.
[§2, Eq. (1)] The notation P_{(x_i,y_i) ∈ D_test} is nonstandard for a probability over a random draw; the coverage guarantee is over the joint distribution of a new exchangeable test point and the calibration data. Please use conventional notation such as P(y_test ∈ C(Ŷ(x_test))) ≥ 1 − α.
[§7] The statement that there are "only 3 possible prediction sets" in the binary case is correct but should be phrased as "at most three nontrivial sets" if the empty set is excluded. Also, the hypothetical 90% coverage examples would be clearer if explicitly labeled as illustrations rather than observed experimental outcomes.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are standard conformal prediction theory, and the illustrative experiments are self-contained against external datasets and external theoretical references.

full rationale

The paper's derivation chain does not reduce to its own inputs. The central claims—that conformal prediction guarantees marginal but not conditional coverage, break under distribution shift, do not control set-size-1 accuracy, and have limited efficiency with few classes—are direct consequences of the split conformal prediction framework as stated in Equation (1) and the exchangeability assumption in Section 2, supported by citations to external theoretical sources such as Vovk et al. and Angelopoulos & Bates. The experiments on HAM10K and CAMELYON17 use public datasets and a standard architecture; no fitted parameter is renamed as a prediction, and no calibrated value is used to justify the theoretical conclusions. The only self-citation is reference [5], used for training setup and the choice of single-network uncertainty baselines; this is not load-bearing for the paper's main argument, which holds independently of those experimental choices. The paper also explicitly frames experiments as illustrative rather than as proofs. Therefore no circular step is present.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are introduced; the paper relies on standard split-conformal and APS validity assumptions and on the representativeness of its two-dataset experimental setup. No invented entities are proposed.

assumptions (4)

standard math Exchangeability of calibration and test data is required for the conformal coverage guarantee.
Section 2 states that conformal predictions 'only necessitate the property of exchangeability between the datasets D_cal and D_test'; this is the background assumption of split conformal prediction.
standard math The APS conformity score produces valid marginal coverage under exchangeability.
Section 3 uses APS [14,15]; its validity is a cited theorem, not re-derived in this paper.
domain assumption A single neural network's softmax confidence is a usable uncertainty measure for the demonstrations.
Section 3 justifies singular networks by citing [4,5]; the paper does not independently benchmark this claim on its own trained pairs of models.
domain assumption The CAMELYON17 train/test split across centers models a realistic domain shift, and the resampled test set models label shift.
Section 5 uses Centers 1, 3, and 5 for training and Centers 2 and 4 for domain shift, and label shift is 'generated by resampling the in-distribution dataset with a shifted label distribution.' This modeling choice is plausible but not externally validated.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Pitfalls of Conformal Predictions for Medical Image Classification." pith.science (2026). https://pith.science/paper/HFN5NQVW

@misc{pith2026250618162,
  author       = {Pith},
  title        = {Pith review of: Pitfalls of Conformal Predictions for Medical Image Classification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HFN5NQVW}},
  note         = {Machine review of arXiv:2506.18162}
}

read the original abstract

Reliable uncertainty estimation is one of the major challenges for medical classification tasks. While many approaches have been proposed, recently the statistical framework of conformal predictions has gained a lot of attention, due to its ability to provide provable calibration guarantees. Nonetheless, the application of conformal predictions in safety-critical areas such as medicine comes with pitfalls, limitations and assumptions that practitioners need to be aware of. We demonstrate through examples from dermatology and histopathology that conformal predictions are unreliable under distributional shifts in input and label variables. Additionally, conformal predictions should not be used for selecting predictions to improve accuracy and are not reliable for subsets of the data, such as individual classes or patient attributes. Moreover, in classification settings with a small number of classes, which are common in medical image classification tasks, conformal predictions have limited practical value.

Figures

Figures reproduced from arXiv: 2506.18162 by the authors.

**Figure 2.** Selective Accuracy curves, using the approach of [12], on HAM10K with 500 calibration points (left) and [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 1.** b illustrates the measured coverage of different predictive set sizes. As can be seen, while the overall coverage [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗

Figures from the paper (1 more)

**Figure 3.** Figure 3: Efficiency of HAM10K with the seven individual [PITH_FULL_IMAGE:figures/full_fig_p005_3.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 9 canonical work pages

[1]

The need for uncertainty quantification in machine-assisted medical decision making

Edmon Begoli, Tanmoy Bhattacharya, and Dimitri Kusnezov. “The need for uncertainty quantification in machine-assisted medical decision making”. In: Nature Machine Intelligence 1.1 (2019), pp. 20–23. DOI: 10.1038/s42256-018-0004-1

doi:10.1038/s42256-018-0004-1 2019
[2]

Deep learning in histopathology: the path to the clinic

Jeroen Van der Laak, Geert Litjens, and Francesco Ciompi. “Deep learning in histopathology: the path to the clinic”. In: Nature medicine 27.5 (2021), pp. 775–784. DOI: 10.1038/s41591-021-01343-4

work page doi:10.1038/s41591-021-01343-4 2021
[3]

Second opinion needed: communicating uncertainty in medical machine learning

Benjamin Kompa, Jasper Snoek, and Andrew L Beam. “Second opinion needed: communicating uncertainty in medical machine learning”. In: NPJ Digital Medicine 4.1 (2021), p. 4. DOI: 10.1038/S41746-020-00367-3

doi:10.1038/s41746-020-00367-3 2021
[4]

A call to reflect on evaluation practices for failure detection in image classification

Paul F Jaeger et al. “A call to reflect on evaluation practices for failure detection in image classification”. In: arXiv preprint arXiv:2211.15259 (2022). DOI: 10.48550/arXiv.2211.15259

arXiv doi:10.48550/arxiv.2211.15259 2022
[5]

Benchmarking common uncertainty estimation methods with histopathological images under domain shift and label noise

Hendrik A. Mehrtens et al. “Benchmarking common uncertainty estimation methods with histopathological images under domain shift and label noise”. In: Medical Image Analysis (2023), p. 102914. ISSN : 1361-8415. DOI: 10.1016/j.media.2023.102914 . URL: https://www.sciencedirect.com/science/article/ pii/S1361841523001743

arXiv 2023
[6]

Vladimir V ovk, Alexander Gammerman, and Glenn Shafer.Algorithmic learning in a random world. V ol. 29. Springer, 2005. DOI: 10.1007/b106715

doi:10.1007/b106715 2005
[7]

Deep learning with conformal prediction for hierarchical analysis of large-scale whole-slide tissue images

Håkan Wieslander et al. “Deep learning with conformal prediction for hierarchical analysis of large-scale whole-slide tissue images”. In: IEEE journal of biomedical and health informatics 25.2 (2020), pp. 371–380. DOI: 10.1109/JBHI.2020.2996300

work page arXiv 2020
[8]

Estimating diagnostic uncertainty in artificial intelligence assisted pathology using confor- mal prediction

Henrik Olsson et al. “Estimating diagnostic uncertainty in artificial intelligence assisted pathology using confor- mal prediction”. In: Nature communications 13.1 (2022), p. 7761. DOI: 10.1038/s41467-022-34945-8

work page doi:10.1038/s41467-022-34945-8 2022

Show all 23 references

[9]

Fair conformal predictors for applications in medical imaging

Charles Lu et al. “Fair conformal predictors for applications in medical imaging”. In: Proceedings of the AAAI Conference on Artificial Intelligence. V ol. 36. 11. 2022, pp. 12008–12016.DOI: 10.1609/aaai.v36i11.21459

2022 doi
[10]

Three applications of conformal prediction for rating breast density in mammography

Charles Lu et al. “Three applications of conformal prediction for rating breast density in mammography”. In: arXiv preprint arXiv:2206.12008 (2022). DOI: 10.48550/ARXIV.2206.12008

2022 doi arXiv
[11]

Improving trustworthiness of ai disease severity rating in medical imaging with ordinal conformal prediction sets

Charles Lu, Anastasios N Angelopoulos, and Stuart Pomerantz. “Improving trustworthiness of ai disease severity rating in medical imaging with ordinal conformal prediction sets”. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springe...

2022 doi arXiv
[12]

A gentle introduction to conformal prediction and distribution- free uncertainty quantification

Anastasios N Angelopoulos and Stephen Bates. “A gentle introduction to conformal prediction and distribution- free uncertainty quantification”. In: arXiv preprint arXiv:2107.07511 (2021)

2021 arXiv
[13]

Selective classification for deep neural networks

Yonatan Geifman and Ran El-Yaniv. “Selective classification for deep neural networks”. In:Advances in neural information processing systems 30 (2017)

2017
[14]

Classification with valid and adaptive coverage

Yaniv Romano, Matteo Sesia, and Emmanuel Candes. “Classification with valid and adaptive coverage”. In: Advances in Neural Information Processing Systems 33 (2020), pp. 3581–3591. DOI: 10.48550/arXiv.2006. 02544

2020 doi
[15]

Uncertainty sets for image classifiers using conformal prediction

Anastasios Angelopoulos et al. “Uncertainty sets for image classifiers using conformal prediction”. In: arXiv preprint arXiv:2009.14193 (2020). DOI: 10.48550/arXiv.2009.14193

2020 doi arXiv
[16]

Deep residual learning for image recognition

Kaiming He et al. “Deep residual learning for image recognition”. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2016, pp. 770–778

2016
[17]

The HAM10000 dataset, a large collection of multi- source dermatoscopic images of common pigmented skin lesions

Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. “The HAM10000 dataset, a large collection of multi- source dermatoscopic images of common pigmented skin lesions”. In: Scientific data 5.1 (2018), pp. 1–9. DOI: 10.1038/sdata.2018.161

2018 doi
[18]

From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge

Peter Bandi et al. “From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge”. In: IEEE transactions on medical imaging 38.2 (2018), pp. 550–560. DOI: 10.1109/TMI.2018.2867350

2018
[19]

Simple and scalable predictive uncertainty estimation using deep ensembles

Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. “Simple and scalable predictive uncertainty estimation using deep ensembles”. In: Advances in neural information processing systems 30 (2017). 7 Pitfalls of Conformal Predictions for Medical Image Classification...

2017
[20]

Dropout as a bayesian approximation: Representing model uncertainty in deep learning

Yarin Gal and Zoubin Ghahramani. “Dropout as a bayesian approximation: Representing model uncertainty in deep learning”. In: international conference on machine learning. PMLR. 2016, pp. 1050–1059

2016
[21]

A simple baseline for bayesian uncertainty in deep learning

Wesley J Maddox et al. “A simple baseline for bayesian uncertainty in deep learning”. In:Advances in neural information processing systems 32 (2019)

2019
[22]

Conditional validity of inductive conformal predictors

Vladimir V ovk. “Conditional validity of inductive conformal predictors”. In: Asian conference on machine learning. PMLR. 2012, pp. 475–490. DOI: 10.1007/s10994-013-5355-6

2012 doi
[23]

Imagenet: A large-scale hierarchical image database

Jia Deng et al. “Imagenet: A large-scale hierarchical image database”. In: 2009 IEEE conference on computer vision and pattern recognition. Ieee. 2009, pp. 248–255. DOI: 10.1109/CVPR.2009.5206848. 8

2009

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

The need for uncertainty quantification in machine-assisted medical decision making

Edmon Begoli, Tanmoy Bhattacharya, and Dimitri Kusnezov. “The need for uncertainty quantification in machine-assisted medical decision making”. In: Nature Machine Intelligence 1.1 (2019), pp. 20–23. DOI: 10.1038/s42256-018-0004-1

doi:10.1038/s42256-018-0004-1 2019

[2] [2]

Deep learning in histopathology: the path to the clinic

Jeroen Van der Laak, Geert Litjens, and Francesco Ciompi. “Deep learning in histopathology: the path to the clinic”. In: Nature medicine 27.5 (2021), pp. 775–784. DOI: 10.1038/s41591-021-01343-4

work page doi:10.1038/s41591-021-01343-4 2021

[3] [3]

Second opinion needed: communicating uncertainty in medical machine learning

Benjamin Kompa, Jasper Snoek, and Andrew L Beam. “Second opinion needed: communicating uncertainty in medical machine learning”. In: NPJ Digital Medicine 4.1 (2021), p. 4. DOI: 10.1038/S41746-020-00367-3

doi:10.1038/s41746-020-00367-3 2021

[4] [4]

A call to reflect on evaluation practices for failure detection in image classification

Paul F Jaeger et al. “A call to reflect on evaluation practices for failure detection in image classification”. In: arXiv preprint arXiv:2211.15259 (2022). DOI: 10.48550/arXiv.2211.15259

arXiv doi:10.48550/arxiv.2211.15259 2022

[5] [5]

Benchmarking common uncertainty estimation methods with histopathological images under domain shift and label noise

Hendrik A. Mehrtens et al. “Benchmarking common uncertainty estimation methods with histopathological images under domain shift and label noise”. In: Medical Image Analysis (2023), p. 102914. ISSN : 1361-8415. DOI: 10.1016/j.media.2023.102914 . URL: https://www.sciencedirect.com/science/article/ pii/S1361841523001743

arXiv 2023

[6] [6]

Vladimir V ovk, Alexander Gammerman, and Glenn Shafer.Algorithmic learning in a random world. V ol. 29. Springer, 2005. DOI: 10.1007/b106715

doi:10.1007/b106715 2005

[7] [7]

Deep learning with conformal prediction for hierarchical analysis of large-scale whole-slide tissue images

Håkan Wieslander et al. “Deep learning with conformal prediction for hierarchical analysis of large-scale whole-slide tissue images”. In: IEEE journal of biomedical and health informatics 25.2 (2020), pp. 371–380. DOI: 10.1109/JBHI.2020.2996300

work page arXiv 2020

[8] [8]

Estimating diagnostic uncertainty in artificial intelligence assisted pathology using confor- mal prediction

Henrik Olsson et al. “Estimating diagnostic uncertainty in artificial intelligence assisted pathology using confor- mal prediction”. In: Nature communications 13.1 (2022), p. 7761. DOI: 10.1038/s41467-022-34945-8

work page doi:10.1038/s41467-022-34945-8 2022