REVIEW 3 major objections 5 minor 29 references
Calibrated Selective Prediction Using Deep Ensembles for ROI-Based Thyroid Nodule Ultrasound Classification Under Dataset Shift: A Retrospective Evaluation
T0 review · 3 major / 5 minor · reviewed 2026-07-15 · grok-4.5
Pith's one-line read A calibrated deep ensemble can support conservative selective triage of thyroid ultrasound images, but its frozen operating thresholds do not transport under dataset shift.
desk verdict Careful empirical stress test of a frozen calibrated ensemble triage policy: strong internal numbers, honest external failure, no clinical overclaim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The three-tier MI-guided triage policy: mutual information among five calibrated ensemble members first defers high-disagreement images; remaining low-disagreement cases receive a No-FNA suggestion only if mean probability is below a high-NPV rule-out threshold, an FNA recommendation only if above a high-PPV rule-in threshold, and otherwise also go to review. This turns ensemble disagreement into an operational deferral score that withholds automated suggestions when they are least reliable.
What would settle it
If local vector-scaling recalibration and re-derivation of the MI and probability thresholds on a modest site-specific validation set drawn from the same distribution as the external cohort recovered at least 95 percent FNA PPV and 98 percent No-FNA NPV with autonomous coverage comparable to the internal rates, the limited-transportability claim would be overturned; if those targets remained unreachable without near-total deferral, the claim would be confirmed.
Extended reading notes
Core claim
On the development dataset the calibrated five-member ConvNeXt-Tiny ensemble reaches pooled out-of-fold AUC-ROC 0.9395 and expected calibration error 0.0088; a mutual-information gate plus probability thresholds can issue autonomous no-FNA and FNA suggestions while capturing 99.83 percent of malignancies. When the identical frozen model and thresholds are applied to an independent external cohort, discrimination and calibration both degrade and the policy largely collapses into radiologist review, showing that selective prediction can flag unsuitable images but that the operating policy itself does not transport without local recalibration.
Load-bearing premise
The paper assumes that image-level splits of the development set, made without patient identifiers, and lesion crops taken from provided boxes or masks are independent and representative enough that the internal thresholds remain a fair test of external transportability.
Editorial extensions
If this is right
- Autonomous image-based No-FNA and FNA suggestions should be issued only for low-disagreement, probability-extreme cases; intermediate or high-disagreement images should be routed to human review.
- Strong internal discrimination and calibration do not guarantee that probability or uncertainty thresholds remain valid under dataset shift.
- Local recalibration of probabilities and re-validation of MI and decision thresholds are required before any frozen policy can be considered at a new site.
- Selective prediction can preserve malignancy capture mainly by increasing the review burden when the data distribution changes.
- Claims of actual biopsy reduction cannot be made from ROI-level retrospective assignment alone; clinical variables and prospective workflow studies remain necessary.
Reading between the lines
- The jump to 83.7 percent external review implies that the uncertainty score itself is distribution-sensitive, so fixed MI cut-offs may need to be replaced by domain-adaptive or conformal deferral rules.
- Because internal splits were image-level without patient IDs, residual same-patient correlation could have inflated the internal calibration and selective-risk numbers against which transportability is judged.
- MSP-derived uncertainty ranked internal errors better than MI; testing which score retains better external ranking would clarify whether epistemic disagreement or pure confidence is the more portable safety signal.
- The same selective-prediction template could be stress-tested on other ultrasound tasks where annotation style and prevalence also shift between public cohorts.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript develops a calibrated five-member deterministic ConvNeXt-Tiny ensemble (with SE attention) for annotation-assisted ROI-based thyroid-nodule ultrasound classification and selective image-based triage. TN5000 is used for five-fold CV, member-wise vector-scaling calibration on held-out val_calib, and fold-specific MI/probability threshold selection on val_es; a frozen deployment model and 50% nominal-MI policy are then applied unchanged to the independent TN3K cohort. Internally the ensemble reports strong pooled OOF discrimination and calibration (AUC-ROC 0.9395, AP 0.9715, ECE 0.0088) and a conservative three-tier policy (No-FNA / FNA / radiologist review) that retains high malignancy capture. Externally discrimination falls, calibration collapses, and the frozen thresholds produce an 83.7% review rate with degraded FNA PPV, supporting the authors’ claim of limited threshold transportability under dataset shift.
Significance. If the reported internal-to-external gap holds, the work supplies a carefully documented empirical demonstration that calibrated ensemble disagreement can support conservative selective referral on a malignant-enriched development set while the same frozen operating policy fails to transport. The clean separation of calibration, threshold derivation, and external evaluation, together with risk-coverage comparisons and explicit non-claims of clinical deployability or biopsy reduction, is a useful methodological contribution for medical imaging under dataset shift. Public datasets, detailed protocol, and code availability further strengthen reproducibility. The result is incremental rather than transformative, but it is a solid, cautious addition to the selective-prediction literature in thyroid ultrasound.
major comments (3)
- Sections 3.1 and 6 correctly flag that TN5000 patient identifiers are unavailable, so image-level stratified splits cannot guarantee patient-level independence. Residual patient-level leakage would inflate the internal OOF AUC/ECE and the internal-to-external gap that underpins the transportability claim. The authors already treat this as a limitation; a quantitative sensitivity analysis (e.g., approximate patient clustering if any metadata exist, or a leave-one-site-style ablation) or a clearer statement that the internal numbers are upper bounds would make the central comparison more robust.
- Section 3.6 and Supplementary S2 show that MSP-derived uncertainty and predictive variance both outperform MI on AURC/AUGRC (0.03020/0.02261 and 0.03234/0.02424 vs MI 0.03778/0.02722). The decision to retain MI for the frozen external policy is hypothesis-driven and transparent, yet the external transportability conclusion is therefore tied to a sub-optimal ranking score. Reporting the same frozen-policy experiment with the best internal score (or at least MSP) would test whether the observed TN3K failure is score-specific or general.
- Supplementary S1 documents that TN5000 ROIs are directly resized while TN3K ROIs use aspect-preserving resize plus black padding; annotation type also differs (boxes vs masks). These preprocessing differences are part of the declared dataset shift, but their contribution to the ECE rise (0.0088 → 0.1899) and MI-gate failure (only 20.5% of TN3K pass the frozen τ_MI) is not isolated. A controlled re-processing of one dataset with the other’s pipeline would strengthen the claim that the degradation is domain shift rather than ROI-construction artifact.
minor comments (5)
- Abstract and Results report slightly different pooled AP values (0.9715 vs fold-mean 0.9720); a single consistent figure should be used throughout.
- Table 7 and the surrounding text use “nominal MI retention”; a one-sentence reminder that this is the val_es MI-gate fraction, not final autonomous coverage, would reduce reader confusion.
- Figure 5 Grad-CAM examples are qualitative only; the caption already states this, but the main text could more explicitly note that they are single-member (seed 42) maps and not used for any decision.
- A few typographical inconsistencies appear (e.g., “au tomated”, “multicentre decision-support”, missing spaces after commas in places); a final copy-edit pass is warranted.
- The web prototype (Supplementary Figure S4) is a nice demonstration but is not evaluated; a single sentence clarifying that it is outside the scientific claims would prevent over-interpretation.
Circularity Check
No significant circularity: empirical hold-out evaluation of a fitted selective-triage policy with honest external degradation.
full rationale
The paper is a standard retrospective ML evaluation. Models, member-wise vector-scaling parameters, MI thresholds, and probability rule-out/rule-in thresholds are fitted exclusively on TN5000 train/val_es/val_calib partitions (Sections 3.4–3.10); held-out fold and pooled OOF metrics, plus the frozen-policy application to TN3K, never re-enter fitting. External TN3K labels are used only for outcome reporting. The three-tier policy and NPV/PPV targets are operational choices evaluated after the fact, not definitions that force the reported discrimination, calibration, or transportability numbers. No self-definitional equations, no fitted quantity renamed as an independent prediction, no load-bearing self-citation uniqueness claims, and no ansatz smuggled via prior author work appear. The central claim (strong internal calibrated selective performance + limited frozen-threshold transportability) is therefore self-contained against the reported benchmarks.
Assumptions & free parameters
free parameters (4)
- vector-scaling weights and biases (per member, per fold)
- MI retention grid and resulting τ_MI, τ_low, τ_high
- ensemble size M=5 and random seeds
- backbone and SE module choice
assumptions (5)
- domain assumption Image-level stratified splits of TN5000 yield sufficiently independent train/val/test estimates despite missing patient IDs.
- domain assumption Provided bounding boxes / masks produce ROIs representative of clinical lesion appearance.
- domain assumption Mutual information of calibrated member probabilities is a usable operational proxy for epistemic disagreement.
- ad hoc to paper 98% NPV and 95% PPV are appropriate retrospective safety targets for image-based triage.
- standard math Standard selective-classification risk-coverage evaluation and deep-ensemble uncertainty theory apply.
invented entities (1)
-
strict three-tier MI-guided image-based triage policy (No-FNA / FNA / review)
Cite this review
Pith. "Pith review of Calibrated Selective Prediction Using Deep Ensembles for ROI-Based Thyroid Nodule Ultrasound Classification Under Dataset Shift: A Retrospective Evaluation." pith.science (2026). https://pith.science/paper/WZWWWITL
@misc{pith2026260712075,
author = {Pith},
title = {Pith review of: Calibrated Selective Prediction Using Deep Ensembles for ROI-Based Thyroid Nodule Ultrasound Classification Under Dataset Shift: A Retrospective Evaluation},
year = {2026},
howpublished = {\url{https://pith.science/paper/WZWWWITL}},
note = {Machine review of arXiv:2607.12075}
}
read the original abstract
Background: Deep learning models can classify thyroid nodules on ultrasound, but reliable clinical decision support also requires calibrated probabilities, uncertainty estimation, and selective referral, particularly under dataset shift. Methods: We developed a calibrated deterministic five-member deep ensemble for ROI-based thyroid nodule classification and selective image-based triage. TN5000 was used for model development, five-fold cross-validation, member-wise vector-scaling calibration, and fold-specific threshold selection. TN3K served as an independent external dataset-shift evaluation. The framework used ConvNeXt-Tiny with squeeze-and-excitation attention, ensemble-mean malignancy probability, and mutual information (MI) as an ensemble-disagreement score. A three-tier policy assigned images to No-FNA suggestion, FNA recommendation, or radiologist review. Results: On pooled out-of-fold TN5000 predictions, the ensemble achieved AUC-ROC 0.9395, AP 0.9715, ECE 0.0088, and Brier score 0.0813. At 50% nominal MI retention, 7.2% of cases received a No-FNA suggestion, 39.9% an FNA recommendation, and 52.9% radiologist review, with 98.3% No-FNA NPV and 99.83% malignancy capture. On TN3K, AUC-ROC decreased to 0.7870, AP to 0.7254, ECE increased to 0.1899, and Brier score to 0.2281. The frozen TN5000 policy assigned 83.7% to review, 1.0% to No-FNA, and 15.3% to FNA recommendation. No malignant image entered the No-FNA pathway, but FNA-recommendation PPV fell to 76.6%. Conclusion: The framework showed strong internal discrimination and calibration, but limited external threshold transportability. Selective prediction may help identify images unsuitable for automated triage, but local recalibration, threshold validation, and prospective clinical evaluation are required before deployment.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
B. R. Haugen et al., “2015 American Thyroid Association management guidelines for adult patients with thyroid nodules and differentiated thyroid cancer: The American Thyroid Association Guidelines Task Force on Thyroid Nodules and Differentiated Thyroid Cancer,” Thyroid, vol. 26, no. 1, pp. 1–133, 2016, doi: 10.1089/thy.2015.0020
-
[2]
ACR Thyroid Imaging, Reporting and Data System (TI-RADS): White paper of the ACR TI-RADS Committee,
F. N. Tessler et al., “ACR Thyroid Imaging, Reporting and Data System (TI-RADS): White paper of the ACR TI-RADS Committee,” Journal of the American College of Radiology , vol. 14, no. 5, pp. 587–595, 2017, doi: 10.1016/j.jacr.2017.01.046
-
[3]
Q. Guan et al., “Deep learning based classification of ultrasound images for thyroid nodules: A large scale of pilot study,” Annals of Translational Medicine , vol. 7, no. 7, Art. no. 137, 2019, doi: 10.21037/atm.2019.04.34
-
[4]
S. Peng et al., “Deep learning-based artificial intelligence model to assist thyroid nodule diagnosis and management: A multicentre diagnostic study,” The Lancet Digital Health, vol. 3, no. 4, pp. e250– e259, 2021, doi: 10.1016/S2589-7500(21)00041-8
-
[5]
Thyroid nodule classification in ultrasound imaging using deep transfer learning,
Y. Xu, M. Xu, Z. Geng, J. Liu, and B. Meng, “Thyroid nodule classification in ultrasound imaging using deep transfer learning,” BMC Cancer, vol. 25, Art. no. 544, 2025, doi: 10.1186/s12885 -025- 13917-3
doi:10.1186/s12885 2025
-
[6]
J. Kim et al ., “Deep learning technology for classification of thyroid nodules using multi -view ultrasound images: Potential benefits and challenges in clinical application,” Endocrinology and Metabolism, vol. 40, no. 2, pp. 216–224, 2025, doi: 10.3803/EnM.2024.2058
-
[7]
Deep learning for the diagnosis of suspicious thyroid nodules based on multimodal ultrasound images,
Y. Tao et al., “Deep learning for the diagnosis of suspicious thyroid nodules based on multimodal ultrasound images,” Frontiers in Oncology , vol. 12, Art. no. 1012724, 2022, doi: 10.3389/fonc.2022.1012724
-
[8]
S. Vahdati, B. Khosravi, K. A. Robinson, P. Rouzrokh, M. Moassefi, Z. Akkus, and B. J. Erickson, “A multi-view deep learning model for thyroid nodules detection and characterization in ultrasound imaging,” Bioengineering, vol. 11, no. 7, Art. no. 648, 2024, doi: 10.3390/bioengineering11070648
Show all 29 references
-
[9]
From bench-to-bedside: How artificial intelligence is changing thyroid nodule diagnostics, a systematic review,
V. R. Sant et al., “From bench-to-bedside: How artificial intelligence is changing thyroid nodule diagnostics, a systematic review,” Journal of Clinical Endocrinology and Metabolism, vol. 109, no. 7, pp. 1684–1693, 2024, doi: 10.1210/clinem/dgae277
2024 doi
-
[10]
Deep learning for classification of thyroid nodules on ultrasound: Validation on an independent dataset,
J. Weng et al., “Deep learning for classification of thyroid nodules on ultrasound: Validation on an independent dataset,” Clinical Imaging , vol. 99, pp. 60 –66, 2023, doi : 10.1016/j.clinimag.2023.04.010
2023 doi
-
[11]
Optimizing thyroid nodule management with artificial intelligence: Multicenter retrospective study on reducing unnecessary fine needle aspirations,
J.-H. Ni et al., “Optimizing thyroid nodule management with artificial intelligence: Multicenter retrospective study on reducing unnecessary fine needle aspirations,” JMIR Medical Informatics, vol. 13, Art. no. e71740, 2025, doi: 10.2196/71740
2025 doi
-
[12]
Enhancing automatic diagnosis of thyroid nodules from ultrasound scans leveraging deep learning models,
A. Rashed, T. Medhat, and A. Elgarayhi, “Enhancing automatic diagnosis of thyroid nodules from ultrasound scans leveraging deep learning models,” Scientific Reports, vol. 15, Art. no. 40364, 2025, doi: 10.1038/s41598-025-25780-0
2025 doi
-
[13]
On calibration of modern neural networks,
C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in Proceedings of the 34th International Conference on Machine Learning, vol. 70, 2017, pp. 1321–1330
2017
-
[14]
A review of uncertainty estimation and its application in medical imaging,
K. Zou, Z. Chen, X. Yuan, X. Shen, M. Wang, and H. Fu, “A review of uncertainty estimation and its application in medical imaging,” Meta-Radiology, vol. 1, no. 1, Art. no. 100003, 2023, doi: 10.1016/j.metrad.2023.100003
2023 doi
-
[15]
Uncertainty-aware deep learning in healthcare: A scoping review,
T. J. Loftus et al ., “Uncertainty-aware deep learning in healthcare: A scoping review,” PLOS Digital Health, vol. 1, no. 8, Art. no. e0000085, 2022, doi: 10.1371/journal.pdig.0000085
2022 doi
-
[16]
Simple and scalable predictive uncertainty estimation using deep ensembles,
B. Lakshminarayanan, A. Pritzel, and C. Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles,” in Advances in Neural Information Processing Systems , vol. 30, 2017
2017
-
[17]
Dropout as a Bayesian approximation: Representing model uncertainty in deep learning,
Y. Gal and Z. Ghahramani, “Dropout as a Bayesian approximation: Representing model uncertainty in deep learning,” in Proceedings of the 33rd International Conference on Machine Learning, vol. 48, 2016, pp. 1050–1059
2016
-
[18]
Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift,
Y. Ovadia et al., “Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift,” in Advances in Neural Information Processing Systems, vol. 32, 2019
2019
-
[19]
Role of calibration in uncertainty-based referral for deep learning,
R. Zhang, C. Gatsonis, and J. A. Steingrimsson, “Role of calibration in uncertainty-based referral for deep learning,” Statistical Methods in Medical Research , vol. 32, no. 5, pp. 927 –943, 2023, doi : 10.1177/09622802231158811
2023 doi
-
[20]
Selective classification for deep neural networks,
Y. Geifman and R. El -Yaniv, “Selective classification for deep neural networks,” Advances in Neural Information Processing Systems, vol. 30, 2017
2017
-
[21]
Overcoming common flaws in the evaluation of selective classification systems,
J. Traub, T. J. Bungert, C. T. Lüth, M. Baumgartner, K. H. Maier -Hein, L. Maier-Hein, and P. F. Jäger, “Overcoming common flaws in the evaluation of selective classification systems,” arXiv:2407.01032, 2024
2024 arXiv
-
[22]
TN5000: An ultrasound image dataset for thyroid nodule detection and classification,
H. Zhang, Q. Liu, X. Han, L. Niu, and W. Sun, “TN5000: An ultrasound image dataset for thyroid nodule detection and classification,” Scientific Data, vol. 12, Art. no. 1437, 2025, doi: 10.1038/s41597- 025-05757-4
2025 doi
-
[23]
An ultrasonography of thyroid nodules dataset with pathological diagnosis annotation for deep learning,
X. Hou et al ., “An ultrasonography of thyroid nodules dataset with pathological diagnosis annotation for deep learning,” Scientific Data, vol. 11, Art. no. 1272, 2024, doi: 10.1038/s41597 -024- 04156-5
2024 doi
-
[24]
Deep learning-based classification of thyroid nodules using uncertainty -aware multi-modal ultrasound imaging,
M. Saini, T. A. Parvar, M. Velarde, N. B. Larson, M. Fatemi, and A. Alizad, “Deep learning-based classification of thyroid nodules using uncertainty -aware multi-modal ultrasound imaging,” Scientific Reports, vol. 16, Art. no. 4938, 2026, doi: 10.1038/s41598-026-35965-w
2026 doi
-
[25]
MTL -TN: An uncertainty -weighted multi-task framework for unified thyroid nodule segmentation and TI -RADS classification,
T. Xiang, B. Li, Z. Hu, X. Wang, Y. Dang, and Y. Deng, “MTL -TN: An uncertainty -weighted multi-task framework for unified thyroid nodule segmentation and TI -RADS classification,” in Proc. 2025 5th Int. Conf. Computer, Internet of Things and Control Engineering (CITCE), 2025,...
2025 doi
-
[26]
Thyroid nodule and lymph node metastasis assessment from ultrasound images using deep learning,
X. Zhao et al., “Thyroid nodule and lymph node metastasis assessment from ultrasound images using deep learning,” Frontiers in Neuroscience , vol. 19, Art. no. 1684104, 2025, doi: 10.3389/fnins.2025.1684104
2025 doi
-
[27]
A ConvNet for the 2020s,
Z. Liu, H. Mao, C. -Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A ConvNet for the 2020s,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 11966–11976, doi: 10.1109/CVPR52688.2022.01167
2022 doi
-
[28]
Squeeze-and-excitation networks,
J. Hu, L. Shen, S. Albanie, G. Sun, and E. Wu, “Squeeze-and-excitation networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 7132–7141, doi: 10.1109/CVPR.2018.00745
2018 doi
-
[29]
Grad-CAM: Visual explanations from deep networks via gradient -based localization,
R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-CAM: Visual explanations from deep networks via gradient -based localization,” in Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 618–626, doi: 10.1109/ICCV.2017.74...
2017 doi
Reviewed July 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.