REVIEW 3 major objections 6 minor 20 references
Can we trust deep learning models diagnosis? The impact of domain shift in chest radiograph classification
T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Chest X-ray AI models drop sharply when used outside their training dataset.
desk verdict Solid four-dataset cross-evaluation of chest X-ray classifiers, but the headline attribution to image domain shift is partly confounded by label mismatch across datasets. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the controlled cross-dataset evaluation protocol: the same DenseNet121 convolutional neural network, with identical hyperparameters and ImageNet pretraining, is trained separately on each of the four datasets, then evaluated on all four test sets using the eight labels common to every dataset. This isolates domain shift as the cause of performance differences, provided label semantics are comparable. The metric is mean Area Under the ROC Curve (AUC) over the eight shared findings.
What would settle it
A reader study that manually re-annotates a random sample of images from all four datasets using a single shared label protocol and then repeats the cross-dataset training; if the AUC gaps vanish or substantially shrink, the paper's attribution of the drops to image domain shift would be refuted.
Extended reading notes
Core claim
The central discovery is that domain shift—differences in image appearance, acquisition protocols, populations, and label generation between chest X-ray collections—substantially degrades model performance. For all four test datasets, the highest mean AUC is achieved by the model trained on that same dataset. The largest gap observed is 0.12 in mean AUC: a model trained on ChestX-ray14 scores 0.6821 on CheXpert while a CheXpert-trained model scores 0.8042. The authors additionally find that models trained on CheXpert and MIMIC-CXR generalize better to other datasets than models trained on ChestX-ray14 or PadChest, and they attribute part of the transfer difficulty to differences in label extraction methods and label noise.
Load-bearing premise
The comparison assumes that the eight labels mean the same clinical finding in every dataset; different NLP labelers, report languages, and merged label definitions may make the measured gaps reflect label mismatch as well as image domain shift.
Editorial extensions
If this is right
- Clinicians and regulators should expect reported accuracy on a source dataset to overstate real-world performance when the deployment population or imaging equipment differs.
- Researchers developing chest X-ray classifiers should prefer CheXpert and MIMIC-CXR as training sources, since models trained on them retain more performance across other datasets.
- A case-by-case external validation strategy is necessary: models should be validated or fine-tuned on small local datasets from the specific machines and settings where they will be used.
- The observed performance gaps are not uniform across findings, so per-finding external evaluation is warranted rather than relying only on a single averaged metric.
Reading between the lines
- If label noise and label extraction differences are the main drivers of the gap, then the measured 'domain shift' may partly be 'annotation shift'; a direct measurement using manually re-labeled images across datasets could separate image-level shift from label-level shift.
- The authors' finding that CheXpert and MIMIC-CXR generalize better might reflect their use of a shared labeler and larger training sizes; a follow-up that matches training set sizes and label distributions could test whether the advantage is intrinsic to those datasets or an artifact of scale.
- The same cross-dataset protocol could be extended to other imaging modalities and tasks, such as CT, MRI, or segmentation, to see whether the ranking of dataset representativeness holds beyond chest X-rays.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript reports a controlled empirical comparison of a fixed DenseNet-121 (CheXNet) multi-label classifier trained on four public chest X-ray datasets (ChestX-ray14, CheXpert, MIMIC-CXR, PadChest) and evaluated on all four test sets. Using eight labels obtained by merging and simplifying the original label sets, it reports per-class and mean AUC in Table 2. The main findings are that same-dataset training yields the best mean AUC on every test set, and that models trained on CheXpert or MIMIC-CXR transfer better to other datasets than models trained on ChestX-ray14 or PadChest. The authors conclude that domain shift causes substantial performance loss and recommend external, case-by-case validation.
Significance. If the headline result is taken only as a statement about cross-dataset transfer of models as deployed, the paper is a useful empirical benchmark and a clear cautionary data point for medical imaging practice. The experimental setup is simple and transparent, the four datasets are public, and the discussion honestly acknowledges labeler-related concerns. The paper also makes a concrete, falsifiable claim about transferability ranking that other groups can test. However, the quantitative contribution is weakened by the absence of uncertainty estimates and by uncontrolled label-generation differences across datasets, so the significance is moderate rather than high.
major comments (3)
- [Section 3.2 and Table 2] The central attribution of the observed AUC drops to image domain shift is not fully secured because the eight shared labels are produced by different annotation pipelines. ChestX-ray14's 'Lesion' is a merge of 'Nodule' and 'Mass', PadChest subtypes are collapsed without a mapping table or validation, and the NLP labelers and report languages differ; Section 4 additionally cites evidence that ChestX-ray14 labels may be 10-30% less accurate than originally reported. A positive label therefore need not denote the same clinical finding across datasets, so the cross-dataset gaps in Table 2 can reflect label mismatch and label noise even if the image distribution were unchanged. Please validate label equivalence, or explicitly reframe the conclusions as measuring transfer under the datasets' existing label definitions rather than image domain shift.
- [Section 4 and Table 2] All results are single point estimates from one training run per dataset with no confidence intervals, error bars, or repeated seeds. The differences that support the transferability ranking (for example, the 0.04 mean-AUC gap between the CheXpert-trained and MIMIC-CXR-trained models on the CheXpert test set) may be within run-to-run variation. Report repeated runs or bootstrap confidence intervals before ranking source datasets.
- [Section 4 and Table 2] The mean AUC entries for two rows do not match the arithmetic mean of the eight per-class values shown in the same row. On the CheXpert test set with ChestX-ray14 training, the listed mean is 0.6821 whereas the eight listed per-class AUCs average to 0.6933; on the MIMIC-CXR test set with ChestX-ray14 training, the listed mean is 0.7406 whereas the eight values average to 0.7494. These inconsistencies affect the quantitative claims in the Discussion and should be corrected.
minor comments (6)
- [Section 3.2] The text says the trained model is evaluated 'with images from the remaining two' datasets, but after training on each of four datasets there are three remaining test sets; please correct this to 'remaining three'.
- [Table 2] Several entries are typeset without spacing, such as '0.93900.6833' and 'MIMIC-CXR0.7942'; these should be formatted as separate columns.
- [References] Reference [20] is cited for ChestX-ray14 but the listed title is 'ChestX-ray8'; please correct the title or the reference.
- [Figure 2] The pixel-intensity density plot lacks axis labels and units; adding them would allow readers to interpret the distributions.
- [Section 3.2] The training hyperparameters (optimizer, learning rate, batch size, number of epochs, and loss function) are not stated, which limits reproducibility despite the architecture being specified; please add a short training-details paragraph.
- [Figure 3] The caption contains a typo ('cotains') and the sentence describing the composite image is hard to parse; please revise.
Circularity Check
No significant circularity: the paper is an empirical cross-dataset evaluation with no derivation or fitted parameter being renamed as a prediction.
full rationale
The paper makes no theoretical derivation and does not claim to predict outcomes from first principles. Its central claim is empirical: models trained on one chest radiograph dataset perform worse on another dataset, as measured by AUC in Table 2. The models are trained on standard training splits and evaluated on held-out test sets from other datasets; no parameter is fitted to the target results, and no result is defined in terms of the claim it supports. The only potentially load-bearing choices are the construction of common labels across datasets (Section 3.2), such as merging ChestX-ray14 Nodule/Mass into Lesion and merging PadChest subtypes into Atelectasis, and the assumption that these labels denote equivalent clinical findings across datasets. These choices could confound label mismatch with image domain shift, but they are not circular reasoning: the labels are input definitions, not outputs derived from the models, and the paper explicitly acknowledges labeler differences and even cites evidence that ChestX-ray14 labels are unreliable. Likewise, citing CheXNet for architecture and prior results is external evidence, not a self-citation chain. The finding that same-dataset training gives the best average AUC is an observed outcome, not an identity forced by construction, because the evaluation metric is computed on independent test data. Therefore, the paper contains no circular step under the stated criteria.
Assumptions & free parameters
assumptions (3)
- domain assumption The eight shared radiological labels denote the same findings in all four datasets.
- domain assumption Uncertainty labels in CheXpert and MIMIC-CXR can be treated as negatives (U-Zeros approach).
- domain assumption The random 70/20/10 splits of CheXpert, MIMIC-CXR, and PadChest are representative and comparable to the official ChestX-ray14 split.
Cite this review
Pith. "Pith review of Can we trust deep learning models diagnosis? The impact of domain shift in chest radiograph classification." pith.science (2026). https://pith.science/paper/4ANRVNI4
@misc{pith2026190901940,
author = {Pith},
title = {Pith review of: Can we trust deep learning models diagnosis? The impact of domain shift in chest radiograph classification},
year = {2026},
howpublished = {\url{https://pith.science/paper/4ANRVNI4}},
note = {Machine review of arXiv:1909.01940}
}
read the original abstract
While deep learning models become more widespread, their ability to handle unseen data and generalize for any scenario is yet to be challenged. In medical imaging, there is a high heterogeneity of distributions among images based on the equipment that generates them and their parametrization. This heterogeneity triggers a common issue in machine learning called domain shift, which represents the difference between the training data distribution and the distribution of where a model is employed. A high domain shift tends to implicate in a poor generalization performance from the models. In this work, we evaluate the extent of domain shift on four of the largest datasets of chest radiographs. We show how training and testing with different datasets (e.g., training in ChestX-ray14 and testing in CheXpert) drastically affects model performance, posing a big question over the reliability of deep learning models trained on public datasets. We also show that models trained on CheXpert and MIMIC-CXR generalize better to other datasets.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
AlBadawy, E.A., Saha, A., Mazurowski, M.A.: Deep learning for segmentation of brain tumors: Impact of cross-institutional training and testing. Medical physics 45(3) (2018)
work page 2018
-
[2]
Bustos, A., Pertusa, A., Salinas, J.M., de la Iglesia-Vay, M.: Padchest: A large chest x-ray image dataset with multi-label annotated reports (2019) The impact of domain shift in chest radiograph classification 9
work page 2019
-
[3]
In: Pro- ceedings of the International Workshop on Machine Learning in Medical Imaging
Chen, C., Dou, Q., Chen, H., Heng, P.A.: Semantic-aware generative adversarial nets for unsupervised domain adaptation in chest x-ray segmentation. In: Pro- ceedings of the International Workshop on Machine Learning in Medical Imaging. Springer (2018)
work page 2018
-
[4]
https://doi.org/10.5152/dir.2016.16187
del Ciello, A., Franchi, P., Contegiacomo, A., Cicchetti, G., Bonomo, L., Larici, A.R.: Missed lung cancer: when, where, and why? Diagnostic and Interventional Radiology 23(2), 118–126 (mar 2017). https://doi.org/10.5152/dir.2016.16187
-
[5]
In: Proceedings of the 27th International Joint Conference on Artificial Intelligence
Dou, Q., Ouyang, C., Chen, C., Chen, H., Heng, P.A.: Unsupervised cross-modality domain adaptation of convnets for biomedical image segmentations with adversar- ial loss. In: Proceedings of the 27th International Joint Conference on Artificial Intelligence. pp. 691–697 (2018)
work page 2018
-
[6]
The Journal of Machine Learning Research 17(1) (Apr 2016)
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., Lempitsky, V.: Domain-adversarial training of neural networks. The Journal of Machine Learning Research 17(1) (Apr 2016)
work page 2016
-
[7]
Gholami, A., Subramanian, S., Shenoy, V., Himthani, N., Yue, X., Zhao, S., Jin, P., Biros, G., Keutzer, K.: A novel domain adaptation framework for medical im- age segmentation. In: Proceedings of the International Medical Image Computing and Computer Assisted Intervention Brainlesion Workshop. pp. 289–298. Springer (2018)
work page 2018
-
[8]
Clinical Cancer Research 7(1), 5–22 (2001)
Hirsch, F.R., Franklin, W.A., Gazdar, A.F., Bunn, P.A.: Early detection of lung cancer: clinical perspectives of recent advances in biology and radiology. Clinical Cancer Research 7(1), 5–22 (2001)
work page 2001
Show all 20 references
-
[9]
In: Proceedings of the International Conference on Machine Learning
Hoffman, J., Tzeng, E., Park, T., Zhu, J.Y., Isola, P., Saenko, K., Efros, A.A., Dar- rell, T.: Cycada: Cycle consistent adversarial domain adaptation. In: Proceedings of the International Conference on Machine Learning. p. 15 (2018)
2018
-
[10]
In: Proceedings of the IEEE conference on computer vision and pattern recognition
Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4700–4708 (2017)
2017
-
[11]
arXiv preprint arXiv:1901.07031 (2019)
Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al.: Chexpert: A large chest ra- diograph dataset with uncertainty labels and expert comparison. arXiv preprint arXiv:1901.07031 (2019)
2019 arXiv
-
[12]
In: Proceedings of the 15th International Sym- posium on Biomedical Imaging
Javanmardi, M., Tasdizen, T.: Domain adaptation for biomedical image segmen- tation using adversarial training. In: Proceedings of the 15th International Sym- posium on Biomedical Imaging. IEEE (2018)
2018
-
[13]
arXiv preprint arXiv:1901.07042 (2019)
Johnson, A.E., Pollard, T.J., Berkowitz, S., Greenbaum, N.R., Lungren, M.P., Deng, C.y., Mark, R.G., Horng, S.: Mimic-cxr: A large publicly available database of labeled chest radiographs. arXiv preprint arXiv:1901.07042 (2019)
2019 arXiv
-
[14]
Medical Image Analysis 42, 60–88 (Dec 2017)
Litjens, G., Kooi, T., Bejnordi, B.E., Setio, A.A.A., Ciompi, F., Ghafoorian, M., van der Laak, J.A., van Ginneken, B., S´ anchez, C.I.: A survey on deep learning in medical image analysis. Medical Image Analysis 42, 60–88 (Dec 2017)
2017
-
[15]
In: IEEE 15th International Sym- posium on Biomedical Imaging (ISBI 2018)
Madani, A., Moradi, M., Karargyris, A., Syeda-Mahmood, T.: Semi-supervised learning with generative adversarial networks for chest X-ray classification with ability of data domain adaptation. In: IEEE 15th International Sym- posium on Biomedical Imaging (ISBI 2018). pp. 1038–10...
2018
-
[16]
IEEE Transactions on Medical Imaging 37, 10 (Jun 2018) 10 Pooch et al
Mahmood, F., Chen, R., Durr, N.J.: Unsupervised reverse domain adaptation for synthetic medical images via adversarial training. IEEE Transactions on Medical Imaging 37, 10 (Jun 2018) 10 Pooch et al
2018
-
[17]
Oakden-Rayner, L.: Exploring large scale public medical image datasets. Tech. rep., The University of Adelaide (2019), https://arxiv.org/pdf/1907.12720.pdf
2019 arXiv
-
[18]
arXiv preprint arXiv:1711.05225 (2017)
Rajpurkar, P., Irvin, J., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D., Bagul, A., Langlotz, C., Shpanskaya, K., et al.: Chexnet: Radiologist-level pneumonia de- tection on chest x-rays with deep learning. arXiv preprint arXiv:1711.05225 (2017)
2017 arXiv
-
[19]
In: Proceedings of the Conference on Computer Vision and Pattern Recognition
Torralba, A., Efros, A.A., et al.: Unbiased look at dataset bias. In: Proceedings of the Conference on Computer Vision and Pattern Recognition. vol. 1, p. 7. Citeseer (2011)
2011
-
[20]
CoRR abs/1705.02315 (2017)
Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., Summers, R.M.: Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised clas- sification and localization of common thorax diseases. CoRR abs/1705.02315 (2017)
2017 arXiv
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.