REVIEW 4 major objections 5 minor 35 references
Is an Ultra Large Natural Image-Based Foundation Model Superior to a Retina-Specific Model for Detecting Ocular and Systemic Diseases?
T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read Task decides which eye-AI model wins: DINOv2 or RETFound.
desk verdict Useful head-to-head benchmark, but the bootstrap SE is miscalculated by a factor of 10, so every significance claim in the abstract is overconfident and needs reanalysis. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The comparison is carried by two pre-trained vision transformers with different pretraining diets: DINOv2, a general-purpose model self-supervised on 142 million natural images through self-distillation, and RETFound, a masked-autoencoder model pre-trained on natural images plus about 1.6 million retinal colour and OCT images. Both are adapted with the same fine-tuning pipeline—MLP head, 100 epochs, cosine learning-rate schedule, AdamW—taken from the RETFound study and applied identically to every model. What the paper adds is a head-to-head benchmark spanning eight open-source ocular datasets and two linked large cohorts, with AUROC as the primary metric and bootstrap/t-test comparisons for significance.
What would settle it
Recompute the AUROC standard errors and p-values from the paper's 100 bootstrap replicates using $\mathrm{SE} = \mathrm{SD}/\sqrt{100}$ and the full test set rather than 20% subsamples; if the diabetic-retinopathy and multi-class eye-disease comparisons no longer reach $P<0.017$ after Bonferroni correction, the paper's headline claim that DINOv2 outperforms RETFound in ocular detection would fail.
Extended reading notes
Core claim
The central discovery is a task-dependent performance split between two families of foundation models. Across three diabetic-retinopathy datasets, DINOv2-large reaches AUROC 0.850–0.952 versus RETFound's 0.823–0.944 (all p≤0.007); on a multi-class retinal-disease dataset it scores 0.892 versus 0.846 (p<0.001); and on a glaucoma dataset DINOv2-base scores 0.958 versus RETFound's 0.940 (p<0.001). For systemic disease incidence, the reverse holds: RETFound predicts heart failure, myocardial infarction, and ischaemic stroke with AUROCs of 0.796, 0.732, and 0.754, beating every DINOv2 variant (0.663–0.771, all p<0.001), and the same direction persists in external validation and under reduced fine-tuning data.
Load-bearing premise
The ranking of models rests on a bootstrap standard-error calculation that divides the observed standard deviation by 100 instead of by $\sqrt{100}$; if that calculation is corrected, many of the reported AUROC differences may no longer be statistically significant.
Editorial extensions
If this is right
- If the results hold, retina-specialist pretraining is not universally superior; for detecting well-defined retinal lesions, a large general-purpose model is at least as good, often better, and its small and base variants need less compute.
- For oculomics—predicting cardiovascular outcomes from retinal images—domain-specific pretraining appears to add real signal, making RETFound the better default for systemic risk prediction.
- The label-efficiency analyses imply that DINOv2 can hold its performance with only 10% of fine-tuning data in several ocular detection tasks, which matters where annotated ophthalmic data are scarce.
- The task-dependence conclusion gives clinicians and model builders a concrete selection rule: match the foundation model to whether the task is focal-lesion detection or subtle systemic-risk inference.
- Because both models beat training from scratch in the paper's comparisons, the findings also reinforce that large-scale pretraining—natural or retinal—is the main driver of downstream gains.
Reading between the lines
- The paper does not disentangle pretraining scale from pretraining content; an equally plausible reading is that DINOv2's advantage comes from seeing 142 million images, not from seeing natural images per se, and a controlled test would require pre-training a retina model at the same scale.
- The reported AUROC gaps for diabetic retinopathy are numerically small (around 0.01–0.03), and the paper itself notes that overlapping confidence intervals may limit clinical meaning; 'statistically significant' and 'clinically superior' should be treated as distinct claims.
- The authors' interpretation predicts that RETFound's advantage will generalise to other oculomics outcomes, such as dementia or kidney disease, and to other imaging modalities like ultra-widefield retinal imaging; that is a testable extension, not a proven fact.
- Before relying on the task-dependent selection rule, a reader should verify the bootstrap standard-error calculation, since the reported confidence intervals and p-values depend on the exact resampling formula used.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript compares three DINOv2 vision foundation models (Small, Base, Large), pretrained on natural images, with RETFound, a retina-specific foundation model, across ocular disease detection (diabetic retinopathy, glaucoma, multi-class eye diseases) and systemic disease incidence prediction (heart failure, myocardial infarction, ischemic stroke). The authors fine-tune all models using the RETFound protocol on eight open-source ocular datasets, the Moorfields AlzEye dataset, and external evaluation on UK Biobank. They report that DINOv2-Large outperforms RETFound for diabetic retinopathy and multi-class eye disease, DINOv2-Base outperforms RETFound for glaucoma, and RETFound outperforms all DINOv2 variants for systemic disease prediction. They also report label-efficiency analyses and calibration comparisons. The central practical claim is that the best foundation model depends on the clinical task, so model selection should be task-specific.
Significance. If the reported comparisons are valid, the study would provide a useful head-to-head benchmark of a general-purpose vision foundation model versus a widely used domain-specific medical foundation model. Its strengths include replication of the RETFound fine-tuning protocol, use of eight public ocular datasets plus two large linked cohorts, external validation, release of code, and explicit attention to calibration and label efficiency. The finding that a natural-image model can sometimes match or exceed a retina-pretrained model, while losing on systemic oculomics tasks, is practically relevant for model selection in resource-constrained settings. However, the statistical inference underlying the headline significance claims is flawed, and the magnitude of the reported AUROC differences is often small; the conclusions therefore require a corrected analysis before the significance-based claims can be accepted.
major comments (4)
- [Methods, 'Performance evaluation and statistical analysis'] The bootstrap standard error is computed incorrectly. The text states: 'The standard error was obtained by dividing the standard deviation by 100.' With 100 bootstrap replicates, the standard error of the mean is SD/sqrt(100) = SD/10, not SD/100. This makes the reported SEs ten times too small, narrows all 95% confidence intervals by a factor of ten, and inflates every two-sided t-statistic by a factor of ten. For example, the APTOS DR comparison (0.952 vs 0.944, P<0.001) has a raw difference of 0.008; with a corrected SE this difference may no longer be significant at the Bonferroni threshold of 0.017. Because this error affects every significance claim in the Abstract and Results, a corrected statistical analysis is required before the central claim of task-dependent superiority can be supported.
- [Methods, 'Performance evaluation and statistical analysis'] The bootstrap procedure resamples 20% of the test data in each of 100 replicates. This estimates the variability of an AUROC computed on a random 20% subset, not the sampling variability of the reported full-test AUROC. The resulting interval is therefore not a valid confidence interval for the reported point estimate, even if the SE divisor were corrected. The authors should either bootstrap the full test set (or an appropriate analytical AUROC standard error such as DeLong's method) or explicitly reframe the intervals as describing partial-sample variability.
- [Results, 'Comparative Performance of Models in Ocular Disease Detection Tasks' and 'Comparative Performance of Models…] The best-performing DINOv2 variant is selected post hoc for each task: DINOv2-Large is highlighted for DR and multi-class eye disease, DINOv2-Base for glaucoma and ischemic stroke, and DINOv2-Base or DINOv2-Large for other systemic outcomes. Since three DINOv2 variants are compared against RETFound, selecting the best variant per outcome after seeing the results creates a multiple-comparisons problem that is not fully addressed by Bonferroni correction for three pairwise comparisons. The headline 'DINOv2 outperforms RETFound' should be qualified by reporting the analysis for a pre-specified variant or by applying an appropriate multiple-testing correction across all variants and tasks.
- [Discussion, fourth paragraph] The Discussion itself concedes that 'the observed AUC differences in diabetic retinopathy, are numerically modest' and that 'some of these differences may reach statistical significance, they may not always reflect meaningful clinical superiority, particularly when confidence intervals overlap.' This is in direct tension with the Abstract and Results, which present the DR differences (e.g., 0.008–0.023 AUROC) as statistically significant and task-defining. Given the flawed SE computation, the manuscript should be revised to present the DR comparison as showing comparable performance with modest, possibly non-significant differences, unless the corrected analysis establishes otherwise.
minor comments (5)
- [Abstract and Results, DR section] The abstract reports 'all P<=0.007' for DR, which is consistent with the reported per-dataset p-values (P<0.001, P<0.001, P=0.007), but the corrected SE analysis is needed to know whether these values remain valid; the reporting should be updated after re-analysis.
- [Results, systemic disease section] The sentence 'outperforming all DINOv2 models (AUROC: 0.523-0.559, all<0.001)' is missing the 'P' before the inequality; it should read 'all P<0.001'.
- [Results, systemic disease section] The text refers to 'Figure 7 & Figure S8-S9' but the systemic-disease internal-test results are described in prose as 'Figure 7 and Figure S8'; the figure numbering and cross-references should be checked for consistency.
- [Discussion, third paragraph] The Discussion mentions 'this trend was not fully replicated in all external validations (Figure S4 and S5)', but Figure S4 is described as confusion matrices and Figure S5 as cross-dataset DR results; the intended figure references should be corrected.
- [Methods, 'Performance evaluation and statistical analysis'] Using the Shapiro-Wilk test on 100 bootstrap replicates is statistically unusual because bootstrap replicates are not independent samples from a population; normality assessment of the bootstrap distribution is better done graphically (e.g., histogram or Q-Q plot) and does not justify a t-test on the replicates.
Circularity Check
No significant circularity: the central comparisons are empirical benchmarks against external pretrained weights and held-out test data.
full rationale
The paper's central claims are direct empirical measurements, not derived quantities. DINOv2 and RETFound are pretrained models released before this study, and the paper fine-tunes and evaluates them on held-out test splits of external datasets (APTOS, IDRiD, MESSIDOR2, PAPILA, Glaucoma Fundus, JSIEC, Retina, OCTID, AlzEye, and UK Biobank). No parameter is fitted to a subset and then renamed as a prediction; no target result is encoded in the model definition or the evaluation protocol by construction. The use of RETFound, AlzEye, and UK Biobank involves overlapping authorship, but the paper actually runs RETFound on these data rather than importing its performance by citation, so the comparison is not forced by a self-citation chain. The bootstrap standard-error computation (SD/100 instead of SD/sqrt(100)) is a statistical validity concern that could overstate significance, but it is a correctness issue, not circularity: correcting the error would change confidence intervals and p-values without making the outcome equivalent to the input. The paper is therefore self-contained as an empirical benchmark and shows no circular derivation.
Assumptions & free parameters
assumptions (3)
- domain assumption The publicly released pretrained weights for DINOv2 and RETFound are the correct, intended models and are loaded without error.
- domain assumption The disease labels in the open-source datasets and the systemic outcome labels derived from Hospital Episode Statistics and UK Biobank linkage are sufficiently accurate for benchmarking.
- domain assumption The fine-tuning hyperparameters, augmentation, and data splits from the original RETFound study transfer fairly to DINOv2 models.
Cite this review
Pith. "Pith review of Is an Ultra Large Natural Image-Based Foundation Model Superior to a Retina-Specific Model for Detecting Ocular and Systemic Diseases?." pith.science (2026). https://pith.science/paper/2FPH6SWL
@misc{pith2026250206289,
author = {Pith},
title = {Pith review of: Is an Ultra Large Natural Image-Based Foundation Model Superior to a Retina-Specific Model for Detecting Ocular and Systemic Diseases?},
year = {2026},
howpublished = {\url{https://pith.science/paper/2FPH6SWL}},
note = {Machine review of arXiv:2502.06289}
}
read the original abstract
The advent of foundation models (FMs) is transforming medical domain. In ophthalmology, RETFound, a retina-specific FM pre-trained sequentially on 1.4 million natural images and 1.6 million retinal images, has demonstrated high adaptability across clinical applications. Conversely, DINOv2, a general-purpose vision FM pre-trained on 142 million natural images, has shown promise in non-medical domains. However, its applicability to clinical tasks remains underexplored. To address this, we conducted head-to-head evaluations by fine-tuning RETFound and three DINOv2 models (large, base, small) for ocular disease detection and systemic disease prediction tasks, across eight standardized open-source ocular datasets, as well as the Moorfields AlzEye and the UK Biobank datasets. DINOv2-large model outperformed RETFound in detecting diabetic retinopathy (AUROC=0.850-0.952 vs 0.823-0.944, across three datasets, all P<=0.007) and multi-class eye diseases (AUROC=0.892 vs. 0.846, P<0.001). In glaucoma, DINOv2-base model outperformed RETFound (AUROC=0.958 vs 0.940, P<0.001). Conversely, RETFound achieved superior performance over all DINOv2 models in predicting heart failure, myocardial infarction, and ischaemic stroke (AUROC=0.732-0.796 vs 0.663-0.771, all P<0.001). These trends persisted even with 10% of the fine-tuning data. These findings showcase the distinct scenarios where general-purpose and domain-specific FMs excel, highlighting the importance of aligning FM selection with task-specific requirements to optimise clinical performance.
Reference graph
Works this paper leans on
-
[1]
Azad,B.,Azad,R.,Eskandari,S.,Bozorgpour,A.,Kazerouni,A.,Rekik,I.,&Merhof,D. (2023). Foundational models in medical imaging: A comprehensive survey and future vision. arXivpreprintarXiv:2310.18689
arXiv 2023
-
[2]
(2023).Gpt-4technicalreport.arXivpreprintarXiv:2303.08774
Achiam,J.,Adler,S.,Agarwal,S.,Ahmad,L.,Akkaya,I.,Aleman,F.L.,...&McGrew,B. (2023).Gpt-4technicalreport.arXivpreprintarXiv:2303.08774
arXiv 2023
-
[3]
Moor, M., Banerjee, O.,Abad, Z. S. H., Krumholz, H. M., Leskovec, J., Topol, E. J., & Rajpurkar, P. (2023). Foundation models for generalist medical artificial intelligence. Nature, 616(7956),259-265
work page 2023
- [4]
-
[5]
Low-resource finetuning of foundation models beats state-of-the-art in histopathology
Roth, B., Koch, V., Wagner, S. J., Schnabel, J. A., Marr, C., & Peng, T. (2024). Low- resource finetuning of foundation models beats state-of-the-art in histopathology. arXiv preprint arXiv:2401.04720
work page Pith review arXiv 2024
-
[6]
Awais,M.,Naseer,M.,Khan,S.,Anwer,R.M.,Cholakkal,H.,Shah,M.,...&Khan,F.S. (2025). Foundation Models Defining a New Era in Vision: a Survey and Outlook. IEEE TransactionsonPatternAnalysisandMachineIntelligence
work page 2025
-
[7]
S.,Williamson,D.J.,Struyven,R.R.,
Zhou,Y.,Chia,M.A.,Wagner,S.K.,Ayhan,M. S.,Williamson,D.J.,Struyven,R.R.,... &Keane,P.A.(2023).Afoundationmodelforgeneralizablediseasedetectionfromretinalimages. Nature,622(7981),156-163
work page 2023
-
[8]
Han,K.,Wang,Y.,Chen,H.,Chen,X.,Guo,J.,Liu,Z., ...&Tao,D.(2022).Asurveyon visiontransformer.IEEEtransactionsonpatternanalysisandmachineintelligence,45(1),87-110
work page 2022
Show all 35 references
-
[9]
Dosovitskiy, A. (2020). An image is worth 16x16 words: Transformers for image recognitionatscale.arXivpreprintarXiv:2010.11929
2020 arXiv
-
[10]
He, K., Chen, X., Xie, S., Li,Y., Dollár, P., & Girshick, R. (2022). Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and patternrecognition(pp.16000-16009)
2022
-
[11]
RETFound-enhancedcommunity-basedfundusdisease screening: real-world evidence and decision curve analysis
Zhang, J.,Lin, S., Cheng,T.et al. RETFound-enhancedcommunity-basedfundusdisease screening: real-world evidence and decision curve analysis. npj Digit. Med. 7, 108 (2024). https://doi.org/10.1038/s41746-024-01109-5
2024 doi
-
[12]
Teresa Araújo, Guilherme Moreira Aresta, Ursula Schmidt-Erfurth, Hrvoje Bogunovic; Robust deep learning for automated AMD screening and staging based on a retinal foundation model.Invest.Ophthalmol.Vis.Sci.2024;65(7):2833
2024
-
[13]
NikhilVarmaGadiraju,DavidKuo,VahidOwnagh,TerryLee,LindseyAChew,GrantA. Justin, Nita Valikodath, Stephanie Hsu, Miroslav Pajic, Majda Hadziahmetovic, Lejla Vajzovic; Evaluating Performance of ResNet-50 and RETFound Image ClassificationAlgorithms on Ultra- widefieldFundusImages....
2024
-
[14]
& Bojanowski, P
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., ... & Bojanowski, P. (2023). Dinov2: Learning robust visual features without supervision. arXiv preprintarXiv:2304.07193
2023 arXiv
-
[15]
& Alber, M
Dippel, J., Feulner, B., Winterhoff, T., Schallenberg, S., Dernbach, G., Kunft, A., ... & Alber, M. (2024). RudolfV: a foundation model by pathologists for pathologists. arXiv preprint arXiv:2401.04079
2024 arXiv
-
[16]
& Poon, H
Xu, H., Usuyama, N., Bagga, J., Zhang, S., Rao, R., Naumann,T., ... & Poon, H. (2024). Awhole-slidefoundationmodelfordigitalpathologyfromreal-worlddata.Nature,1-8
2024
-
[17]
& Zhang, D
Fan, W., Yang, Y., Qi, J., Zhang, Q., Liao, C., Wen, L., ... & Zhang, D. (2024).A deep- learning-based framework for identifying and localizing multiple abnormalities and assessing cardiomegalyinchestX-ray.NatureCommunications,15(1),1347
2024
-
[18]
& Fei-Fei, L
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., ... & Fei-Fei, L. (2015).Imagenetlargescalevisualrecognitionchallenge.Internationaljournalofcomputervision, 115,211-252
2015
-
[19]
Porwal,P.,Pachade,S.,Kokare,M.,Deshmukh,G.,Son,J.,Bae,W.,...&Meriaudeau,F. (2020). Idrid: Diabetic retinopathy–segmentation and grading challenge. Medical image analysis, 59,101561
2020
-
[20]
Indian diabetic retinopathy image dataset (IDRiD): a database for diabetic retinopathyscreeningresearch.Data,3(3),25
Porwal, P., Pachade, S., Kamble, R., Kokare, M., Deshmukh, G., Sahasrabuddhe, V., & Meriaudeau, F.(2018). Indian diabetic retinopathy image dataset (IDRiD): a database for diabetic retinopathyscreeningresearch.Data,3(3),25
2018
-
[21]
& Klein, J
Decencière, E., Zhang, X., Cazuguel, G., Lay,B., Cochener, B., Trone, C., ... & Klein, J. C. (2014). Feedback on a publicly distributed image database: the Messidor database. Image Analysis&Stereology,231-234
2014
-
[22]
D., Folk, J
Abràmoff, M. D., Folk, J. C., Han,D. P.,Walker,J. D.,Williams, D. F., Russell, S. R., ... & Niemeijer, M. (2013).Automated analysis of retinal images for detection of referable diabetic retinopathy.JAMAophthalmology,131(3),351-357
2013
-
[23]
Krause, J., Gulshan, V., Rahimy, E., Karth, P., Widner, K., Corrado, G. S., ... & Webster, D. R.(2018).Gradervariability and theimportanceof referencestandardsfor evaluating machine learningmodelsfordiabeticretinopathy.Ophthalmology,125(8),1264-1272
2018
-
[24]
Available from: https://kaggle.com/competitions/aptos2019-blindness-detection
APTOS 2019 Blindness Detection [Internet]. Available from: https://kaggle.com/competitions/aptos2019-blindness-detection
2019
-
[25]
Kovalyk, O., Morales-Sánchez, J., Verdú-Monedero, R., Sellés-Navarro, I., Palazón- Cabanes, A., & Sancho-Gómez, J. L. (2022). PAPILA: Dataset with fundus images and clinical dataofbotheyesofthesamepatientforglaucomaassessment.ScientificData,9(1),291
2022
-
[26]
M., Kim, S., Ahn, K
Ahn, J. M., Kim, S., Ahn, K. S., Cho, S. H., Lee, K. B., & Kim, U. S. (2018). A deep learning model for the detection of both advanced and early glaucoma using fundus photography. PloSone,13(11),e0207982
2018
-
[27]
P., Ji, J., Lin, J
Cen, L. P., Ji, J., Lin, J. W., Ju, S. T., Lin, H. J., Li, T. P., ... & Zhang, M. (2021). Automaticdetectionof39fundusdiseasesandconditionsinretinalphotographsusingdeepneural networks.Naturecommunications,12(1),4828
2021
-
[28]
Retina[Internet].Availablefrom:https://www.kaggle.com/datasets/jr2ngb/cataractdataset
-
[29]
K., & Lakshminarayanan, V
Gholami, P., Roy, P., Parthasarathy, M. K., & Lakshminarayanan, V. (2020). OCTID: Opticalcoherencetomographyimagedatabase.Computers&ElectricalEngineering,81,106532
2020
-
[30]
K., Hughes, F., Cortina-Borja, M., Pontikos, N., Struyven, R., Liu, X.,
Wagner, S. K., Hughes, F., Cortina-Borja, M., Pontikos, N., Struyven, R., Liu, X., ... & Keane,P.A.(2022).AlzEye:longitudinalrecord-levellinkageofophthalmicimagingandhospital admissionsof353157patientsinLondon,UK.BMJopen,12(3),e058552
2022
-
[31]
Huang,Y., Zou, J., Meng, L.,Yue, X., Zhao, Q., Li, J., ... & Fu, G. (2024). Comparative Analysis of ImageNet Pre-Trained Deep Learning Models and DINOv2 in Medical Imaging Classification.arXivpreprintarXiv:2402.07595
2024 arXiv
-
[32]
F., Restrepo, D., Matos, J., Ribeiro, L
Nakayama, L. F., Restrepo, D., Matos, J., Ribeiro, L. Z., Malerbi, F. K., Celi, L. A., & Regatieri, C. S. (2024). BRSET: a Brazilian multilabel ophthalmological dataset of retina fundus photos.medRxiv
2024
-
[33]
Baharoon, M., Qureshi, W., Ouyang, J., Xu, Y., Aljouie, A., & Peng, W. (2023). Evaluatinggeneralpurposevisionfoundationmodelsformedicalimageanalysis:Anexperimental studyofdinov2onradiologybenchmarks.arXivpreprintarXiv:2312.02366
2023 arXiv
-
[34]
D., KS, S., Patil, U., Bhushan,C.,
Anand,D., Singhal, V., Shanbhag, D. D., KS, S., Patil, U., Bhushan,C., ... & Kass-Hout, T. (2023). One-shot localization and segmentation of medical images with foundation models. arXivpreprintarXiv:2310.18642
2023 arXiv
-
[35]
Topol, E. J. (2019). High-performance medicine: the convergence of human and artificial intelligence.Naturemedicine,25(1),44-56
2019
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.