{"id":"409cdef6-5454-4ec1-bd54-68060deb3fe6","arxiv_id":"2505.05291","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Across seven fundus datasets, a general-purpose ImageNet-pretrained Vision Transformer (iBOT) performs at least as well as retina-pretrained foundation models for detecting moderate-to-late AMD.","lead":"This preprint benchmarks six self-supervised vision models, plus a no-pretraining baseline, for detecting intermediate-to-late age-related macular degeneration across seven retinal image datasets. It reports that a natural-image-pretrained model (iBOT) matches or beats retina-specific foundation models on average and releases a new public dataset from Brazil, BRAMD.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"In-domain pretraining does add value within the same architecture: RETFound (MAE + 900k fundus images) beats MAE (IN-1k only) by about 0.06 average AUROC, so the claim that in-domain pretraining is unnecessary confounds domain with pretraining-data scale.","rationale":"The paper contributes a valuable multi-dataset benchmark and a new open dataset (BRAMD), and it reports bootstrapped AUROCs with standard deviations. My stress-test pass converges on the reader's weakest assumption: the benchmark cannot isolate the effect of pretraining domain because the models differ in architecture (VisionFM is ViT-B) and, more importantly, in total pretraining data volume. The cleanest internal evidence is the MAE vs RETFound pair: RETFound is initialized from the same MAE IN-1k weights and then pretrained on about 900k fundus images, so the only deliberate difference is the addition of in-domain data. Table 4 shows RETFound improves average AUROC by about 0.06 over MAE, which directly contradicts the 'in-domain pretraining is unnecessary' conclusion. The comparison that favors natural models (iBOT/DINOv2 vs RETFound) also involves a roughly 15x difference in pretraining corpus size, so scale, not domain, is a plausible driver. I also noticed an internal inconsistency: the Results claim general models were better in five of six OOD domains, but Table 4 shows in-domain models achieve the best AUROC on RFMiD1 and BRAMD. A concrete paired test on the reported bootstrap distributions would settle whether the in-domain increment is significant and force a more careful conclusion. The paper is still conditionally acceptable if the authors reframe the claim and add the missing statistical comparison.","tokens_in":11456,"tokens_out":14825,"duration_ms":128297,"concrete_test":"Using the already reported bootstrapped AUROC distributions, perform a paired bootstrap or Wilcoxon signed-rank test on the six OOD datasets comparing MAE and RETFound. If RETFound is significantly better, then adding about 900k in-domain fundus images to the same ViT-L/IN-1k initialization improves OOD AMD detection, directly contradicting the conclusion that in-domain pretraining is unnecessary. As a secondary check, recompute the per-domain winner counts in Table 4 to verify the 'five out of six' statement in the Results.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim that in-domain pretraining is unnecessary is not supported by its own Table 4 because the comparison conflates pretraining domain with pretraining data volume. Within the same ViT-L/16 architecture, MAE pretrained on IN-1k only achieves mean AUROC 0.818 over all seven datasets and 0.800 over the six OOD datasets, while RETFound, which is initialized from MAE's IN-1k weights and additionally pretrained on about 900k fundus images, achieves 0.879 mean and 0.865 OOD. The roughly 0.06 OOD improvement is precisely the incremental value of adding ophthalmic pretraining data. That iBOT and DINOv2, trained on IN-1k plus IN-22k (about 14M images), outperform RETFound likely reflects larger total pretraining data, not the irrelevance of domain. Without controlling for pretraining corpus size, the abstract's 'challenge the assumption that in-domain pretraining is necessary' does not follow. Additionally, the Results text states that general models performed better in five of six target domains, but Table 4 shows in-domain models (RETFound or VisionFM) achieve the highest AUROC on RFMiD1 and BRAMD, so the correct count is four of six; this internal inconsistency further weakens the empirical claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper benchmarks six self-supervised vision transformer foundation models (MAE, Mugs, iBOT, DINOv2, RETFound, and VisionFM) plus a randomly initialized ViT-L baseline, all fine-tuned on the AREDS training split, for binary classification of intermediate-to-late age-related macular degeneration in seven fundus image datasets totaling roughly 70,000 images. The authors report AUROC on AREDS-test and six external out-of-distribution datasets, introduce a new open-access Brazilian dataset BRAMD (n=587), and present AMDNet, which uses an iBOT backbone with a leave-one-domain-out multi-source fine-tuning strategy and is compared against DeepSeeNet. The headline claims are that iBOT, pretrained on natural images, achieves the highest out-of-distribution generalization and that in-domain ophthalmic pretraining is not necessary for this task.","tokens_in":11753,"tokens_out":8272,"duration_ms":76805,"significance":"If the central comparison were unconfounded, the paper would provide a valuable, reproducible benchmark for AMD detection from fundus images. The BRAMD dataset, the use of seven diverse datasets, bootstrap standard deviations, and the public release of AMDNet are concrete contributions. The paper also repeats an important and contested observation in the community: large natural-image SSL models can match or exceed retinal-specific foundation models on a downstream ophthalmic task. However, the strength of the claim is currently undermined by uncontrolled confounds between pretraining domain, model architecture, and pretraining data volume, by an internal inconsistency in the reported per-dataset win counts, and by an undefined aggregation measure in the headline comparison. The scientific contribution is therefore conditional on the authors substantially reframing or re-analyzing the evidence.","major_comments":[{"comment":"The central claim that in-domain pretraining is unnecessary is confounded with pretraining-data volume and architecture. On the same ViT-L/16 architecture, MAE pretrained only on ImageNet-1k has a mean OOD AUROC of about 0.800 over the six external datasets, while RETFound, which is initialized from MAE and further pretrained on about 900k fundus images, reaches 0.865, an improvement of about 0.06 that demonstrates the incremental value of in-domain pretraining at fixed architecture. The advantage of iBOT and DINOv2 over RETFound likely reflects pretraining on ImageNet-1k plus ImageNet-22k (about 14M images) rather than the irrelevance of domain, and VisionFM additionally uses a much smaller ViT-B backbone (86M parameters). Without matching architecture and pretraining corpus size, the abstract's statement that the results 'challenge the assumption that in-domain pretraining is necessary' is not supported by the data as presented.","section":"Abstract; §3, Table 3; §4, Table 4"},{"comment":"The Results text states that general foundation models performed better in five of six target domains and that only on BRAMD did VisionFM outperform the general foundation models; Table 4 contradicts this. On RFMiD1, VisionFM (0.872) is the top fine-tuned model, above the best general model DINOv2 (0.864). On BRAMD, RETFound (0.960) is the top model, above iBOT (0.952), while VisionFM (0.943) is below iBOT. The correct count of OOD datasets where a general foundation model attains the highest AUROC is four (HYAMD, ADAM, FIVES, STARE), not five, and the 'only in the case of BRAMD' sentence is inaccurate. This internal inconsistency directly weakens the empirical basis for the general-versus-in-domain conclusion.","section":"§4, Results; Table 4"},{"comment":"The aggregation measure underlying the headline comparison is undefined. The Performance Measures paragraph states that Figure 1a reports the median AUROC across the seven domains, while the Figure 1a caption and the Results text describe the same quantity as the average AUROC. The distinction matters: for the two leading models, the mean across the seven datasets is approximately 0.889 for iBOT and 0.888 for DINOv2, whereas the median is 0.900 for both, tying the two models. The authors must specify whether mean or median is used, and they should provide a paired significance test or otherwise quantify the uncertainty of the difference, because the raw 0.002 mean-AUROC margin is within the bootstrap standard deviations reported in Table 4.","section":"§3, Performance Measures; §4, Results, Figure 1a"},{"comment":"The AMDNet out-of-distribution evaluation is selection-optimistic. The iBOT backbone was chosen after evaluating average AUROC over all seven datasets, including each external target domain that subsequently serves as the left-out test domain in the leave-one-domain-out protocol. This uses held-out information during model selection and likely inflates the reported OOD AUROC of AMDNet. A nested model-selection procedure, or an explicit statement that the reported AMDNet numbers are optimistic upper bounds affected by selection on the test domains, is required before AMDNet's superiority over DeepSeeNet can be assessed fairly.","section":"§3, Multi-source domain training; §4, AMDNet"}],"minor_comments":[{"comment":"The sentence 'foundation models outperformed the baseline ViT-L (p<0.05)' does not state which statistical test was used, whether the comparison was paired, or whether any multiple-comparison correction was applied; please provide this information.","section":"§4, Results"},{"comment":"The caption says 'The highest out-of-distribution (OOD) results are marked with †,' but the dagger symbols are attached to all AMDNet rows, not to the highest OOD entries among the six foundation models, and the AREDS-test column is in-domain rather than OOD; please revise the caption so the marker convention is unambiguous.","section":"Table 4 caption"},{"comment":"The BRAMD control group is described as diabetic retinopathy patients, so 'non-AMD' in this dataset does not mean healthy controls; please state explicitly in Table 1 or the dataset description that controls may have other retinal pathology, as this affects interpretation of the non-AMD class.","section":"§2, BRAMD"},{"comment":"The model name is written inconsistently as 'Dinov2' in several places (e.g., Table 3 and the Results text) and 'DINOv2' elsewhere; please standardize the capitalization.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful dataset and a broadly transparent benchmark, but the main scientific claim is currently stronger than the evidence. The within-architecture MAE-versus-RETFound comparison actually points in the opposite direction to the abstract's claim about in-domain pretraining, and the AMDNet selection protocol leaks test-domain information. These are fixable within the manuscript's scope by reframing the conclusions, adding matched-scale or within-architecture analyses, and reporting a significance test for the iBOT-versus-DINOv2 difference. I would not reject the paper, but it needs substantial revision before the central claims can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe useful part of this paper is the benchmark and the BRAMD dataset; the headline conclusion about in-domain pretraining is not supported by the paper's own numbers. You'll want to cite this for the empirical comparison, not for the claim.\n\nWhat's new: six SSL-pretrained ViTs, seven datasets, ~70k images, all fine-tuned on AREDS with the same protocol, bootstrap CIs, and a new open-access Brazilian dataset (BRAMD, n=587, OCT-confirmed). That's a solid contribution. The AMDNet model, which uses iBOT with a leave-one-dataset-out multi-source fine-tuning, shows a meaningful gain over single-source fine-tuning (about 5% AUROC) and clearly beats DeepSeeNet.\n\nThe soft spots are real. First, iBOT's edge over DINOv2 is 0.890 vs 0.888 mean AUROC—effectively a tie, and the paper gives no significance test for that comparison. Second, the natural-versus-in-domain comparison is confounded by pretraining scale and architecture: iBOT/DINOv2 used ImageNet-22k (~14M images), RETFound/VisionFM used ~1M fundus images. The paper's own Table 4 shows RETFound beats MAE by about 0.06 AUROC on the OOD average (0.865 vs 0.800) even though RETFound is initialized from MAE's weights and then pretrained on 900k fundus images. That gap is the value of in-domain data; the paper's claim doesn't follow.\n\nThere's also an internal inconsistency: the results text says general models performed better in five of six target domains, but Table 4 shows in-domain models win on RFMiD1 (VisionFM) and BRAMD (RETFound)—so the correct count is four of six. That needs fixing. Finally, AMDNet's OOD numbers come after the same external datasets were used to pick iBOT as the backbone, so those values are optimistically selected; an independent validation split or external center would be needed to trust the absolute OOD figures.\n\nThe central benchmark—six models fine-tuned identically on AREDS—is clean and not circular. The problems are in the interpretation and the model-selection protocol. I'd send this to review, but I'd insist the authors address the scale confound, fix the five-of-six error, add significance testing for iBOT vs DINOv2, and present AMDNet with a proper held-out evaluation. With those changes, the paper would be a useful reference; as it stands, the abstract overclaims.\n\nRecommendation: accept to peer review, expect major revision.","headline":"Useful benchmark and a genuinely new dataset, but the headline claim that in-domain pretraining is unnecessary is confounded and contradicted by the paper's own table.","tokens_in":12298,"tokens_out":3565,"would_cite":true,"duration_ms":31817,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Across seven fundus-image datasets, a natural-image self-supervised model achieves the best out-of-distribution AUROC for clinically significant AMD detection, beating two retina-pretrained foundation models and a no-pretraining baseline.","keywords":["age-related macular degeneration","foundation models","self-supervised learning","vision transformers","digital fundus images","out-of-distribution generalization","AMDNet","BRAMD"],"falsifier":"Run the same seven-dataset benchmark with a retina-pretrained vision transformer that matches iBOT's architecture and pretraining scale (about 307M parameters, roughly one million fundus images); if that model matches or exceeds iBOT's out-of-distribution AUROCs, the claim that in-domain pretraining is unnecessary is refuted. The complementary check is to shrink iBOT to the size of the smaller in-domain model and see whether its advantage disappears.","tokens_in":11275,"feed_emoji":"👁️","tokens_out":11668,"duration_ms":102789,"temperature":0.7,"pith_summary":"This paper asks whether self-supervised pretraining on retinal images is necessary for detecting clinically significant (intermediate-to-late) age-related macular degeneration from digital fundus photographs. The authors benchmark six pretrained vision transformers on 70,000 expert-annotated images from seven datasets, fine-tuning all models on the training split of the large AREDS dataset and testing on six external domains. They find that iBOT, a vision transformer pretrained on natural images with self-supervised learning, achieves the highest out-of-distribution AUROCs (area under the ROC curve, a standard discrimination score) of 0.80-0.97, better than retina-pretrained models (0.78-0.96) and a no-pretraining baseline (0.68-0.91). If true, the result means that general natural-image pretraining is sufficient for this task and that building large in-domain ophthalmic foundation models may not be necessary. The paper also releases BRAMD, a new open-access dataset of 587 fundus images from Brazil, and AMDNet, its best model obtained by fine-tuning iBOT with multi-source domain training.","feed_headline":"Natural-image model beats retina-specialized AI on AMD detection","feed_subtitle":"A general self-supervised vision transformer posted the best out-of-distribution scores across 70,000 fundus images.","key_machinery":"The argument is carried by a comparison protocol intended to isolate pretraining: six self-supervised pretrained vision transformers are fine-tuned with the same augmentations, learning-rate schedule, loss, and 10 epochs on the same AREDS-train split, so the pretraining recipe is the main variable. The best backbone, iBOT, is then taken through a multi-source leave-one-domain-out training stage that pools all but the target dataset and balances classes with inverse-frequency weights; the resulting model is AMDNet. AUROC (area under the receiver-operating-characteristic curve) is the ordering metric, and Grad-CAM maps are used to check that the final model focuses on the macula.","core_discovery":"On the paper's own terms, the central finding is that pretraining domain does not decide performance: iBOT, a self-supervised vision transformer pretrained only on natural images, delivers the best average AUROC (0.890) across the seven datasets and the best out-of-distribution AUROC on three of six external domains, while general natural-image models beat both retina-pretrained models in five of six external domains. Applying the same multi-source leave-one-domain-out training to iBOT yields AMDNet, which reaches out-of-distribution AUROCs of 0.842-0.977 on six unseen datasets and outperforms DeepSeeNet by 10.2% on average. The authors read these results as evidence that in-domain pretraining is not necessary for moderate-to-late AMD identification, and that the value of foundation models for this task comes from large-scale self-supervised pretraining on natural images.","pith_inferences":["If the result holds for other ophthalmic tasks, the same leave-one-dataset-out protocol could be run on diabetic retinopathy, glaucoma, or OCT data; a natural-image backbone winning there would turn this single-task finding into a general principle about medical imaging pretraining.","Because the natural-image winners are 307M-parameter ViT-L models while one in-domain competitor is an 86M-parameter ViT-B, a matched-scale comparison is the natural next experiment before concluding that in-domain pretraining is unnecessary.","The false-positive analysis implies that a deployed AMDNet would need comorbidity-aware triage: the conditions most often mistaken for AMD (macular scar, exudation, chorioretinitis, RPE changes, myopia, retinitis pigmentosa) mimic macular disease rather than random noise."],"forward_implications":["Fine-tuning only on one large dataset can transfer: a model fine-tuned solely on the AREDS training split identifies intermediate-to-late AMD across US, Indian, Israeli, Chinese, and Brazilian fundus images with out-of-distribution AUROCs of 0.80-0.97.","Retina-specific pretraining on up to about one million fundus images does not beat natural-image SSL pretraining for this task, since general models outperformed the in-domain models in five of six external domains.","The multi-source AMDNet raises the bar for an open AMD detection baseline, with out-of-distribution AUROCs of 0.842-0.977 and an average 10.2-point improvement over DeepSeeNet.","BRAMD adds a Brazilian two-device benchmark of 587 images, giving the community a new external test set for AMD models."],"supporting_citations":[{"why":"Supplies iBOT, the natural-image pretrained backbone that posts the best OOD AUROCs and becomes AMDNet.","marker":"[15]"},{"why":"Supplies RETFound, the in-domain fundus-pretrained model that the benchmark shows is outperformed.","marker":"[8]"},{"why":"Supplies VisionFM, the other in-domain ophthalmic-pretrained model outperformed on most external domains.","marker":"[9]"},{"why":"Supplies MAE, a natural-image SSL-pretrained baseline in the comparison.","marker":"[13]"},{"why":"Supplies DINOv2, the natural-image model that nearly matches iBOT and beats both in-domain models.","marker":"[16]"},{"why":"Supplies Mugs, another natural-image SSL-pretrained baseline.","marker":"[14]"},{"why":"AREDS is the large annotated dataset used for fine-tuning all models and the in-distribution test set.","marker":"[19]"},{"why":"DeepSeeNet is the open-source state-of-the-art AMD model that AMDNet is benchmarked against.","marker":"[28]"},{"why":"Establishes the multi-source leave-one-domain-out training approach that turns iBOT into AMDNet.","marker":"[5]"},{"why":"Supplies the no-pretraining ViT-L baseline whose lower AUROCs establish the value of pretraining.","marker":"[27]"}],"fun_headline_variants":["Natural-image ViT outdoes retina-specialized models on AMD","General vision model beats retina-specialist on AMD","Natural pretraining wins over retina-specific for AMD detection","iBOT, trained on nature, tops retina-specialized AMD models","General SSL ViT beats retina-pretrained on AMD datasets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the six models differ only in pretraining domain, but they also differ in model size and pretraining data volume, so the natural-image advantage could be a scale effect rather than a domain effect.","fun_headline_variants_meta":{"raw":{"variants":["Natural-image ViT outdoes retina-specialized models on AMD","General vision model beats retina-specialist on AMD","Natural pretraining wins over retina-specific for AMD detection","iBOT, trained on nature, tops retina-specialized AMD models","General SSL ViT beats retina-pretrained on AMD datasets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001331,"raw_usage":{"total_tokens":5415,"prompt_tokens":950,"completion_tokens":4465,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":4379}},"tokens_in":566,"tokens_out":4465,"duration_ms":30525,"temperature":1.0,"reasoning_tokens":4379,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:07:45.597297+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same seven-dataset benchmark with a retina-pretrained vision transformer that matches iBOT's architecture and pretraining scale (about 307M parameters, roughly one million fundus images); if that model matches or exceeds iBOT's out-of-distribution AUROCs, the claim that in-domain pretraining is unnecessary is refuted. The complementary check is to shrink iBOT to the size of the smaller in-domain model and see whether its advantage disappears.","supporting_citations":[{"cited_title":"Qiu et al., Development and validation of a multimodal multitask vision foundation model for generalist ophthalmic artificial intelligence, NEJM AI 1 (12) (2024) AIoa2300221","cited_arxiv_id":null,"evidence_quote":"Supplies VisionFM, the other in-domain ophthalmic-pretrained model outperformed on most external domains."},{"cited_title":"[On- line]","cited_arxiv_id":null,"evidence_quote":"AREDS is the large annotated dataset used for fine-tuning all models and the in-distribution test set."},{"cited_title":"Men et al., Deep learning generalization for diabetic retinopathy staging from fundus images, Physiological Mea- surement 13 (1) (2025).doi:10.1088/1361-6579/ada86a","cited_arxiv_id":null,"evidence_quote":"Establishes the multi-source leave-one-domain-out training approach that turns iBOT into AMDNet."}],"review_version":1}