REVIEW 3 major objections 4 minor 49 references
An integrated language-vision foundation model for conversational diagnostics and triaging in primary eye care
T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read AI chat tool reads eye photos at ophthalmologist level
desk verdict A serious engineering integration, but the external validation is partly contaminated by pre-training overlap and the abstract's headline numbers contradict the paper's own tables. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism that carries the argument is the embedding-as-router design. The large language model, initialized from LLaVA and fine-tuned with Low-Rank Adaptation, expands its vocabulary with a special <Router> token; when a user supplies a fundus image and a text query, the final-layer embedding of that token is projected through a multilayer perceptron to choose one of eight vision foundation models. Each vision foundation model begins from RetFound pretrained weights and is further pretrained as a masked autoencoder with a 75% mask ratio, then fine-tuned on the SEED cohort. The routing loss and the text-generation loss are trained jointly, which is what lets one system answer questions about disease, severity, and signs from the same image.
What would settle it
Re-run the claimed external experiments after deleting every image from ODIR-5K and SP2, and any other overlapping dataset, from pretraining; if the reported external AUCs drop materially, the generalization claim is refuted. A simpler check is to inspect Table S1 and Table S3 side by side, as both list ODIR-5K and SP2.
Extended reading notes
Core claim
Meta-EyeFM's central claim is that a conversational eye-care assistant can be built by connecting a large language model to eight specialized vision foundation models through a routing layer. On the internal SEED test set, the router sent every fundus image to the correct expert, and the experts reached AUCs of 97.4% for referable diabetic retinopathy, 91.2% for AMD, 98.8% for referable myopic macular degeneration, 94.2% for glaucoma, and 93.9% for visually significant cataract, with disease-detection accuracy at or above 82.2%, severity differentiation at or above 89%, and sign identification at or above 76%. On external datasets the model kept AUCs of at least 81% for retinal disease detection and outperformed Gemini-1.5-flash and ChatGPT-4o by 11 to 43 percent depending on disease. In a 60-image benchmark against clinicians, Meta-EyeFM's F1 score (0.853) was closest to the best ophthalmologist (0.857) and above the other graders. The authors position the system as decision support for primary care and online fundus evaluation, while noting that further diagnostic improvement is needed before screening use.
Load-bearing premise
The external test results measure generalization only if the external datasets were not part of the self-supervised pretraining corpus; ODIR-5K and SP2 appear in both tables, so that independence is not established.
Editorial extensions
If this is right
- A single system can handle disease detection, severity grading, and sign identification from one fundus photo through natural-language questions, removing the need for separate apps per task.
- The routing design lets a general-purpose LLM handle conversation while specialized vision models do the image diagnosis, sidestepping the known weakness of LLMs on visual medical tasks.
- Fine-tuning with only 10% of the training data still kept internal AUCs within a few points of full-data training, supporting data-efficient expansion to rarer diseases and underrepresented groups.
- Sign identification (drusen, exudates, microaneurysms) gives clinicians a visible basis for the diagnosis, which the authors argue supports documentation and helps right-site referrals.
- Systemic disease prediction from fundus photos opens a non-invasive screening channel for diabetes, hypertension, and chronic kidney disease, although the authors state that performance must improve before real use.
Reading between the lines
- Because ODIR-5K and SP2 appear in both the pretraining list (Table S1) and the external test list (Table S3), the external accuracy figures should be treated as likely optimistic for truly unseen data until the model is retested on datasets excluded from pretraining.
- A natural next test is to run the same routing mechanism on OCT or anterior-segment photos, since the design is not eye-specific; if it transfers, the same architecture could become a general medical-image triage shell.
- An extension that follows directly from the design is to measure whether the conversational interface changes real referral decisions, not just label accuracy, in a prospective primary-care workflow.
- The routing accuracy was measured only on three query types; testing with free-form, out-of-distribution patient language would show whether the 100% routing figure survives real-world phrasing.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents Meta-EyeFM, a system that combines a LoRA-fine-tuned LLaVA-based multimodal LLM with eight task-specific vision foundation models (RetFound-initialized MAEs) for fundus image analysis. The LLM is trained to route user queries and images to the appropriate VFM, which performs detection of ocular and systemic diseases, severity differentiation, and sign identification. The authors report 100% routing accuracy on an internal SEED test set, internal AUCs between 79.8% and 98.8%, external AUCs above 59%, performance 11-43% better than Gemini-1.5-flash and ChatGPT-4o on several disease detection benchmarks, and F1-based parity with an ophthalmologist on a 60-image benchmark. The central claims are that the system is an accurate conversational diagnostic and triaging tool for primary eye care and that its external performance demonstrates generalizability.
Significance. If the empirical claims held, the paper would be a valuable contribution to conversational AI in ophthalmology: the router design is sensible, the comparison with general-purpose LMMs addresses a practical deployment question, and the few-shot analysis provides useful evidence on data efficiency. The use of a large multi-ethnic SEED cohort with adjudicated labels is a genuine strength, and the explicit definition of referable disease classes aids clinical interpretation. However, the external generalizability claim is compromised by the use of pre-training datasets in external testing, and several abstract/discussion-level performance numbers do not match the supplementary tables. As a result, the paper's headline claims are not currently supported by the evidence as presented.
major comments (3)
- [Methods, Table S1 vs Table S3] The self-supervised pre-training corpus in Table S1 includes ODIR-5K (6,377 images) and SP2 (9,139 images), and the same two datasets appear in Table S3 as external test sets. Because MAE pre-training reconstructs images directly from these datasets, the images are not unseen, so the external AUCs and accuracies for AMD, glaucoma, cataract, diabetes, and hypertension in Table S6 that draw on ODIR-5K and SP2 cannot support the Discussion's claim of 'robust external performance' or generalizability to unseen acquisition protocols. The authors should exclude all pre-trained datasets from external validation and re-run the analysis, or provide evidence such as near-duplicate detection that no image-level overlap exists.
- [Abstract and Discussion vs Tables S4 and S7] The Discussion states that Meta-EyeFM detected major ocular diseases with AUC ≥95% and severity accuracy ≥89%, and the abstract repeats the ≥89% severity claim. However, Table S4 reports internal AUCs of 91.2% for AMD, 93.9% for cataract, and 94.2% for glaucoma, and only referable DR and referable MMD reach 95% or higher. Table S7 reports severity accuracy of 82.5% for early AMD and 84.0% for mild NPDR. These internal inconsistencies must be corrected in the text and abstract, or the tables must be revised to support the stated minima.
- [Clinical benchmarking, Table S10] The claim that Meta-EyeFM is 'comparable to an ophthalmologist' is based on a single benchmark of 60 images (68 labels) with no confidence intervals, paired hypothesis tests, or measures of inter-grader variability. The glaucoma row alone shows F1-scores ranging from 0 to 0.706 across graders, so a single F1 value for Meta-EyeFM (0.696) does not establish statistical equivalence. The authors should report a paired analysis with uncertainty or substantially soften the claim.
minor comments (4)
- [Table S9 caption] The caption describes Gemini-1.5 and GPT-4o as 'open-source LLMs', but both are proprietary API-based models; this should be corrected to 'general-purpose LMMs'.
- [Figure 5 footnote and Table S9 footnote] The quoted prompt (3) is missing a closing quotation mark and contains a grammatical error ('presents in this image'); please fix the quotation and wording.
- [Results, Ocular disease detection, first paragraph] The phrase 'F1-scores ≤0.814' is uninformative because 0.814 is the largest F1 value in Table S4; the authors should report the range (e.g., 0.236-0.814) instead.
- [Results, External testing, first paragraph] The text states 'AUCs ≥81%' for referable DR and AMD, but Table S6 reports 80.7% for MESSIDOR2 (DR) and 80.5% for PIONEER (AMD); either the text or the table must be corrected.
Circularity Check
External test sets overlap pre-training (ODIR-5K, SP2); otherwise no constructional circularity in the derivation.
-
other
[Methods — Data used in algorithm development; Table S1 and Table S3]
"For model pre-training, we utilized 249,925 fundus photographs (41.3% Caucasians; 58.7% Asians) from 22 population-based studies across 4 continents (Table S1). ... We further included 65,664 fundus photographs from 10 private and open-source datasets for external testing (Table S3). Table S1 includes 'ODIR-5K Open source China 6377' and 'SP2 Private Singapore 9139'; Table S3 uses 'ODIR-5K' for AMD/glaucoma/cataract/diabetes/hypertension and 'SP2' for diabetes/hypertension."
External validation requires that the test images were not used in model development. Table S1 places ODIR-5K and SP2 in the 249,925-image self-supervised pre-training corpus, and Table S3 subsequently reports these same datasets as 'external' test sets. The pre-training autoencoder directly models those fundus images, so the reported external accuracies/AUCs for cataract (ODIR-5K), AMD/glaucoma (ODIR-5K), and diabetes/hypertension (ODIR-5K and SP2) are not held-out measurements. The claim of 'robust external performance' in the Discussion is therefore a re-evaluation of data already consumed by the model, i.e., a validation circularity, rather than an independent prediction on unseen inputs.
full rationale
The paper's derivation chain is empirical rather than analytic: the VFMs are pre-trained and then fine-tuned on SEED, the LLM router is LoRA-fine-tuned on SEED query/image pairs, and the detection/severity/sign/systemic metrics are measured on held-out internal splits or external sets. I found no equation-level step in which an output quantity is identical to an input quantity by definition, no fitted parameter renamed as a prediction, and no load-bearing self-citation chain; RetFound is used as initialization and LISA is cited for the embedding-as-router idea, but these are external precedents, not uniqueness arguments. The one concrete validation flaw is that Table S1 lists ODIR-5K and SP2 in the pre-training corpus while Table S3 uses the same datasets as 'external' test sets for AMD/glaucoma/cataract/diabetes/hypertension; self-supervised pre-training consumes those images, so those external numbers are not measurements on unseen data. That is a generalizability/data-leakage problem and a form of validation circularity, but it does not make the model's outputs equivalent to its inputs by construction. In addition, the abstract's '≥89% severity accuracy' and the Discussion's 'AUC ≥95% for major diseases' are contradicted by Table S7 (early AMD 82.5%; mild NPDR 84.0%) and Table S4 (AMD 91.2%; cataract 93.9%; glaucoma 94.2%); these are numerical reporting inconsistencies rather than circular reasoning.
Assumptions & free parameters
free parameters (3)
- Classification thresholds =
Per-disease Youden's J thresholds
- Loss weights lambda_T and lambda_R =
Not specified
- LoRA hyperparameters (rank, alpha, learning rate) =
Not specified
assumptions (5)
- domain assumption SEED manual gradings are reliable ground truth for ocular disease labels.
- domain assumption RetFound-pretrained weights provide a suitable initialization for downstream fundus tasks.
- domain assumption Fundus photographs contain sufficient signal to predict diabetes, hypertension, and chronic kidney disease.
- domain assumption The LLM router trained on SEED-style queries will route real-world user queries correctly.
- domain assumption MAE pre-training on a large fundus corpus improves downstream classification.
Cite this review
Pith. "Pith review of An integrated language-vision foundation model for conversational diagnostics and triaging in primary eye care." pith.science (2026). https://pith.science/paper/HA2IZ5V7
@misc{pith2026250508414,
author = {Pith},
title = {Pith review of: An integrated language-vision foundation model for conversational diagnostics and triaging in primary eye care},
year = {2026},
howpublished = {\url{https://pith.science/paper/HA2IZ5V7}},
note = {Machine review of arXiv:2505.08414}
}
abstract
Current deep learning models are mostly task specific and lack a user-friendly interface to operate. We present Meta-EyeFM, a multi-function foundation model that integrates a large language model (LLM) with vision foundation models (VFMs) for ocular disease assessment. Meta-EyeFM leverages a routing mechanism to enable accurate task-specific analysis based on text queries. Using Low Rank Adaptation, we fine-tuned our VFMs to detect ocular and systemic diseases, differentiate ocular disease severity, and identify common ocular signs. The model achieved 100% accuracy in routing fundus images to appropriate VFMs, which achieved $\ge$ 82.2% accuracy in disease detection, $\ge$ 89% in severity differentiation, $\ge$ 76% in sign identification. Meta-EyeFM was 11% to 43% more accurate than Gemini-1.5-flash and ChatGPT-4o LMMs in detecting various eye diseases and comparable to an ophthalmologist. This system offers enhanced usability and diagnostic performance, making it a valuable decision support tool for primary eye care or an online LLM for fundus evaluation.
Reference graph
Works this paper leans on
-
[1]
Singapore Eye Research Institute, Singapore National Eye Centre, Singapore Address: 20 College Road, Singapore 169856
-
[2]
Institute of High Performance Computing, Agency of Science, Technology and Research, Singapore Address: 1 Fusionopolis Way, Singapore 138632
-
[3]
Department of Radiology, University of Pennsylvania, Philadephia, USA Address: Philadephia, PA 19104, USA
-
[4]
Ophthalmology & Visual Sciences Academic Clinical Program, Duke-NUS Medical School, Singapore Address: 8 College Road, Singapore 169857
-
[5]
Department of Ophthalmology, Columbia University, New York, USA Address: 622 W 168th Street, 3rd Floor, New York, NY10032, United States
-
[6]
Centre for Innovation and Precision Eye Health, National University of Singapore, Singapore Address: 1E Kent Ridge Road, NUHS Tower Block, Level 7, Singapore 119228
-
[7]
123, Dapi Rd., Niaosong District, Kaohsiung City, 83301, Taiwan
Department of Ophthalmology, Chang Gung Memorial Hospital, Linkou Medical Centre, Taoyuan, Taiwan Address: No. 123, Dapi Rd., Niaosong District, Kaohsiung City, 83301, Taiwan
-
[8]
College of Medicine, Chang Gung University, Taoyuan, Taiwan Address: N0 259, Wenhua 1st road, Guishan District, Taoyuan, 333, Taiwan
Show all 49 references
-
[9]
1440, Hongqiao Road, Changning District, Shanghai, China
Shanghai Eye Disease Prevention & Treatment Center, Shanghai Eye Hospital, China Address: No. 1440, Hongqiao Road, Changning District, Shanghai, China. 2
-
[10]
Address: 8 College Road, Singapore 169857
Duke-NUS Medical School, National University of Singapore, Singapore. Address: 8 College Road, Singapore 169857
-
[11]
Rothschild Foundation Hospital, Institut Français de Myopie, Paris, France Address: 25-29 Rue Manin, 75019 Paris, France
-
[12]
Panda-Jonas Address: Adenauerplatz 2, 69115, Heidelberg, Germany
Privatpraxis Prof Jonas and Dr. Panda-Jonas Address: Adenauerplatz 2, 69115, Heidelberg, Germany
-
[13]
Beijing Visual Science and Translational Eye Research Institute, Beijing Tsinghua Changgung Hospital, Tsinghua Medicine, Tsinghua University, Beijing, China Address: 30 Shuangqing Road, Haidian District, Beijing, 100190, China
-
[14]
Levy PI, New York, NY 10029, United States
New York eye and Ear Infirmary of Mount Sinai, Icahn School of Medicine at Mount Sinai, New York, United States of America Address: 1 Gustave L. Levy PI, New York, NY 10029, United States
-
[15]
559, New Colony, Nagpur, Maharashtra 440001, India
Suraj Eye Institute, Nagpur, India Address: Plot No. 559, New Colony, Nagpur, Maharashtra 440001, India
-
[16]
Department of Ophthalmology, Medical Faculty Heidelberg, Heidelberg University, Germany Address: Grabengasse 1, 69117 Heidelberg, Germany
-
[17]
Beijing Institute of Ophthalmology, Beijing Tongren Hospital, Capital Meical University, Beijing, China Address: Fengtai District, 100054, China
-
[18]
Could you identify the specific disease present in this fundus image?
Department of Ophthalmology, National University of Singapore, Singapore Address: 1E Kent Ridge Road, NUHS Tower Block, Level 7, Singapore 119228. Financial support: This study was funded by the National Medical Research Council, Singapore (NMRC/CIRG/1488/2018, NMRC/CIRG33jul-...
2018
-
[19]
Collaborators G. Global, regional, and national incidence, prevalence, and years lived with disability for 354 diseases and injuries for 195 countries and territories, 1990-2017: a systematic analysis for the Global Burden of Disease Study 2017. 2018
1990
-
[20]
The Lancet global health Commission on global eye health: vision beyond 2020
Burton MJ, Ramke J, Marques AP , et al. The Lancet global health Commission on global eye health: vision beyond 2020. The Lancet Global Health. 2021;9(4):e489-e551
2020
-
[21]
Prevalence of undiagnosed age-related macular degeneration in primary eye care
Neely DC, Bray KJ, Huisingh CE, Clark ME, McGwin G, Owsley C. Prevalence of undiagnosed age-related macular degeneration in primary eye care. JAMA ophthalmology. 2017;135(6):570-575
2017
-
[22]
The global extent of undetected glaucoma in adults: a systematic review and meta-analysis
Soh ZD, Yu M, Betzler BK, et al. The global extent of undetected glaucoma in adults: a systematic review and meta-analysis. Ophthalmology. 2021;128(10):1393-1404
2021
-
[23]
Prevalence, Risk Factors, and Impact of Undiagnosed Visually Significant Cataract: The Singapore Epidemiology of Eye Diseases Study
Chua J, Lim B, Fenwick EK, et al. Prevalence, Risk Factors, and Impact of Undiagnosed Visually Significant Cataract: The Singapore Epidemiology of Eye Diseases Study. PLOS ONE. 2017;12(1):e0170804
2017
-
[24]
World report on vision
Organization WH. World report on vision. 2019
2019
-
[25]
Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs
Gulshan V, Peng L, Coram M, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. jama. 2016;316(22):2402-2410
2016
-
[26]
Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes
Ting DSW, Cheung CY-L, Lim G, et al. Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. Jama. 2017;318(22):2211-2223
2017
-
[27]
A deep learning algorithm for prediction of age- related eye disease study severity scale for age-related macular degeneration from color fundus photography
Grassmann F, Mengelkamp J, Brandl C, et al. A deep learning algorithm for prediction of age- related eye disease study severity scale for age-related macular degeneration from color fundus photography. Ophthalmology. 2018;125(9):1410-1420
2018
-
[28]
Deep learning is effective for classifying normal versus age- related macular degeneration OCT images
Lee CS, Baughman DM, Lee AY . Deep learning is effective for classifying normal versus age- related macular degeneration OCT images. Ophthalmology Retina. 2017;1(4):322-327
2017
-
[29]
Self-supervised learning in medicine and healthcare
Krishnan R, Rajpurkar P , Topol EJ. Self-supervised learning in medicine and healthcare. Nature Biomedical Engineering. 2022;6(12):1346-1352
2022
-
[30]
A simple framework for contrastive learning of visual representations
Chen T, Kornblith S, Norouzi M, Hinton G. A simple framework for contrastive learning of visual representations. Paper presented at: International conference on machine learning2020
-
[31]
Big self-supervised models are strong semi-supervised learners
Chen T, Kornblith S, Swersky K, Norouzi M, Hinton GE. Big self-supervised models are strong semi-supervised learners. Advances in neural information processing systems. 2020;33:22243-22255
2020
-
[32]
A foundation model for generalizable disease detection from retinal images
Zhou Y , Chia MA, Wagner SK, et al. A foundation model for generalizable disease detection from retinal images. Nature. 2023;622(7981):156-163
2023
-
[33]
Capabilities of gpt-4 on medical challenge problems
Nori H, King N, McKinney SM, Carignan D, Horvitz E. Capabilities of gpt-4 on medical challenge problems. arXiv preprint arXiv:230313375. 2023
2023
-
[34]
Vision–language foundation model for echocardiogram interpretation
Christensen M, Vukadinovic M, Yuan N, Ouyang D. Vision–language foundation model for echocardiogram interpretation. Nature Medicine. 2024:1-8
2024
-
[35]
Can gpt-4v (ision) serve medical applications? case studies on gpt- 4v for multimodal medical diagnosis
Wu C, Lei J, Zheng Q, et al. Can gpt-4v (ision) serve medical applications? case studies on gpt- 4v for multimodal medical diagnosis. arXiv preprint arXiv:231009909. 2023
2023
-
[36]
Cohort profile: the Singapore epidemiology of eye diseases study (seed)
Majithia S, Tham Y-C, Chee M-L, et al. Cohort profile: the Singapore epidemiology of eye diseases study (seed). International journal of epidemiology. 2021;50(1):41-52
2021
-
[37]
Lora: Low-rank adaptation of large language models
Hu EJ, Shen Y , Wallis P , et al. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:210609685. 2021
2021
-
[38]
Visual instruction tuning
Liu H, Li C, Wu Q, Lee YJ. Visual instruction tuning. Advances in neural information processing systems. 2024;36. 21
2024
-
[39]
Lisa: Reasoning segmentation via large language model
Lai X, Tian Z, Chen Y , et al. Lisa: Reasoning segmentation via large language model. Paper presented at: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition2024
-
[40]
Masked autoencoders are scalable vision learners
He K, Chen X, Xie S, Li Y , Dollár P , Girshick R. Masked autoencoders are scalable vision learners. Paper presented at: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition2022
-
[41]
Index for rating diagnostic tests
WJ Y . Index for rating diagnostic tests. Cancer. 1950;3:32-35
1950
-
[42]
On OCT image classification via deep learning
Wang D, Wang L. On OCT image classification via deep learning. IEEE Photonics Journal. 2019;11(5):1-14
2019
-
[43]
Assessment of optometrists' referral accuracy and contributing factors: A review
Carmichael J, Abdi S, Balaskas K, Costanza E, Blandford A. Assessment of optometrists' referral accuracy and contributing factors: A review. Ophthalmic and Physiological Optics. 2023;43(5):1255-1277
2023
-
[44]
More than one-third of US individuals use the Internet to self-diagnose
Kuehn BM. More than one-third of US individuals use the Internet to self-diagnose. Jama. 2013;309(8):756-757
2013
-
[45]
Optometrists. Co. College of Optometrists guideline C143: Communication, Partnership and Teamwork. https://www.college-optometrists.org/clinical- guidance/guidance/communication,-partnership-and-teamwork/working-with-colleagues. Accessed 29 May, 2024
2024
-
[46]
The false hope of current approaches to explainable artificial intelligence in health care
Ghassemi M, Oakden-Rayner L, Beam AL. The false hope of current approaches to explainable artificial intelligence in health care. The Lancet Digital Health. 2021;3(11):e745- e750
2021
-
[47]
Understanding the impact of explanations on advice-taking: a user study for AI-based clinical Decision Support Systems
Panigutti C, Beretta A, Giannotti F, Pedreschi D. Understanding the impact of explanations on advice-taking: a user study for AI-based clinical Decision Support Systems. Paper presented at: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems2022
2022
-
[48]
Global prevalence and major risk factors of diabetic retinopathy
Yau JW, Rogers SL, Kawasaki R, et al. Global prevalence and major risk factors of diabetic retinopathy. Diabetes care. 2012;35(3):556-564
2012
-
[49]
what is the disease in this image
Bourne RR, Flaxman SR, Braithwaite T, et al. Magnitude, temporal trends, and projections of the global prevalence of blindness and distance and near vision impairment: a systematic review and meta-analysis. The Lancet Global Health. 2017;5(9):e888-e897. 22 Supplemental informa...
2017
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.