REVIEW 3 major objections 5 minor 1 cited by
Evaluating Large Language Models for Multimodal Simulated Ophthalmic Decision-Making in Diabetic Retinopathy and Glaucoma Screening
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper reports that GPT-4, reading structured text descriptions of retinal fundus photographs, reaches only moderate agreement on diabetic retinopathy severity and essentially no skill at glaucoma referral, and that adding patient…
desk verdict Honest negative result, but the paper must disclose how the textual descriptions were made before its headline numbers mean anything. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is a structured prompt that converts each fundus photograph into a text-only clinical vignette and asks the model to perform four linked tasks: list diabetic retinopathy signs, assign an ICDR score from 0 to 4 with panretinal photocoagulation scars forcing level 4, recommend DR referral when the score is at least 2 or macular edema is present, and estimate the cup-to-disc ratio, referring for glaucoma when the ratio exceeds 0.6. The same prompt is run under three conditions, image description alone, with real patient metadata, and with synthetic metadata, and the three outputs are compared through accuracy, $F_1$, Cohen's $\kappa$, McNemar's test, and pairwise change rates. The design isolates the contribution of metadata by keeping the description fixed and varying only the demographic and clinical context.
What would settle it
Inspect the description-generation step: if the structured text fed to GPT-4 was authored from the reference labels or by unblinded raters, the central claim collapses. A clean test is to have blinded graders or an automated captioning system write fresh descriptions for the same 300 images and rerun the identical prompts; if ICDR $\kappa$ or glaucoma $F_1$ rises sharply, the original numbers were an artifact of the descriptions, and if they stay near 0.25 and 0.03, the paper's conclusion holds.
Extended reading notes
Core claim
The central claim is that GPT-4 can simulate only the coarse parts of ophthalmic screening from text, and that its predictions are insensitive to clinical metadata. In 300 annotated fundus images from mBRSET, the model assigned ICDR severity levels with limited agreement with the reference standard ($\kappa=0.250$, accuracy 67.5%, macro $F_1=0.330$, weighted $F_1=0.672$), and its accuracy was concentrated in the normal class (class 0 $F_1=0.82$), with $F_1=0.00$ for mild and severe non-proliferative DR. The binary DR referral task fared better (accuracy 82.3%, $F_1=0.54$, $\kappa=0.436$), while glaucoma referral from estimated cup-to-disc ratio was essentially absent ($F_1<0.04$, $\kappa<0.03$). Neither real nor synthetic metadata produced a statistically significant shift in any task ($p>0.05$ by McNemar's test), and the authors read this stability as evidence that the model is not integrating patient-specific context into its decisions.
Load-bearing premise
The evaluation stands or falls on the structured textual descriptions of the 300 images being accurate, complete, and independent of the reference labels — if those descriptions came from the ground-truth diagnoses or from raters who saw them, the reported scores would reflect label leakage rather than GPT-4's decision-making.
Editorial extensions
If this is right
- A text-only GPT-4 pipeline should not be used for standalone screening or triage in diabetic retinopathy or glaucoma, because its near-zero performance on referable DR classes and on glaucoma referral would miss exactly the patients who need follow-up.
- Adding demographic or clinical metadata to prompts is not a shortcut: because predictions move in under 7% of cases, efforts to improve LLM screening should focus on how the image content is represented rather than on enriching the patient context.
- Binary referral decisions are a more realistic target for text-only LLMs than fine-grained severity grading, since the model's DR referral performance was substantially stronger than its ICDR multiclass performance.
- The model's near-total blindness to metadata has a fairness implication: its decisions do not adjust for age, sex, or comorbidities, so any use in annotation or documentation would inherit this context-free behavior.
Reading between the lines
- Editorial extension: if the structured image descriptions were written from the ground-truth labels or by raters who had seen them, the reported $\kappa$ and $F_1$ values would partly measure label leakage through the descriptions rather than GPT-4's clinical reasoning; the paper does not disclose who authored the descriptions or whether they were blinded.
- Editorial extension: a direct next experiment is to feed the same prompt template to a genuinely multimodal model that reads the pixels, which would separate the loss caused by the text bottleneck from the intrinsic difficulty of the grading tasks.
- Editorial extension: because the model's decisions are nearly deterministic across prompt conditions, it could serve as a cheap first-pass annotator for easy normal cases, with human review concentrated on the small fraction of cases where the predicted label shifts.
- Editorial extension: the glaucoma failure may be a task-format artifact as much as a model limitation; prompting for quantified optic-disc features such as neuroretinal rim width instead of a single cup-to-disc ratio could give a language model a fairer chance.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript reports a retrospective evaluation of GPT-4 as a text-only simulator of ophthalmic decision-making. The authors selected 300 fundus photographs from mBRSET, converted each image into a structured textual description, and prompted GPT-4 under three conditions: image description alone, image description plus real patient metadata, and image description plus synthetic metadata. The model was asked to output an ICDR severity score, a diabetic retinopathy referral decision, and a cup-to-disc-ratio based glaucoma referral decision. The main findings are moderate agreement for ICDR grading (kappa 0.25, macro F1 0.33), improved but still limited performance for binary DR referral (F1 0.54, kappa 0.44), essentially no skill for glaucoma referral (F1 below 0.04), and no statistically significant or practically large effect of adding metadata. The authors conclude that GPT-4 is not clinically reliable for these tasks but may be useful for educational, documentation, or annotation workflows.
Significance. If the central empirical claims hold, the paper provides a useful early benchmark: it shows that a state-of-the-art LLM, fed only with textual descriptions of retinal images, achieves only moderate performance on simple DR referral and essentially no skill on glaucoma referral, and that adding patient metadata does not change predictions. The study design has genuine strengths: the three-condition comparison is clear, the use of paired McNemar tests and change-rate analysis is appropriate for the research question, and the Discussion is appropriately cautious about clinical applicability. The synthetic metadata condition is a sensible perturbation test. However, the value of the results depends entirely on two unverified preconditions: that the textual image descriptions are a faithful and label-independent representation of the photographs, and that the analyzed 265-image subset is representative of the stated 300-image sample.
major comments (3)
- [§2.4] The central precondition of the study is that the structured textual descriptions accurately and completely convey the fundus image content without incorporating the ground-truth labels. Section 2.4 states only that 'all image content was conveyed through structured textual descriptions' and does not identify who authored the descriptions, whether they were derived from clinical reports, automated captioning, or the labels themselves, or whether the authors were blinded to the outcome. This matters because the Appendix prompt instructs the model to treat panretinal laser scars as ICDR level 4; if the descriptions were written with knowledge of the labels, then a correct ICDR 4 classification would simply recover a label-derived phrase rather than demonstrate reasoning. The same issue affects the glaucoma result: if the descriptions omit or include optic-disc detail in a label-dependent way, the poor F1 could be an artifact of the description content. The authors must provide the full description-generation protocol and, ideally, sample descriptions with the corresponding images and labels, or at least a clear statement that the descriptions were produced by graders blinded to the reference standard and validated for accuracy and completeness.
- [§3] The manuscript states that 300 images were selected, but all reported class counts in Section 3 sum to 265, not 300: ICDR counts 198+27+22+15+3 = 265, DR referral counts 220+45 = 265, and glaucoma referral counts 206+59 = 265. The 35 excluded images are never mentioned or explained. If the exclusions are related to image quality, missing metadata, or description-generation failures, the reported accuracy, kappa, and McNemar results may be biased. The authors must report the number of excluded images, the reasons for exclusion, and ideally a comparison of included versus excluded cases on available variables. Without this, all denominators and agreement estimates rest on an unexplained subset, and the 67.5% ICDR accuracy and other headline figures cannot be taken at face value.
- [§2.4/§2.6] The evaluation appears to use a single GPT-4 response per prompt, accessed through the ChatGPT platform with 'temperature and sampling parameters left unchanged.' This gives no information about run-to-run variability, and the paper reports no confidence intervals around the point estimates. The central null finding that metadata does not change predictions (McNemar p > 0.05, change rates under 7%) is only meaningful if the underlying predictions are stable. With a stochastic model, a single run cannot distinguish a true null effect from sampling noise, especially for the small change rates reported. The authors should either run each prompt multiple times and report the distribution of metrics and prediction-change rates, or justify why single-run outputs are sufficient for the claims made.
minor comments (5)
- [§2.7] There is a typographical error in 'thestatsmodels library'; it should read 'the statsmodels library.'
- [Appendix] The output format templates contain mismatched braces and parentheses, e.g., 'Yes/No { with a brief explanation' appears twice; these should be corrected to avoid ambiguity in the prompt specification.
- [§2.5] The synthetic metadata generation rule assigns 20% of ages as '≥90' and the rest as random integers between 20 and 89; the paper should state whether this distribution was chosen to match mBRSET or for another reason, and how the remaining variables were sampled independently.
- [§3] For the McNemar tests that yield p-values of 1.00, the paper should report the number of discordant pairs; a p-value of 1.00 can arise from very few changes, and without the discordant cell counts the reader cannot assess the power of the comparison.
- [References] Reference [15] is titled 'Evaluating Large Language Models for Simulated Ophthalmic Decision-Making,' which closely resembles the current manuscript's title; please confirm this is a distinct prior publication and clarify the relationship.
Circularity Check
No circularity identified; the evaluation is self-contained against external labels and involves no fitted parameters or self-citation-dependent claims.
full rationale
The study is an empirical benchmark, not a derivation. GPT-4's outputs are prompted with hand-specified rules (ICDR scoring, referral thresholds, CDR cutoff) and compared with fixed dataset labels from mBRSET. No parameter is fitted to the test labels, and no prediction is statistically forced by an earlier fit, so the fitted-input and self-definitional patterns do not apply. The cited prior work, including mBRSET and earlier LLM ophthalmology evaluations, supplies dataset provenance and context rather than the load-bearing justification for the headline numbers; the central empirical results come from the current experiment's prompts and external labels. The DR-referral and glaucoma-referral tasks are thresholded versions of the model's own ICDR and CDR outputs, so their performance is a deterministic function of those outputs, but this is an evaluation-design property, not a circular derivation: the ground-truth referral labels are external, and the paper does not claim to predict ICDR from the referral label. One substantive methodological gap is that the manuscript never discloses who authored the structured textual image descriptions or whether they were blinded to the reference labels (Section 2.4). If those descriptions were generated from the labels, the kappa/F1 values would be partly circular; however, the text does not exhibit that reduction, and the limitations section acknowledges only reduced image granularity, not label leakage. A separate numerical inconsistency (Section 3 counts sum to 265 rather than the stated 300 images) undermines reproducibility but is not a circularity. These are correctness and reporting risks, not demonstrated circular steps, so the score is 0.
Assumptions & free parameters
free parameters (1)
- Synthetic metadata generation rule =
Age uniform 20-89 plus 20% '>=90'; binary variables random
assumptions (4)
- domain assumption mBRSET reference labels are correct for all 265 analyzed images.
- ad hoc to paper Textual image descriptions faithfully represent the fundus images without loss or label leakage.
- ad hoc to paper Single-run GPT-4 outputs are stable enough for the reported statistics.
- ad hoc to paper The 35 images lost between the stated 300 and the analyzed 265 are missing at random.
Cite this review
Pith. "Pith review of Evaluating Large Language Models for Multimodal Simulated Ophthalmic Decision-Making in Diabetic Retinopathy and Glaucoma Screening." pith.science (2026). https://pith.science/paper/YYPVRWW6
@misc{pith2026250701278,
author = {Pith},
title = {Pith review of: Evaluating Large Language Models for Multimodal Simulated Ophthalmic Decision-Making in Diabetic Retinopathy and Glaucoma Screening},
year = {2026},
howpublished = {\url{https://pith.science/paper/YYPVRWW6}},
note = {Machine review of arXiv:2507.01278}
}
read the original abstract
Large language models (LLMs) can simulate clinical reasoning based on natural language prompts, but their utility in ophthalmology is largely unexplored. This study evaluated GPT-4's ability to interpret structured textual descriptions of retinal fundus photographs and simulate clinical decisions for diabetic retinopathy (DR) and glaucoma screening, including the impact of adding real or synthetic clinical metadata. We conducted a retrospective diagnostic validation study using 300 annotated fundus images. GPT-4 received structured prompts describing each image, with or without patient metadata. The model was tasked with assigning an ICDR severity score, recommending DR referral, and estimating the cup-to-disc ratio for glaucoma referral. Performance was evaluated using accuracy, macro and weighted F1 scores, and Cohen's kappa. McNemar's test and change rate analysis were used to assess the influence of metadata. GPT-4 showed moderate performance for ICDR classification (accuracy 67.5%, macro F1 0.33, weighted F1 0.67, kappa 0.25), driven mainly by correct identification of normal cases. Performance improved in the binary DR referral task (accuracy 82.3%, F1 0.54, kappa 0.44). For glaucoma referral, performance was poor across all settings (accuracy ~78%, F1 <0.04, kappa <0.03). Metadata inclusion did not significantly affect outcomes (McNemar p > 0.05), and predictions remained consistent across conditions. GPT-4 can simulate basic ophthalmic decision-making from structured prompts but lacks precision for complex tasks. While not suitable for clinical use, LLMs may assist in education, documentation, or image annotation workflows in ophthalmology.
Figures
Forward citations
Cited by 1 Pith paper
-
An Agentic AI Framework Overcomes Fundamental Limitations of Large Language Models for Glaucoma Detection from Fundus Photography
A three-step agentic workflow with LLM function calling and reflection improved glaucoma classification, CDR estimation, and repeatability over LLM-alone baselines, approaching specialist-level accuracy.
Reference graph
Works this paper leans on
-
[15]
Evaluating Large Language Models for Simulated Ophthalmic Decision-Making
Liu X, Zhang T, Chen M, et al. Evaluating Large Language Models for Simulated Ophthalmic Decision-Making. Ophthalmol Sci. 2023
work page 2023
-
[1]
Sun H, Saeedi P, Karuranga S, et al. IDF Diabetes Atlas: Global, regional and country-level diabetes prevalence estimates for 2021 and projections for 2045. Dia- betes Res Clin Pract.2022;183:109119
work page 2021
-
[2]
Teo ZL, Tham YC, Yu M, et al. Global Prevalence of Diabetic Retinopathy and Projection of Burden through 2045: Systematic Review and Meta-analysis. Ophthal- mology. 2021;128(11):1580-1591
work page 2021
-
[3]
Artificial intelligence for telemedicine diabetic retinopathy screening: a review
Nakayama LF, Ribeiro LZ, Novaes F, et al. Artificial intelligence for telemedicine diabetic retinopathy screening: a review. Ann Med.2023;55(2):2258149
work page 2023
-
[4]
The pathophysiology and treatment of glau- coma: a review
Weinreb RN, Aung T, Medeiros F A. The pathophysiology and treatment of glau- coma: a review. JAMA. 2014;311(18):1901-1911
work page 2014
-
[5]
Tan RE, Teo KYC, Husain R, et al. Evaluating the outcome of screening for glau- coma using colour fundus photography-based referral criteria in a teleophthalmology screening programme for diabetic retinopathy. Br J Ophthalmol.2023
work page 2023
-
[6]
Crowston JG, Hopley CR, Healey PR, et al. The effect of optic disc diameter on ver- tical cup to disc ratio percentiles in a population based cohort: the Blue Mountains Eye Study. Br J Ophthalmol.2004;88(6):766-770
work page 2004
-
[7]
Language models are few-shot learners
Brown T, Mann B, Ryder N, et al. Language models are few-shot learners. Adv Neur Inf Proc Sys.2020;33:1877–1901
work page 2020
Show all 20 references
-
[8]
Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models
Kung TH, Cheatham M, Medenilla A, et al. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLoS Digit Health. 2023;2(2):e0000198
2023
-
[9]
Performance of ChatGPT-4 in answering questions from the Brazilian National Examination for Medical Degree Revalidation
Gobira M, Nakayama LF, Moreira R, et al. Performance of ChatGPT-4 in answering questions from the Brazilian National Examination for Medical Degree Revalidation. Rev Assoc Med Bras.2023;69(10)
2023
-
[10]
Evaluating the performance of ChatGPT in ophthalmology: An analysis of its successes and shortcomings
Antaki F, Touma S, Milad D, et al. Evaluating the performance of ChatGPT in ophthalmology: An analysis of its successes and shortcomings. Ophthalmol Sci. 2023;3(4):100324
2023
-
[11]
Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs.JAMA
Gulshan V, Peng L, Coram M, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs.JAMA. 2016;316(22):2402–2410
2016
-
[12]
Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes
Ting DSW, Cheung CY, Lim G, et al. Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. JAMA. 2017;318(22):2211–2223
2017
-
[13]
Interobserver agree- ment for the clinical assessment of optic discs
Garway-Heath DF, Ruben S, Viswanathan A, Hitchings RA. Interobserver agree- ment for the clinical assessment of optic discs. Br J Ophthalmol.1998;82(6):644–647
1998
-
[14]
Diagnostic accuracy of optic nerve head imaging for glaucoma: a systematic review and meta-analysis
Cheung CY, Rulli E, Schulze A, et al. Diagnostic accuracy of optic nerve head imaging for glaucoma: a systematic review and meta-analysis. Lancet Digit Health. 2018;1(4):e172–e182. 10
2018
-
[16]
GPT-4 Technical Report
OpenAI. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774.2023
2023 arXiv
-
[17]
mBRSET: The Brazilian Ophthal- mological Dataset for Diabetic Retinopathy and Multimodal Retinal AI
Nakayama LF, Malerbi FK, Regatieri CV, et al. mBRSET: The Brazilian Ophthal- mological Dataset for Diabetic Retinopathy and Multimodal Retinal AI. In prepa- ration. 2024
2024
-
[18]
Large Language Models in Medicine: Current Limitations and Future Potential
Stolz D, Keestra S, et al. Large Language Models in Medicine: Current Limitations and Future Potential. npj Digit Med.2023
2023
-
[19]
A portable retina fundus photos dataset for clinical, demographic, and diabetic retinopathy prediction
Wu C, Restrepo D, Nakayama LF, et al. A portable retina fundus photos dataset for clinical, demographic, and diabetic retinopathy prediction. Sci Data.2025;12(1):323
2025
-
[20]
Proposed international clinical diabetic retinopathy and diabetic macular edema disease severity scales
Wilkinson CP, Ferris FL, Klein RE, et al. Proposed international clinical diabetic retinopathy and diabetic macular edema disease severity scales. Ophthalmology. 2003;110(9):1677–1682. 11 Appendix: Example of Real and Synthetic Metadata Input Real Metadata Prompt Please review...
2003
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.