Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Evaluating Large Language Models for Multimodal Simulated Ophthalmic Decision-Making in Diabetic Retinopathy and Glaucoma Screening

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read This paper reports that GPT-4, reading structured text descriptions of retinal fundus photographs, reaches only moderate agreement on diabetic retinopathy severity and essentially no skill at glaucoma referral, and that adding patient…

desk verdict Honest negative result, but the paper must disclose how the textual descriptions were made before its headline numbers mean anything. read the letter →

arxiv 2507.01278 v1 pith:YYPVRWW6 submitted 2025-07-02 cs.CL

classification cs.CL
keywords GPT-4largelanguagemodelsdiabeticretinopathyscreeningglaucomaretinalfundusphotographsstructuredtextpromptsclinicalmetadataICDRseveritygrading
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The authors are trying to establish that a text-only large language model, GPT-4, cannot replace clinical screening for diabetic retinopathy or glaucoma when it is fed structured descriptions of retinal photographs rather than the images themselves. They report moderate performance only on the coarsest distinction: ICDR severity grading reached 67.5% accuracy but $\kappa=0.25$ and macro $F_1=0.33$, driven by correct detection of normal cases, while mild and severe DR classes scored $F_1=0.00$. Reframing the task as binary referral for diabetic retinopathy improved results (accuracy 82.3%, $F_1=0.54$, $\kappa=0.44$), but glaucoma referral based on cup-to-disc ratio remained near zero across every condition ($F_1<0.04$, $\kappa<0.03$). Adding real or synthetic patient metadata changed predictions in fewer than 7% of ICDR cases and under 3% of DR referral cases, with no significant McNemar differences, so the authors conclude the model leans on image descriptions and pretrained medical priors, not patient context. If the results hold, text-only LLMs are not clinically usable for these screening tasks, though they may still serve educational, documentation, or annotation workflows.

What carries the argument

The mechanism is a structured prompt that converts each fundus photograph into a text-only clinical vignette and asks the model to perform four linked tasks: list diabetic retinopathy signs, assign an ICDR score from 0 to 4 with panretinal photocoagulation scars forcing level 4, recommend DR referral when the score is at least 2 or macular edema is present, and estimate the cup-to-disc ratio, referring for glaucoma when the ratio exceeds 0.6. The same prompt is run under three conditions, image description alone, with real patient metadata, and with synthetic metadata, and the three outputs are compared through accuracy, $F_1$, Cohen's $\kappa$, McNemar's test, and pairwise change rates. The design isolates the contribution of metadata by keeping the description fixed and varying only the demographic and clinical context.

What would settle it

Inspect the description-generation step: if the structured text fed to GPT-4 was authored from the reference labels or by unblinded raters, the central claim collapses. A clean test is to have blinded graders or an automated captioning system write fresh descriptions for the same 300 images and rerun the identical prompts; if ICDR $\kappa$ or glaucoma $F_1$ rises sharply, the original numbers were an artifact of the descriptions, and if they stay near 0.25 and 0.03, the paper's conclusion holds.

Watch

Extended reading notes

Core claim

The central claim is that GPT-4 can simulate only the coarse parts of ophthalmic screening from text, and that its predictions are insensitive to clinical metadata. In 300 annotated fundus images from mBRSET, the model assigned ICDR severity levels with limited agreement with the reference standard ($\kappa=0.250$, accuracy 67.5%, macro $F_1=0.330$, weighted $F_1=0.672$), and its accuracy was concentrated in the normal class (class 0 $F_1=0.82$), with $F_1=0.00$ for mild and severe non-proliferative DR. The binary DR referral task fared better (accuracy 82.3%, $F_1=0.54$, $\kappa=0.436$), while glaucoma referral from estimated cup-to-disc ratio was essentially absent ($F_1<0.04$, $\kappa<0.03$). Neither real nor synthetic metadata produced a statistically significant shift in any task ($p>0.05$ by McNemar's test), and the authors read this stability as evidence that the model is not integrating patient-specific context into its decisions.

Load-bearing premise

The evaluation stands or falls on the structured textual descriptions of the 300 images being accurate, complete, and independent of the reference labels — if those descriptions came from the ground-truth diagnoses or from raters who saw them, the reported scores would reflect label leakage rather than GPT-4's decision-making.

Editorial extensions

If this is right

  • A text-only GPT-4 pipeline should not be used for standalone screening or triage in diabetic retinopathy or glaucoma, because its near-zero performance on referable DR classes and on glaucoma referral would miss exactly the patients who need follow-up.
  • Adding demographic or clinical metadata to prompts is not a shortcut: because predictions move in under 7% of cases, efforts to improve LLM screening should focus on how the image content is represented rather than on enriching the patient context.
  • Binary referral decisions are a more realistic target for text-only LLMs than fine-grained severity grading, since the model's DR referral performance was substantially stronger than its ICDR multiclass performance.
  • The model's near-total blindness to metadata has a fairness implication: its decisions do not adjust for age, sex, or comorbidities, so any use in annotation or documentation would inherit this context-free behavior.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: if the structured image descriptions were written from the ground-truth labels or by raters who had seen them, the reported $\kappa$ and $F_1$ values would partly measure label leakage through the descriptions rather than GPT-4's clinical reasoning; the paper does not disclose who authored the descriptions or whether they were blinded.
  • Editorial extension: a direct next experiment is to feed the same prompt template to a genuinely multimodal model that reads the pixels, which would separate the loss caused by the text bottleneck from the intrinsic difficulty of the grading tasks.
  • Editorial extension: because the model's decisions are nearly deterministic across prompt conditions, it could serve as a cheap first-pass annotator for easy normal cases, with human review concentrated on the small fraction of cases where the predicted label shifts.
  • Editorial extension: the glaucoma failure may be a task-format artifact as much as a model limitation; prompting for quantified optic-disc features such as neuroretinal rim width instead of a single cup-to-disc ratio could give a language model a fairer chance.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript reports a retrospective evaluation of GPT-4 as a text-only simulator of ophthalmic decision-making. The authors selected 300 fundus photographs from mBRSET, converted each image into a structured textual description, and prompted GPT-4 under three conditions: image description alone, image description plus real patient metadata, and image description plus synthetic metadata. The model was asked to output an ICDR severity score, a diabetic retinopathy referral decision, and a cup-to-disc-ratio based glaucoma referral decision. The main findings are moderate agreement for ICDR grading (kappa 0.25, macro F1 0.33), improved but still limited performance for binary DR referral (F1 0.54, kappa 0.44), essentially no skill for glaucoma referral (F1 below 0.04), and no statistically significant or practically large effect of adding metadata. The authors conclude that GPT-4 is not clinically reliable for these tasks but may be useful for educational, documentation, or annotation workflows.

Significance. If the central empirical claims hold, the paper provides a useful early benchmark: it shows that a state-of-the-art LLM, fed only with textual descriptions of retinal images, achieves only moderate performance on simple DR referral and essentially no skill on glaucoma referral, and that adding patient metadata does not change predictions. The study design has genuine strengths: the three-condition comparison is clear, the use of paired McNemar tests and change-rate analysis is appropriate for the research question, and the Discussion is appropriately cautious about clinical applicability. The synthetic metadata condition is a sensible perturbation test. However, the value of the results depends entirely on two unverified preconditions: that the textual image descriptions are a faithful and label-independent representation of the photographs, and that the analyzed 265-image subset is representative of the stated 300-image sample.

major comments (3)
  1. [§2.4] The central precondition of the study is that the structured textual descriptions accurately and completely convey the fundus image content without incorporating the ground-truth labels. Section 2.4 states only that 'all image content was conveyed through structured textual descriptions' and does not identify who authored the descriptions, whether they were derived from clinical reports, automated captioning, or the labels themselves, or whether the authors were blinded to the outcome. This matters because the Appendix prompt instructs the model to treat panretinal laser scars as ICDR level 4; if the descriptions were written with knowledge of the labels, then a correct ICDR 4 classification would simply recover a label-derived phrase rather than demonstrate reasoning. The same issue affects the glaucoma result: if the descriptions omit or include optic-disc detail in a label-dependent way, the poor F1 could be an artifact of the description content. The authors must provide the full description-generation protocol and, ideally, sample descriptions with the corresponding images and labels, or at least a clear statement that the descriptions were produced by graders blinded to the reference standard and validated for accuracy and completeness.
  2. [§3] The manuscript states that 300 images were selected, but all reported class counts in Section 3 sum to 265, not 300: ICDR counts 198+27+22+15+3 = 265, DR referral counts 220+45 = 265, and glaucoma referral counts 206+59 = 265. The 35 excluded images are never mentioned or explained. If the exclusions are related to image quality, missing metadata, or description-generation failures, the reported accuracy, kappa, and McNemar results may be biased. The authors must report the number of excluded images, the reasons for exclusion, and ideally a comparison of included versus excluded cases on available variables. Without this, all denominators and agreement estimates rest on an unexplained subset, and the 67.5% ICDR accuracy and other headline figures cannot be taken at face value.
  3. [§2.4/§2.6] The evaluation appears to use a single GPT-4 response per prompt, accessed through the ChatGPT platform with 'temperature and sampling parameters left unchanged.' This gives no information about run-to-run variability, and the paper reports no confidence intervals around the point estimates. The central null finding that metadata does not change predictions (McNemar p > 0.05, change rates under 7%) is only meaningful if the underlying predictions are stable. With a stochastic model, a single run cannot distinguish a true null effect from sampling noise, especially for the small change rates reported. The authors should either run each prompt multiple times and report the distribution of metrics and prediction-change rates, or justify why single-run outputs are sufficient for the claims made.
minor comments (5)
  1. [§2.7] There is a typographical error in 'thestatsmodels library'; it should read 'the statsmodels library.'
  2. [Appendix] The output format templates contain mismatched braces and parentheses, e.g., 'Yes/No { with a brief explanation' appears twice; these should be corrected to avoid ambiguity in the prompt specification.
  3. [§2.5] The synthetic metadata generation rule assigns 20% of ages as '≥90' and the rest as random integers between 20 and 89; the paper should state whether this distribution was chosen to match mBRSET or for another reason, and how the remaining variables were sampled independently.
  4. [§3] For the McNemar tests that yield p-values of 1.00, the paper should report the number of discordant pairs; a p-value of 1.00 can arise from very few changes, and without the discordant cell counts the reader cannot assess the power of the comparison.
  5. [References] Reference [15] is titled 'Evaluating Large Language Models for Simulated Ophthalmic Decision-Making,' which closely resembles the current manuscript's title; please confirm this is a distinct prior publication and clarify the relationship.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity identified; the evaluation is self-contained against external labels and involves no fitted parameters or self-citation-dependent claims.

full rationale

The study is an empirical benchmark, not a derivation. GPT-4's outputs are prompted with hand-specified rules (ICDR scoring, referral thresholds, CDR cutoff) and compared with fixed dataset labels from mBRSET. No parameter is fitted to the test labels, and no prediction is statistically forced by an earlier fit, so the fitted-input and self-definitional patterns do not apply. The cited prior work, including mBRSET and earlier LLM ophthalmology evaluations, supplies dataset provenance and context rather than the load-bearing justification for the headline numbers; the central empirical results come from the current experiment's prompts and external labels. The DR-referral and glaucoma-referral tasks are thresholded versions of the model's own ICDR and CDR outputs, so their performance is a deterministic function of those outputs, but this is an evaluation-design property, not a circular derivation: the ground-truth referral labels are external, and the paper does not claim to predict ICDR from the referral label. One substantive methodological gap is that the manuscript never discloses who authored the structured textual image descriptions or whether they were blinded to the reference labels (Section 2.4). If those descriptions were generated from the labels, the kappa/F1 values would be partly circular; however, the text does not exhibit that reduction, and the limitations section acknowledges only reduced image granularity, not label leakage. A separate numerical inconsistency (Section 3 counts sum to 265 rather than the stated 300 images) undermines reproducibility but is not a circularity. These are correctness and reporting risks, not demonstrated circular steps, so the score is 0.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The evaluation has no fitted model parameters. The hand-chosen synthetic metadata rule is tangential. The load-bearing assumptions are the fidelity and label-independence of the text descriptions, the correctness of the reference labels, the stability of a single GPT-4 run, and the randomness of the 35-case attrition.

free parameters (1)
  • Synthetic metadata generation rule = Age uniform 20-89 plus 20% '>=90'; binary variables random
    Hand-chosen rule for generating synthetic patient metadata; affects only the synthetic condition, not the main finding that metadata has no effect.
assumptions (4)
  • domain assumption mBRSET reference labels are correct for all 265 analyzed images.
    Ground truth comes from an external annotated dataset; label errors would shift all performance metrics.
  • ad hoc to paper Textual image descriptions faithfully represent the fundus images without loss or label leakage.
    The paper never describes how these descriptions were created or who authored them; the entire evaluation depends on this mapping.
  • ad hoc to paper Single-run GPT-4 outputs are stable enough for the reported statistics.
    No seed, default temperature, one run per prompt; stochastic decoding could change borderline classifications.
  • ad hoc to paper The 35 images lost between the stated 300 and the analyzed 265 are missing at random.
    No exclusion criteria or handling of missing data is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating Large Language Models for Multimodal Simulated Ophthalmic Decision-Making in Diabetic Retinopathy and Glaucoma Screening." pith.science (2026). https://pith.science/paper/YYPVRWW6

@misc{pith2026250701278,
  author       = {Pith},
  title        = {Pith review of: Evaluating Large Language Models for Multimodal Simulated Ophthalmic Decision-Making in Diabetic Retinopathy and Glaucoma Screening},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YYPVRWW6}},
  note         = {Machine review of arXiv:2507.01278}
}
read the original abstract

Large language models (LLMs) can simulate clinical reasoning based on natural language prompts, but their utility in ophthalmology is largely unexplored. This study evaluated GPT-4's ability to interpret structured textual descriptions of retinal fundus photographs and simulate clinical decisions for diabetic retinopathy (DR) and glaucoma screening, including the impact of adding real or synthetic clinical metadata. We conducted a retrospective diagnostic validation study using 300 annotated fundus images. GPT-4 received structured prompts describing each image, with or without patient metadata. The model was tasked with assigning an ICDR severity score, recommending DR referral, and estimating the cup-to-disc ratio for glaucoma referral. Performance was evaluated using accuracy, macro and weighted F1 scores, and Cohen's kappa. McNemar's test and change rate analysis were used to assess the influence of metadata. GPT-4 showed moderate performance for ICDR classification (accuracy 67.5%, macro F1 0.33, weighted F1 0.67, kappa 0.25), driven mainly by correct identification of normal cases. Performance improved in the binary DR referral task (accuracy 82.3%, F1 0.54, kappa 0.44). For glaucoma referral, performance was poor across all settings (accuracy ~78%, F1 <0.04, kappa <0.03). Metadata inclusion did not significantly affect outcomes (McNemar p > 0.05), and predictions remained consistent across conditions. GPT-4 can simulate basic ophthalmic decision-making from structured prompts but lacks precision for complex tasks. While not suitable for clinical use, LLMs may assist in education, documentation, or image annotation workflows in ophthalmology.

Figures

Figures reproduced from arXiv: 2507.01278 by the authors.

Figure 1
Figure 1. Experiments and evaluation framework. 2.2 Dataset The source dataset was the Brazilian Ophthalmological Dataset (mBRSET), which con￾tains annotated retinal fundus photographs with labels for diabetic retinopathy severity, referral recommendations, and optic nerve evaluations [3, 19]. From this dataset, a sub￾set of 300 images was randomly selected for this study. Each image was paired with corresponding demographic … view at source ↗
Figure 2
Figure 2. Imaging examples of normal fundus, diabetic retinopathy and increased cup [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. An Agentic AI Framework Overcomes Fundamental Limitations of Large Language Models for Glaucoma Detection from Fundus Photography

    cs.AI 2026-08 conditional novelty 5.0 of 10

    A three-step agentic workflow with LLM function calling and reflection improved glaucoma classification, CDR estimation, and repeatability over LLM-alone baselines, approaching specialist-level accuracy.

Reference graph

Works this paper leans on

20 extracted references · 19 canonical work pages · cited by 1 Pith paper

  1. [15]

    Evaluating Large Language Models for Simulated Ophthalmic Decision-Making

    Liu X, Zhang T, Chen M, et al. Evaluating Large Language Models for Simulated Ophthalmic Decision-Making. Ophthalmol Sci. 2023

  2. [1]

    IDF Diabetes Atlas: Global, regional and country-level diabetes prevalence estimates for 2021 and projections for 2045

    Sun H, Saeedi P, Karuranga S, et al. IDF Diabetes Atlas: Global, regional and country-level diabetes prevalence estimates for 2021 and projections for 2045. Dia- betes Res Clin Pract.2022;183:109119

  3. [2]

    Global Prevalence of Diabetic Retinopathy and Projection of Burden through 2045: Systematic Review and Meta-analysis

    Teo ZL, Tham YC, Yu M, et al. Global Prevalence of Diabetic Retinopathy and Projection of Burden through 2045: Systematic Review and Meta-analysis. Ophthal- mology. 2021;128(11):1580-1591

  4. [3]

    Artificial intelligence for telemedicine diabetic retinopathy screening: a review

    Nakayama LF, Ribeiro LZ, Novaes F, et al. Artificial intelligence for telemedicine diabetic retinopathy screening: a review. Ann Med.2023;55(2):2258149

  5. [4]

    The pathophysiology and treatment of glau- coma: a review

    Weinreb RN, Aung T, Medeiros F A. The pathophysiology and treatment of glau- coma: a review. JAMA. 2014;311(18):1901-1911

  6. [5]

    Evaluating the outcome of screening for glau- coma using colour fundus photography-based referral criteria in a teleophthalmology screening programme for diabetic retinopathy

    Tan RE, Teo KYC, Husain R, et al. Evaluating the outcome of screening for glau- coma using colour fundus photography-based referral criteria in a teleophthalmology screening programme for diabetic retinopathy. Br J Ophthalmol.2023

  7. [6]

    The effect of optic disc diameter on ver- tical cup to disc ratio percentiles in a population based cohort: the Blue Mountains Eye Study

    Crowston JG, Hopley CR, Healey PR, et al. The effect of optic disc diameter on ver- tical cup to disc ratio percentiles in a population based cohort: the Blue Mountains Eye Study. Br J Ophthalmol.2004;88(6):766-770

  8. [7]

    Language models are few-shot learners

    Brown T, Mann B, Ryder N, et al. Language models are few-shot learners. Adv Neur Inf Proc Sys.2020;33:1877–1901

Show all 20 references
  1. [8]

    Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models

    Kung TH, Cheatham M, Medenilla A, et al. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLoS Digit Health. 2023;2(2):e0000198

  2. [9]

    Performance of ChatGPT-4 in answering questions from the Brazilian National Examination for Medical Degree Revalidation

    Gobira M, Nakayama LF, Moreira R, et al. Performance of ChatGPT-4 in answering questions from the Brazilian National Examination for Medical Degree Revalidation. Rev Assoc Med Bras.2023;69(10)

  3. [10]

    Evaluating the performance of ChatGPT in ophthalmology: An analysis of its successes and shortcomings

    Antaki F, Touma S, Milad D, et al. Evaluating the performance of ChatGPT in ophthalmology: An analysis of its successes and shortcomings. Ophthalmol Sci. 2023;3(4):100324

  4. [11]

    Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs.JAMA

    Gulshan V, Peng L, Coram M, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs.JAMA. 2016;316(22):2402–2410

  5. [12]

    Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes

    Ting DSW, Cheung CY, Lim G, et al. Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. JAMA. 2017;318(22):2211–2223

  6. [13]

    Interobserver agree- ment for the clinical assessment of optic discs

    Garway-Heath DF, Ruben S, Viswanathan A, Hitchings RA. Interobserver agree- ment for the clinical assessment of optic discs. Br J Ophthalmol.1998;82(6):644–647

  7. [14]

    Diagnostic accuracy of optic nerve head imaging for glaucoma: a systematic review and meta-analysis

    Cheung CY, Rulli E, Schulze A, et al. Diagnostic accuracy of optic nerve head imaging for glaucoma: a systematic review and meta-analysis. Lancet Digit Health. 2018;1(4):e172–e182. 10

  8. [16]

    GPT-4 Technical Report

    OpenAI. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774.2023

  9. [17]

    mBRSET: The Brazilian Ophthal- mological Dataset for Diabetic Retinopathy and Multimodal Retinal AI

    Nakayama LF, Malerbi FK, Regatieri CV, et al. mBRSET: The Brazilian Ophthal- mological Dataset for Diabetic Retinopathy and Multimodal Retinal AI. In prepa- ration. 2024

  10. [18]

    Large Language Models in Medicine: Current Limitations and Future Potential

    Stolz D, Keestra S, et al. Large Language Models in Medicine: Current Limitations and Future Potential. npj Digit Med.2023

  11. [19]

    A portable retina fundus photos dataset for clinical, demographic, and diabetic retinopathy prediction

    Wu C, Restrepo D, Nakayama LF, et al. A portable retina fundus photos dataset for clinical, demographic, and diabetic retinopathy prediction. Sci Data.2025;12(1):323

  12. [20]

    Proposed international clinical diabetic retinopathy and diabetic macular edema disease severity scales

    Wilkinson CP, Ferris FL, Klein RE, et al. Proposed international clinical diabetic retinopathy and diabetic macular edema disease severity scales. Ophthalmology. 2003;110(9):1677–1682. 11 Appendix: Example of Real and Synthetic Metadata Input Real Metadata Prompt Please review...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.