Pith. sign in

REVIEW 3 major objections 5 minor 52 references

BEnchmarking LLMs for Ophthalmology (BELO) for Ophthalmological Knowledge and Reasoning

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a 900-question, expert-refined benchmark, BELO, can serve as a standardized test of both accuracy and reasoning for ophthalmology LLMs, and that on its first run the o1 model leads while all models' explanations lag.

desk verdict A genuinely useful expert-curated ophthalmology QA resource, but the reasoning score in Table 2 rests on unvalidated text-generation metrics; the accuracy results are solid. read the letter →

arxiv 2507.15717 v1 pith:4OEIIU5W submitted 2025-07-21 cs.CL cs.AI

Sahana Srinivasan , Xuguang Ai , Thaddaeus Wai Soon Lo , Aidan Gilson , Minjie Zou , Ke Zou , Hyunjae Kim , Mingjia Yang
show 24 more authors
Krithi Pushpanathan Samantha Yew Wan Ting Loke Jocelyn Goh Yibing Chen Yiming Kong Emily Yuelei Fu Michelle Ongyong Hui Kristen Nwanyanwu Amisha Dave Kelvin Zhenghao Li Chen-Hsin Sun Mark Chia Gabriel Dawei Yang Wendy Meihua Wong David Ziyou Chen Dianbo Liu Maxwell Singer Fares Antaki Lucian V Del Priore Jost Jonas Ron Adelman Qingyu Chen Yih-Chung Tham
This is my paper
classification cs.CLcs.AI
keywords ophthalmologybenchmarklargelanguagemodelsclinicalreasoningevaluationmultiple-choicequestionstext-generationmetricsexpert-curateddatasethold-outLLMleaderboard
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

BELO (BEnchmarking LLMs for Ophthalmology) is a new evaluation benchmark built from 900 multiple-choice questions filtered from five medical QA datasets and refined through multiple rounds of checking by 13 ophthalmologists. The authors' central claim is that BELO supplies a standardized, held-out test that measures not only whether a language model picks the correct answer but also the quality of the reasoning it gives, by comparing model explanations against expert-written reference explanations. To show the benchmark works, they ran six current LLMs through it: the o1 model achieved the highest accuracy (0.882) and macro-F1 (0.890), while explanation-quality scores from five text-generation metrics were uniformly modest, with the best aggregate reasoning score at 0.804 on a 0-1 scale. If this claim holds, the field gains a common ruler for comparing ophthalmology LLMs and a concrete target—improving explanation quality—that accuracy alone had hidden.

What carries the argument

The central object is the BELO dataset itself: 900 MCQ items drawn from BCSC, BioASQ, MedMCQA, MedQA, and PubMedQA, extracted by keyword matching plus a fine-tuned PubMedBERT classifier, then graded, amended, and adjudicated by 13 ophthalmologists so that every question carries a reference explanation. The evaluation machinery has two parts: accuracy and macro-F1 measure whether the chosen answer is right, while five text-generation metrics (ROUGE-L, METEOR, BERTScore, BARTScore, AlignScore) measure how closely each model's explanation matches the expert reference, collapsed into a weighted normalized score. The dataset is kept as a hold-out, evaluation-only set with a public leaderboard, so future models face the same questions without prior exposure.

What would settle it

Have two or more ophthalmologists blindly rate, on a 5-point scale, a random sample (say 100) of model explanations from BELO for reasoning quality, then compute the rank correlation between the mean expert ratings and the five text-generation metric scores; a near-zero or negative correlation would refute the claim that the benchmark measures clinical reasoning.

Watch

Extended reading notes

Core claim

The paper's discovery is a validation set rather than a new model: 900 expert-checked ophthalmology MCQs, each paired with a reference explanation, that can rank LLMs on two dimensions at once. On the demonstration run, answer accuracy and macro-F1 clearly separated the models (o1 first at 0.882/0.890, Gemini 1.5 Pro last at 0.596/0.639), but the five explanation metrics (ROUGE-L, METEOR, BERTScore, BARTScore, AlignScore) all produced low absolute scores across the board once rescaled, with o1 again first on the weighted normalized score (0.804) and Gemini 1.5 Pro last (0.037). The authors interpret this gap as evidence that current models can often identify the right answer yet provide explanations that fall short of expert-level reasoning.

Load-bearing premise

The load-bearing premise is that the five text-generation metrics, when comparing a model's explanation to the expert reference, genuinely measure the quality of clinical reasoning; if those metrics track style or wording instead of reasoning, BELO's claim to assess reasoning is not supported.

Editorial extensions

If this is right

  • Any future LLM in ophthalmology can now be compared against six already-published scores on the same 900 questions, because BELO is a fixed, held-out benchmark.
  • The near-universally low explanation-metric scores indicate that reasoning quality, not answer selection, is the current bottleneck in ophthalmology LLMs.
  • The benchmark's composition—572 exam-style questions from MedMCQA and 260 from BCSC out of 900—means current results mostly measure knowledge recall rather than real-world clinical management.
  • Because the dataset is withheld from public release, models cannot be legitimately fine-tuned on it, preserving the validity of future head-to-head comparisons.
  • The planned extensions to visual question answering and clinical scenario tasks will broaden the benchmark beyond text-only MCQs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The five text-generation metrics are never validated against expert ratings in the paper; a direct calibration study comparing metric scores with ophthalmologists' judgments on the same outputs would show whether the reasoning ranking is measuring reasoning or just lexical similarity.
  • The low absolute text-generation scores could partly reflect reference explanations being longer and more detailed than model outputs, so a length-controlled or rubric-based evaluation might reorder the models relative to the reported ranking.
  • Because the dataset is evaluation-only and held out, model developers cannot build a development set from it; without a companion training or validation split, BELO cannot be used to drive iterative improvement, only to assess finished models.
  • The extraction pipeline's keyword method had 0.718 sensitivity on MedMCQA, meaning nearly a third of true ophthalmology questions were missed; a future version could recover those by applying the higher-sensitivity classifier-only approach followed by expert screening.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces BELO, a benchmark of 900 ophthalmology multiple-choice questions curated from five existing medical QA datasets, with multiple rounds of expert checking by ophthalmologists and expert-written reasoning references. To demonstrate the benchmark's utility, the authors evaluate six LLMs on accuracy, macro-F1, and five text-generation metrics, and additionally report a small qualitative evaluation by two ophthalmologists on 50 outputs from three models. The central claim is that BELO is a robust, clinically relevant benchmark for assessing both ophthalmological knowledge and clinical reasoning.

Significance. If the reasoning-assessment claim were supported, BELO would be a valuable community resource: it is externally grounded, expert-curated, hold-out in design, and accompanied by a public leaderboard. The curation pipeline is clearly described, the expert checking is a genuine strength, and the hold-out decision is methodologically sound. However, the reasoning dimension rests on an unvalidated composite of text-generation metrics, and the human validation is too small, too biased, and not linked to those metrics. Since the paper's advertised novelty over prior ophthalmology benchmarks is precisely the evaluation of reasoning, this validity gap is load-bearing. The accuracy and macro-F1 results are informative, but the reasoning claims need either validation or reframing before the central conclusion is warranted.

major comments (3)
  1. [Methods, 'Demonstrative Quantitative Analysis'; Table 2] The composite 'weighted normalized score' is defined as the equal-weighted average of five min-max normalized text-generation metrics (ROUGE-L, METEOR, BERTScore, BARTScore, AlignScore). These are reference-based lexical and semantic similarity measures; the paper provides no evidence that any of them tracks clinical reasoning quality in ophthalmology. The Abstract and Discussion use this composite to conclude that low scores indicate 'room for improvement in clinical reasoning,' so the reasoning claim is entirely dependent on an unvalidated proxy. Please either validate these metrics against expert judgments (e.g., correlation or agreement with ophthalmologist ratings) or explicitly reframe the reasoning claims as claims about textual similarity to expert-written references.
  2. [Methods, 'Demonstrative Qualitative Evaluation'; Supplementary Table 5] The qualitative evaluation is restricted to 50 questions on which GPT-4o, Llama-3-8B, and Gemini 1.5 Pro all selected the correct answer, and it excludes o1, o3-mini, and DeepSeek-R1—the models that rank highest on the automatic metrics. No correlation or agreement is reported between the ophthalmologists' Likert ratings and any of the five text-generation metrics. Consequently, this human evaluation cannot validate the metric-based reasoning ranking in Table 2. Please report a suitable validation analysis, or remove the implication that the qualitative evaluation supports the quantitative reasoning metric.
  3. [Methods, 'Demonstrative Quantitative Analysis'; Figure 3 legend; Table 2] The macro-F1 subset is inconsistently specified. The Methods state that macro-F1 was computed using 'only questions with four options, which were questions from BCSC and MedMCQA,' but the Figure 3 legend and Table 2 report 872 questions and explicitly include MedQA. Since BCSC (260) plus MedMCQA (572) totals 832, while adding MedQA (40) gives 872, this discrepancy materially affects reproducibility. Please correct the Methods or the figure/table and state the actual subset used.
minor comments (5)
  1. [Abstract] The abstract reports o1's macro-F1 as '0.78, 95% CI: 0.869–0.910,' which is inconsistent with Table 2's value of 0.890 and the same CI; the point estimate appears to be a typo.
  2. [Table 2] The Accuracy row does not report a 95% confidence interval for Llama-3-8B, although intervals are given in the Results text and for all other models in the table.
  3. [Methods, 'QA Quality Check'] The description of the manual check lists 'two optometrists (SY, WTL)' and then 'six research staff (SS, XA, TWSL, SY, WTL, YC),' with SY and WTL appearing in both lists; the personnel counts should be clarified.
  4. [Throughout] The terms 'comprehensiveness' (Abstract) and 'completeness' (Methods, Results, Supplementary Table S5) are used interchangeably for the same construct; this should be harmonized.
  5. [Table 2, note] The score is called a 'weighted normalized score' even though the five normalized metrics are averaged with equal weights; 'equal-weight normalized score' would be less misleading.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: BELO is an externally grounded benchmark; the reasoning metrics are standard external tools, and the paper's self-citations are methodological rather than load-bearing.

full rationale

The paper's central claim is that BELO provides a clinically relevant benchmark. The benchmark is constructed from five existing external datasets (BCSC, BioASQ, MedMCQA, MedQA, PubMedQA) with expert checking and reasoning amendment; none of the 900 items is generated from the models being evaluated, and no model output is used to define the reference answers. The only fitted component is the PubMedBERT ophthalmology-questions classifier, which is fine-tuned on MedMCQA subject labels and evaluated against those labels (sensitivity 0.937, specificity 0.962); this is a curation step, not a prediction of LLM performance. The five text-generation metrics (ROUGE-L, METEOR, BERTScore, BARTScore, AlignScore) are standard external NLP instruments with published definitions; the weighted normalized score is an explicit min-max average of those measured values, so the ranking it produces is a stated descriptive aggregation rather than a prediction forced by construction. The self-citations (e.g., ref. 12 for reasoning-quality criteria and for use of text-generation metrics) are methodological precedents and are not used to justify the benchmark's validity or to forbid alternatives; the human expert evaluation provides an independent, if limited, check. The qualitative evaluation's restriction to questions all three models answered correctly, and the absence of a correlation between automatic metrics and expert ratings, are validity limitations for the 'reasoning' interpretation; they do not make the derivation circular. No step in the paper reduces a predicted quantity to an input by definition or imports a uniqueness claim from prior same-author work.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim depends on the assumptions that expert explanations are gold standards, that text-generation metrics capture reasoning quality, that source dataset answers are correct, and that the hold-out design prevents contamination. These are domain assumptions, not derived results.

free parameters (1)
  • Min-max normalization of text-generation metrics = 0 to 1 per metric across the six evaluated models
    The paper assigns 0 to the lowest observed score and 1 to the highest for each metric, then averages equally to produce a 'weighted normalized score'. The scale depends on the particular set of models evaluated, so the aggregate is relative rather than an absolute quality measure.
assumptions (4)
  • domain assumption Expert-written explanations in BELO are gold-standard references for correct clinical reasoning.
    The benchmark scores model reasoning by comparing outputs to explanations written by a team of ophthalmologists; this assumes these are definitive, which is not independently established.
  • domain assumption Text-generation metrics (ROUGE-L, METEOR, BERTScore, BARTScore, AlignScore) are valid measures of clinical reasoning quality.
    The paper uses these metrics to quantify reasoning without validating them against human judgments on a full sample; the human validation covers only 50 correct-answer cases for three models.
  • domain assumption Source datasets (BCSC, MedMCQA, MedQA, BioASQ, PubMedQA) provide accurate ground-truth answers and questions.
    BELO inherits labels and question quality from these sources; only a subset of items were expert-validated, so errors in source answers would propagate.
  • domain assumption The held-out design of BELO prevents LLM training-data contamination.
    The paper assumes that keeping the benchmark unreleased prevents memorization, but the source datasets (e.g., MedQA, MedMCQA) are public and may already be in model training corpora.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BEnchmarking LLMs for Ophthalmology (BELO) for Ophthalmological Knowledge and Reasoning." pith.science (2026). https://pith.science/paper/4OEIIU5W

@misc{pith2026250715717,
  author       = {Pith},
  title        = {Pith review of: BEnchmarking LLMs for Ophthalmology (BELO) for Ophthalmological Knowledge and Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4OEIIU5W}},
  note         = {Machine review of arXiv:2507.15717}
}
read the original abstract

Current benchmarks evaluating large language models (LLMs) in ophthalmology are limited in scope and disproportionately prioritise accuracy. We introduce BELO (BEnchmarking LLMs for Ophthalmology), a standardized and comprehensive evaluation benchmark developed through multiple rounds of expert checking by 13 ophthalmologists. BELO assesses ophthalmology-related clinical accuracy and reasoning quality. Using keyword matching and a fine-tuned PubMedBERT model, we curated ophthalmology-specific multiple-choice-questions (MCQs) from diverse medical datasets (BCSC, MedMCQA, MedQA, BioASQ, and PubMedQA). The dataset underwent multiple rounds of expert checking. Duplicate and substandard questions were systematically removed. Ten ophthalmologists refined the explanations of each MCQ's correct answer. This was further adjudicated by three senior ophthalmologists. To illustrate BELO's utility, we evaluated six LLMs (OpenAI o1, o3-mini, GPT-4o, DeepSeek-R1, Llama-3-8B, and Gemini 1.5 Pro) using accuracy, macro-F1, and five text-generation metrics (ROUGE-L, BERTScore, BARTScore, METEOR, and AlignScore). In a further evaluation involving human experts, two ophthalmologists qualitatively reviewed 50 randomly selected outputs for accuracy, comprehensiveness, and completeness. BELO consists of 900 high-quality, expert-reviewed questions aggregated from five sources: BCSC (260), BioASQ (10), MedMCQA (572), MedQA (40), and PubMedQA (18). A public leaderboard has been established to promote transparent evaluation and reporting. Importantly, the BELO dataset will remain a hold-out, evaluation-only benchmark to ensure fair and reproducible comparisons of future models.

Figures

Figures reproduced from arXiv: 2507.15717 by the authors.

Figure 1
Figure 1. Overview of the BELO Dataset Curation Pipeline [PITH_FULL_IMAGE:figures/full_fig_p023_1.png] view at source ↗
Figure 2
Figure 2. End-to-End Pipeline for Building the BELO Benchmark Dataset [PITH_FULL_IMAGE:figures/full_fig_p024_2.png] view at source ↗
Figure 3
Figure 3. Comparison of Model Performance on BELO using A) Accuracy and B) Macro-F1 [PITH_FULL_IMAGE:figures/full_fig_p025_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

52 extracted references · 42 canonical work pages

  1. [1]

    Attention is All you Need [Internet]

    Vaswani A, Shazeer N, Parmar N, et al. Attention is All you Need [Internet]. In: Guyon I, Luxburg UV, Bengio S, et al., editors. Advances in Neural Information Processing Systems. Curran Associates, Inc.; 2017. Available from: https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a8 45aa-Paper.pdf

    work page 2017

  2. [2]

    Towards Expert-Level Medical Question Answering with Large Language Models [Internet]

    Singhal K, Tu T, Gottweis J, et al. Towards Expert-Level Medical Question Answering with Large Language Models [Internet]. 2023;Available from: https://arxiv.org/abs/2305.09617

    arXiv 2023

  3. [3]

    Towards Conversational Diagnostic AI [Internet]

    Tu T, Palepu A, Schaekermann M, et al. Towards Conversational Diagnostic AI [Internet]. 2024 [cited 2025 May 14];Available from: http://arxiv.org/abs/2401.05654

    arXiv 2024

  4. [4]

    Opportunities and challenges for ChatGPT and large language models in biomedicine and health

    Tian S, Jin Q, Yeganova L, et al. Opportunities and challenges for ChatGPT and large language models in biomedicine and health. Briefings in Bioinformatics 2024;25(1):bbad493

    work page 2024

  5. [5]

    Benchmarking large language models for biomedical natural language processing applications and recommendations

    Chen Q, Hu Y, Peng X, et al. Benchmarking large language models for biomedical natural language processing applications and recommendations. Nat Commun 2025;16(1):1–16

    work page 2025

  6. [6]

    Radiology-Llama2: Best-in-Class Large Language Model for Radiology [Internet]

    Liu Z, Li Y, Shu P, et al. Radiology-Llama2: Best-in-Class Large Language Model for Radiology [Internet]. 2023;Available from: https://arxiv.org/abs/2309.06419

    arXiv 2023

  7. [7]

    Towards a general-purpose foundation model for computational pathology

    Chen RJ, Ding T, Lu MY, et al. Towards a general-purpose foundation model for computational pathology. Nature Medicine 2024;30(3):850–62

    work page 2024

  8. [8]

    Language Enhanced Model for Eye (LEME): An Open-Source Ophthalmology-Specific Large Language Model [Internet]

    Gilson A, Ai X, Xie Q, et al. Language Enhanced Model for Eye (LEME): An Open-Source Ophthalmology-Specific Large Language Model [Internet]. 2024 [cited 2024 Dec 3];Available from: http://arxiv.org/abs/2410.03740

    work page arXiv 2024

Show all 52 references
  1. [9]

    Utilizing Large Language Models to Simplify Radiology Reports: a comparative analysis of ChatGPT3.5, ChatGPT4.0, Google Bard, and Microsoft Bing

    Doshi R, Amin K, Khosla P, Bajaj S, Chheang S, Forman HP. Utilizing Large Language Models to Simplify Radiology Reports: a comparative analysis of ChatGPT3.5, ChatGPT4.0, Google Bard, and Microsoft Bing. medRxiv [Internet] 2023;Available from: https://www.medrxiv.org/content/e...

    2023

  2. [10]

    Can OpenAI’s New o1 Model Outperform Its Predecessors in Common Eye Care Queries? Ophthalmology Science [Internet] 2025 [cited 2025 Apr 26];5(4)

    Pushpanathan K, Zou M, Srinivasan S, et al. Can OpenAI’s New o1 Model Outperform Its Predecessors in Common Eye Care Queries? Ophthalmology Science [Internet] 2025 [cited 2025 Apr 26];5(4). Available from: https://www.ophthalmologyscience.org/article/S2666- 9145(25)00043-0/fulltext

    2025

  3. [11]

    EyeGPT Ophthalmic Assistant with Large Language Models [Internet]

    Chen X, Zhao Z, Zhang W, et al. EyeGPT Ophthalmic Assistant with Large Language Models [Internet]. 2024;Available from: httpsarxiv.orgabs2403.00840

    2024 arXiv

  4. [12]

    Can OpenAI o1’s Enhanced Reasoning Capabilities Extend to Ophthalmology? A Benchmark Study Across Large Language Models and Text Generation Metrics [Internet]

    Srinivasan S, Ai X, Zou M, et al. Can OpenAI o1’s Enhanced Reasoning Capabilities Extend to Ophthalmology? A Benchmark Study Across Large Language Models and Text Generation Metrics [Internet]. 2025 [cited 2025 Jan 17];Available from: https://www.ssrn.com/abstract=5091694

    2025

  5. [13]

    ChatGPT-Generated Differential Diagnosis Lists for Complex Case–Derived Clinical Vignettes: Diagnostic Accuracy Evaluation

    Hirosawa T, Kawamura R, Harada Y, et al. ChatGPT-Generated Differential Diagnosis Lists for Complex Case–Derived Clinical Vignettes: Diagnostic Accuracy Evaluation. JMIR Med Inform 2023;11:e48808

    2023

  6. [14]

    Performance of ChatGPT in Ophthalmic Registration and Clinical Diagnosis: Cross-Sectional Study

    Ming S, Yao X, Guo X, et al. Performance of ChatGPT in Ophthalmic Registration and Clinical Diagnosis: Cross-Sectional Study. J Med Internet Res 2024;26:e60226

    2024

  7. [15]

    Can off-the-shelf visual large language models detect and diagnose ocular diseases from retinal photographs? BMJ Open Ophth [Internet] 2025 [cited 2025 May 15];10(1)

    Srinivasan S, Ji H, Chen DZ, et al. Can off-the-shelf visual large language models detect and diagnose ocular diseases from retinal photographs? BMJ Open Ophth [Internet] 2025 [cited 2025 May 15];10(1). Available from: https://bmjophth.bmj.com/content/10/1/e002076

    2025

  8. [16]

    ChatGPT: the future of discharge summaries? The Lancet Digital Health 2023;5(3):e107–8

    Patel SB, Lam K. ChatGPT: the future of discharge summaries? The Lancet Digital Health 2023;5(3):e107–8

    2023

  9. [17]

    Large Language Models Seem Miraculous, but Science Abhors Miracles

    Szolovits P. Large Language Models Seem Miraculous, but Science Abhors Miracles. NEJM AI 2024;1(6):AIp2300103

    2024

  10. [18]

    Medical Ethics of Large Language Models in Medicine

    Ong JCL, Chang SY-H, William W, et al. Medical Ethics of Large Language Models in Medicine. NEJM AI 2024;1(7):AIra2400038

    2024

  11. [19]

    Google for Developers

    MedGemma | Health AI Developer Foundations [Internet]. Google for Developers. [cited 2025 Jun 3];Available from: https://developers.google.com/health-ai-developer- foundations/medgemma

    2025

  12. [20]

    Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review

    Bedi S, Liu Y, Orr-Ewing L, et al. Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review. JAMA 2025;333(4):319–28

    2025

  13. [21]

    Measuring Massive Multitask Language Understanding [Internet]

    Hendrycks D, Burns C, Basart S, et al. Measuring Massive Multitask Language Understanding [Internet]. 2021 [cited 2025 May 19];Available from: http://arxiv.org/abs/2009.03300

    2021 arXiv

  14. [22]

    HealthBench: Evaluating Large Language Models Towards Improved Human Health

    Arora RK, Wei J, Hicks RS, et al. HealthBench: Evaluating Large Language Models Towards Improved Human Health

  15. [23]

    PathVQA: 30000+ Questions for Medical Visual Question Answering

    He X, Zhang Y, Mou L, Xing E, Xie P. PathVQA: 30000+ Questions for Medical Visual Question Answering. arXiv preprint arXiv:200310286 2020

    2020

  16. [24]

    MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports

    Johnson AEW, Pollard TJ, Berkowitz SJ, et al. MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Scientific Data 2019;6(1):317

    2019

  17. [25]

    What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams [Internet]

    Jin D, Pan E, Oufattole N, Weng W-H, Fang H, Szolovits P. What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams [Internet]. 2020 [cited 2025 May 15];Available from: http://arxiv.org/abs/2009.13081

    2020 arXiv

  18. [26]

    EyecareGPT: Boosting Comprehensive Ophthalmology Understanding with Tailored Dataset, Benchmark and Model [Internet]

    Li S, Lin T, Lin L, et al. EyecareGPT: Boosting Comprehensive Ophthalmology Understanding with Tailored Dataset, Benchmark and Model [Internet]. 2025 [cited 2025 May 15];Available from: http://arxiv.org/abs/2504.13650

    2025 arXiv

  19. [27]

    Multi-OphthaLingua: A Multilingual Benchmark for Assessing and Debiasing LLM Ophthalmological QA in LMICs [Internet]

    Restrepo D, Wu C, Tang Z, et al. Multi-OphthaLingua: A Multilingual Benchmark for Assessing and Debiasing LLM Ophthalmological QA in LMICs [Internet]. 2024 [cited 2025 May 21];Available from: http://arxiv.org/abs/2412.14304

    2024 arXiv

  20. [28]

    Unveiling the clinical incapabilities: a benchmarking study of GPT-4V(ision) for ophthalmic multimodal image analysis

    Xu P, Chen X, Zhao Z, Shi D. Unveiling the clinical incapabilities: a benchmarking study of GPT-4V(ision) for ophthalmic multimodal image analysis. British Journal of Ophthalmology 2024;108(10):1384–9

    2024

  21. [29]

    Evaluating the Performance of ChatGPT in Ophthalmology: An Analysis of Its Successes and Shortcomings

    Antaki F, Touma S, Milad D, El-Khoury J, Duval R. Evaluating the Performance of ChatGPT in Ophthalmology: An Analysis of Its Successes and Shortcomings. Ophthalmology Science [Internet] 2023 [cited 2024 Aug 5];3(4). Available from: https://doi.org/10.1016/j.xops.2023.100324

    2023

  22. [30]

    Evaluating Large Language Models in Ophthalmology [Internet]

    Holmes J, Ye S, Li Y, et al. Evaluating Large Language Models in Ophthalmology [Internet]. 2023 [cited 2025 May 16];Available from: http://arxiv.org/abs/2311.04933

    2023 arXiv

  23. [31]

    OphthBench: A Comprehensive Benchmark for Evaluating Large Language Models in Chinese Ophthalmology [Internet]

    Zhou C, Wang J, Qin J, Wang Y, Sun L, Dai W. OphthBench: A Comprehensive Benchmark for Evaluating Large Language Models in Chinese Ophthalmology [Internet]. 2025 [cited 2025 May 29];Available from: http://arxiv.org/abs/2502.01243

    2025 arXiv

  24. [32]

    EYE-Llama, an in-domain large language model for ophthalmology

    Haghighi T, Gholami S, Sokol JT, et al. EYE-Llama, an in-domain large language model for ophthalmology. bioRxiv [Internet] 2024;Available from: https://www.biorxiv.org/content/early/2024/04/29/2024.04.26.591355

    2024

  25. [33]

    BioASQ-QA: A manually curated corpus for Biomedical Question Answering

    Krithara A, Nentidis A, Bougiatiotis K, Paliouras G. BioASQ-QA: A manually curated corpus for Biomedical Question Answering. Sci Data 2023;10(1):1–12

    2023

  26. [34]

    Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering

    Pal A, Umapathi LK, Sankarasubbu M. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In: Conference on health, inference, and learning. PMLR; 2022. p. 248–60

    2022

  27. [35]

    PubMedQA: A Dataset for Biomedical Research Question Answering

    Jin Q, Dhingra B, Liu Z, Cohen W, Lu X. PubMedQA: A Dataset for Biomedical Research Question Answering. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-I...

    2019

  28. [36]

    Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing

    Gu Y, Tinn R, Cheng H, et al. Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing. ACM Trans Comput Healthcare 2022;3(1):1–23

    2022

  29. [37]

    How Does ChatGPT Perform on the United States Medical Licensing Examination? The Implications of Large Language Models for Medical Education and Knowledge Assessment

    Gilson A, Safranek CW, Huang T, et al. How Does ChatGPT Perform on the United States Medical Licensing Examination? The Implications of Large Language Models for Medical Education and Knowledge Assessment. JMIR Med Educ 2023;9:e45312

    2023

  30. [38]

    Benchmarking Next-Generation Reasoning-Focused Large Language Models in Ophthalmology: A Head-to-Head Evaluation on 5,888 Items [Internet]

    Zou M, Srinivasan S, Lo TWS, et al. Benchmarking Next-Generation Reasoning-Focused Large Language Models in Ophthalmology: A Head-to-Head Evaluation on 5,888 Items [Internet]. arXiv.org. 2025 [cited 2025 May 15];Available from: https://arxiv.org/abs/2504.11186v1

    2025 arXiv

  31. [39]

    ROUGE: A Package for Automatic Evaluation of Summaries [Internet]

    Lin C-Y. ROUGE: A Package for Automatic Evaluation of Summaries [Internet]. In: Text Summarization Branches Out. Barcelona, Spain: Association for Computational Linguistics; 2004. p. 74–81.Available from: https://aclanthology.org/W04-1013

    2004

  32. [40]

    BERTScore: Evaluating Text Generation with BERT [Internet]

    Zhang T, Kishore V, Wu F, Weinberger KQ, Artzi Y. BERTScore: Evaluating Text Generation with BERT [Internet]. 2020;Available from: https://arxiv.org/abs/1904.09675

    2020 arXiv

  33. [41]

    BARTScore: Evaluating Generated Text as Text Generation [Internet]

    Yuan W, Neubig G, Liu P. BARTScore: Evaluating Generated Text as Text Generation [Internet]. 2021;Available from: https://arxiv.org/abs/2106.11520

    2021 arXiv

  34. [42]

    AlignScore: Evaluating Factual Consistency with a Unified Alignment Function [Internet]

    Zha Y, Yang Y, Li R, Hu Z. AlignScore: Evaluating Factual Consistency with a Unified Alignment Function [Internet]. 2023;Available from: https://arxiv.org/abs/2305.16739

    2023 arXiv

  35. [43]

    A visual-language foundation model for computational pathology

    Lu MY, Chen B, Williamson DFK, et al. A visual-language foundation model for computational pathology. Nat Med 2024;30(3):863–74

    2024

  36. [44]

    Developing and Evaluating Large Language Model– Generated Emergency Medicine Handoff Notes

    Hartman V, Zhang X, Poddar R, et al. Developing and Evaluating Large Language Model– Generated Emergency Medicine Handoff Notes. JAMA Network Open 2024;7(12):e2448723

    2024

  37. [45]

    CPMI-ChatGLM: parameter-efficient fine-tuning ChatGLM with Chinese patent medicine instructions

    Liu C, Sun K, Zhou Q, et al. CPMI-ChatGLM: parameter-efficient fine-tuning ChatGLM with Chinese patent medicine instructions. Sci Rep 2024;14(1):6403

    2024

  38. [46]

    Reducing hallucinations of large language models via hierarchical semantic piece

    Liu Y, Yang Q, Tang J, et al. Reducing hallucinations of large language models via hierarchical semantic piece. Complex Intell Syst 2025;11(5):231

    2025

  39. [47]

    Meteor: an automatic metric for MT evaluation with high levels of correlation with human judgments

    Lavie A, Agarwal A. Meteor: an automatic metric for MT evaluation with high levels of correlation with human judgments. In: Proceedings of the Second Workshop on Statistical Machine Translation. USA: Association for Computational Linguistics; 2007. p. 228–31

    2007

  40. [48]

    LMOD: A Large Multimodal Ophthalmology Dataset and Benchmark for Large Vision-Language Models [Internet]

    Qin Z, Yin Y, Campbell D, et al. LMOD: A Large Multimodal Ophthalmology Dataset and Benchmark for Large Vision-Language Models [Internet]. 2025 [cited 2025 May 19];Available from: http://arxiv.org/abs/2410.01620 Table 1: Compositions of the Curated BELO dataset Dataset BCSC Bi...

    2025 arXiv

  41. [49]

    medical QA dataset

    Collection of medical datasets We systematically reviewed and collected existing medical QA datasets (Figures 1 and 2). The search performed across February 2024, using multiple platforms, including Hugging Face, Google Scholar, PubMed Central, Kaggle, and Papers with Code. We...

    2024

  42. [50]

    Each question offers four options, out of which only one is correct

    Detailed Descriptions of the Datasets Used to Curate BELO Basic and Clinical Science Course (BCSC) The BCSC dataset consists of ophthalmology-related MCQs derived from the BCSC textbook, which is a comprehensive educational resource published by the American Academy of Ophthal...

    2013

  43. [51]

    glaucoma,

    Detailed Descriptions of the Methodologies Used to Extract Ophthalmological QAs 3.1 Details of the Keyword Matching Methodology for Ophthalmological QA Extraction We assembled a list of ophthalmology -specific terms (e.g. “glaucoma,” “macula,” “cataract”) informed by terminolo...

  44. [52]

    Definitions of The Text-Generation Metrics Used in This Study Five text-generation metrics were used in this study to evaluate model reasoning. The first text-generation metric used was the Recall-Oriented Understudy for Gisting Evaluation (ROUGE -L) which measures the longest...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.