Pith. sign in

REVIEW 4 major objections 4 minor 44 references

Can OpenAI o1 Reason Well in Ophthalmology? A 6,990-Question Head-to-Head Evaluation Study

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read OpenAI o1 answers ophthalmology MCQs more accurately than five rival models, yet its explanations rank behind GPT-4o and GPT-4 when scored by text-generation metrics.

desk verdict Solid accuracy benchmark for o1 in ophthalmology, but the reasoning claim rests on unvalidated text-similarity metrics and should be reframed. read the letter →

arxiv 2501.13949 v1 pith:XEGRWJMY submitted 2025-01-20 cs.CL cs.AI

Sahana Srinivasan , Xuguang Ai , Minjie Zou , Ke Zou , Hyunjae Kim , Thaddaeus Wai Soon Lo , Krithi Pushpanathan , Yiming Kong
show 9 more authors
Anran Li Maxwell Singer Kai Jin Fares Antaki David Ziyou Chen Dianbo Liu Ron A. Adelman Qingyu Chen Yih Chung Tham
This is my paper · ORCID
classification cs.CLcs.AI
keywords OpenAIo1largelanguagemodelsophthalmologymedicalquestionansweringMedMCQAreasoningevaluationtext-generationmetricsGPT-4o
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether OpenAI o1's general-purpose reasoning enhancements carry over to a specialised medical field, ophthalmology. Using 6,990 multiple-choice questions from the MedMCQA benchmark, it compares o1 with GPT-4o, GPT-4, GPT-3.5, Llama-3-8b, and Gemini 1.5 Pro under a standardised prompt and greedy decoding. o1 achieves the highest answer accuracy (0.88 ± 0.33) and macro-F1 (0.70), significantly above every comparator. But when reasoning is scored by five text-generation metrics that compare each model's explanation with the dataset's reference explanations, o1 ranks third behind GPT-4o and GPT-4 on the weighted aggregate. The authors take this accuracy–reasoning gap as evidence that o1's enhancements may not transfer to ophthalmology and that domain-specific refinement is still needed.

What carries the argument

The argument runs through the 6,990-item ophthalmology subset of MedMCQA and a five-metric text-generation battery. Each model receives the same zero-shot prompt ('assistant specializing in ophthalmology'), with temperature set to 0, and its explanation is scored against the MedMCQA ground-truth explanation by ROUGE-L, BERTScore, BARTScore, AlignScore, and METEOR; per-metric scores are normalised to 0–1 and averaged into a weighted aggregate. The accuracy results come from the same dataset's answer labels, so one benchmark carries both halves of the dissociation. Sensitivity analyses subset the questions by reference-explanation length, by five largest ophthalmic topic groups, by items all models answered correctly, and by a 100-item set with complete explanations, with a 40-item human qualitative review as a check on what the automatic metrics miss.

What would settle it

Rescore the same 6,990 explanations with output length normalised, or have ophthalmologist raters judge reasoning quality blind to model identity on a larger sample than 40 items. If o1 then ties or beats GPT-4o and GPT-4, the claim that its reasoning lags in ophthalmology is not general; if it still ranks third, the claim is confirmed.

Watch

Extended reading notes

Core claim

The central finding is a dissociation between answering correctly and explaining like the reference. o1 answers 88% of the 6,990 MCQs correctly, the best of the six models, and its macro-F1 of 0.70 also leads. On the weighted normalised aggregate of ROUGE-L, BERTScore, BARTScore, AlignScore, and METEOR, however, o1's score (0.72) is below GPT-4o (0.83) and GPT-4 (0.74). The deficit is driven by ROUGE-L, BERTScore, and AlignScore, where o1 lags; o1 leads on BARTScore and ties with GPT-4o on METEOR. Subgroup analyses show o1 first in Lens and Glaucoma, second to GPT-4o in three other topic groups, and stronger on items whose reference explanations exceed 100 words.

Load-bearing premise

The conclusion that o1's reasoning lags GPT-4o and GPT-4 rests on the assumption that the five explanation-similarity metrics are valid proxies for reasoning quality; if those metrics mostly punish o1's longer, differently styled outputs, the reasoning ranking loses its force even though the accuracy result stands.

Editorial extensions

If this is right

  • If o1's accuracy lead is what matters for exam-style knowledge retrieval, o1 is currently the strongest of the six evaluated models on this ophthalmology benchmark.
  • If the reasoning scores are trusted, o1's chain-of-thought design does not automatically beat earlier GPT models on ophthalmology explanations, so general reasoning claims should not be extrapolated to specialty medicine.
  • Domain-specific performance varies by subtopic: o1 is best in Lens and Glaucoma but trails GPT-4o in Corneal and external diseases, Vitreous and retina, and Oculoplastic and orbital diseases.
  • Because o1's relative reasoning scores improve when reference explanations are longer, evaluations on short-reference items may understate o1's reasoning quality.
  • Benchmark accuracy on MCQs does not establish clinical utility; the authors frame the result as a first step toward structured, domain-specific validation datasets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A length-controlled rescoring of the same 6,990 outputs, truncating or summarising o1's explanations to the reference length, would show whether the o1 deficit on ROUGE-L, BERTScore, and AlignScore is mostly a verbosity artifact; this is a direct test the paper did not run.
  • The 'reasoning' ranking is really a ranking of similarity to MedMCQA's reference explanation style, so a clinician reading the same outputs could legitimately rank o1's step-by-step answers above the aggregate score without contradicting the reported numbers.
  • The same evaluation protocol could be applied to MedMCQA's other specialty subsets to test whether the accuracy-reasoning dissociation is specific to ophthalmology or a general o1 pattern.
  • The qualitative examples of confident, structured errors, such as confusing two dry-eye tests, suggest that even a top-accuracy model needs independent verification before its explanations are used in clinical education or triage.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This manuscript evaluates OpenAI's o1 against five other large language models (GPT-4o, GPT-4, GPT-3.5, Llama-3-8b, Gemini 1.5 Pro) on 6,990 ophthalmology multiple-choice questions drawn from the MedMCQA dataset. The authors measure performance using accuracy and macro-F1, and reasoning using five text-generation metrics (ROUGE-L, BERTScore, BARTScore, AlignScore, METEOR) plus a normalized aggregate score. They report that o1 achieves the highest accuracy (0.88 ± 0.33) and macro-F1 (0.70), but ranks third behind GPT-4o and GPT-4 on the aggregate reasoning score. Subgroup analyses examine performance by ophthalmic subtopic and ground-truth explanation length, and a qualitative human review of 40 items is also included. The paper concludes that o1's reasoning enhancements may not extend equivalently to ophthalmology, underscoring the need for domain-specific LLMs.

Significance. The accuracy benchmark is a useful contribution: it is, to the authors' claim, the largest ophthalmology MCQ evaluation to date, uses a public dataset, covers six models, and applies standardized prompting. The finding that o1 outperforms other models in answer accuracy is straightforward and clearly reported. If the reasoning conclusion were well supported, the paper would be important for guiding decisions about whether domain-specific medical LLMs are needed. However, the reasoning conclusion rests on text-generation metrics that are not validated as measures of reasoning quality, and the paper itself acknowledges their limitations. The statistical analysis of those metrics also uses an inappropriate test. The qualitative review is anecdotal and does not resolve the inconsistency. The paper's significance is therefore mainly in the accuracy benchmarking and in raising the question of how to evaluate reasoning, not in establishing that o1's reasoning lags in ophthalmology.

major comments (4)
  1. [Methods: Evaluation Metrics; Discussion] The five text-generation metrics (ROUGE-L, BERTScore, BARTScore, AlignScore, METEOR) are similarity measures between model outputs and the MedMCQA reference explanations, not validated measures of reasoning quality in ophthalmology. The Discussion concedes that these metrics are general, not medical-specific, and are sensitive to length mismatches, and that o1 tends to produce longer outputs due to chain-of-thought reasoning. The sensitivity analyses in Table 3 and Supplementary Tables 4 and 5 stratify by ground-truth explanation length or by correct-answer status, but they do not control for model output length, so they do not address the length confound. Consequently, the central claim that o1's reasoning lags behind GPT-4o and GPT-4 is not supported; the metrics support at most a claim about lexical and semantic similarity to the reference explanations, not about reasoning quality.
  2. [Methods: Statistical Analysis] For the text-generation metrics, all models are evaluated on the same 6,990 items, so the observations are paired by question. The paper states that the two-tailed Wilcoxon rank-sum test was used; the rank-sum test (Mann-Whitney U) assumes independent samples and is therefore inappropriate for paired data. The correct test would be the Wilcoxon signed-rank test or a paired permutation test. This error affects the validity of every reported p-value for the text-generation metrics in Table 2, Table 3, and the supplementary tables, and the authors should re-run these analyses with an appropriate paired test.
  3. [Methods: Evaluation Metrics; Table 2 footnote] The 'weighted normalised aggregate score' is described as weighted, but it is computed as an unweighted mean of min-max normalized scores across the five metrics, with the normalization performed across the six models in this study. The term 'weighted' is misleading. More importantly, because the min-max normalization is relative to the specific set of models evaluated, the aggregate score and the resulting third-place ranking for o1 are not absolute properties of the model; they would change if the model set changed. The equal weighting is also an arbitrary choice, and the ranking is heavily influenced by ROUGE-L, BERTScore, and AlignScore, which are the metrics most susceptible to length differences. The authors should either remove the claim of weighting, justify the equal weighting, or report a sensitivity analysis over alternative aggregation schemes.
  4. [Methods: Subset with Human Expert Qualitative Review; Results: Human Expert Review] The qualitative review is based on only 40 randomly selected items, with no structured rating instrument, no inter-rater reliability statistic, and no quantitative summary of the evaluators' judgments. As presented, it consists of selected examples and is anecdotal rather than a systematic evaluation. The Discussion's characterization of o1's responses as 'more detailed and methodical' is in tension with the text-metric ranking that places o1 third, and this inconsistency is not resolved. The qualitative review should either be formalized (e.g., with a scoring rubric and agreement measures) or explicitly labeled as illustrative, and it should not be used to support the conclusion that o1's reasoning is superior or inferior without further evidence.
minor comments (4)
  1. [Abstract and Throughout] The capitalization of the model name 'o1' is inconsistent (for example, 'O1' appears in the abstract's Meaning section and elsewhere); please standardize to 'o1' throughout.
  2. [Methods: Accuracy definition] The definition of accuracy contains a typo: 'proportion of correct predictions made out of the total predictions mad' should read '... total predictions made.'
  3. [Methods: Prompt Engineering and hyperparameters] The sentence on greedy decoding says 'temperature set to 0'; please clarify whether this was temperature = 0 or temperature = 0.0 and note whether the OpenAI o1 API allows exact control of this parameter, since o1's internal reasoning settings may differ from standard chat models.
  4. [Discussion] Reference 29 (Xie et al.) is cited for o1's superior performance on ROUGE-1, but the current study reports ROUGE-L; please clarify which metric the cited study used and avoid ambiguity.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the evaluation uses external benchmark data and externally defined metrics, with no fitted parameters or definitional reductions; acknowledged metric-validity limitations are correctness concerns, not circularity.

full rationale

The paper's derivation chain is: take 6,990 ophthalmology MCQs from the external MedMCQA dataset; query six LLMs with a standardized prompt; compute accuracy and macro-F1 from correct answers; compute five standard text-generation metrics (ROUGE-L, BERTScore, BARTScore, AlignScore, METEOR) comparing model explanations to MedMCQA ground-truth explanations; normalize and average those scores into an aggregate; then rank the models. None of these steps defines a quantity in terms of the conclusion, and no parameter is fitted to the data and then renamed as a prediction. The aggregate score is described as 'weighted' but the table footnote states it is the mean of normalized scores; that is a labeling or calculation inconsistency, not a circular reduction. The central interpretive claim that o1's reasoning lags behind GPT-4o and GPT-4 is explicitly conditional on text-generation metrics being reasonable proxies for reasoning. The authors themselves note in the Discussion that these metrics are 'general and not specific to medical context' and are 'sensitive to length mismatches,' which is a construct-validity limitation rather than a case of the result being equivalent to its inputs by construction. The paper's self-citations (e.g., references 25 and 41) are used only to support the need for medical-specific metrics or to note that text-generation metrics alone may not fully capture reasoning; these citations are not load-bearing for the measured rankings. No uniqueness theorem, ansatz, or fitted premise from prior author work is imported to force the conclusion. The qualitative human review is an independent, external check and does not enter the quantitative metric definitions. Therefore, the paper is self-contained against external benchmarks, and no specific circular step can be exhibited.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central results are empirical measurements, so the ledger is light. One hand-chosen aggregation scheme and several domain assumptions about metric validity and benchmark quality carry the interpretive weight of the paper.

free parameters (1)
  • Equal weights in the aggregate reasoning score = 1/5 per metric after min-max normalization across models
    The headline ranking 'o1 third in reasoning' depends on this hand-chosen aggregation of ROUGE-L, BERTScore, BARTScore, AlignScore, and METEOR; different weights or normalization could change the ranking, and the paper does not test this sensitivity.
assumptions (4)
  • domain assumption Text-generation metrics are valid proxies for reasoning quality
    The paper equates 'reasoning abilities' with five lexical and semantic similarity scores against MedMCQA reference explanations, while later acknowledging the metrics are general, not medical-specific, and length-sensitive (Methods, 'Evaluation Metrics'; Discussion limitations).
  • domain assumption MedMCQA ground-truth explanations are suitable reference reasoning
    The reasoning metrics compare model outputs to these explanations; the authors note some explanations are suboptimal (listing options, restating answers, or containing keywords) and only partially address this with a 100-item sensitivity analysis (Methods, 'Subgroup Evaluations'; Supplementary Table 2).
  • domain assumption Accuracy on the MedMCQA ophthalmology subset reflects ophthalmology knowledge
    The paper's conclusion about o1's performance 'in ophthalmology' treats MCQ accuracy as a meaningful domain-knowledge measure, a standard but unverified assumption for this benchmark.
  • domain assumption Greedy decoding with temperature 0 is applied equivalently across all models
    The authors state temperature 0 was used across models; if any API, such as o1, does not honor this setting, model outputs may not be strictly comparable (Methods, 'Prompt Engineering and hyperparameters').

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can OpenAI o1 Reason Well in Ophthalmology? A 6,990-Question Head-to-Head Evaluation Study." pith.science (2026). https://pith.science/paper/XEGRWJMY

@misc{pith2026250113949,
  author       = {Pith},
  title        = {Pith review of: Can OpenAI o1 Reason Well in Ophthalmology? A 6,990-Question Head-to-Head Evaluation Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XEGRWJMY}},
  note         = {Machine review of arXiv:2501.13949}
}
read the original abstract

Question: What is the performance and reasoning ability of OpenAI o1 compared to other large language models in addressing ophthalmology-specific questions? Findings: This study evaluated OpenAI o1 and five LLMs using 6,990 ophthalmological questions from MedMCQA. O1 achieved the highest accuracy (0.88) and macro-F1 score but ranked third in reasoning capabilities based on text-generation metrics. Across subtopics, o1 ranked first in ``Lens'' and ``Glaucoma'' but second to GPT-4o in ``Corneal and External Diseases'', ``Vitreous and Retina'' and ``Oculoplastic and Orbital Diseases''. Subgroup analyses showed o1 performed better on queries with longer ground truth explanations. Meaning: O1's reasoning enhancements may not fully extend to ophthalmology, underscoring the need for domain-specific refinements to optimize performance in specialized fields like ophthalmology.

Figures

Figures reproduced from arXiv: 2501.13949 by the authors.

Figure 1
Figure 1. Standardized Prompt Format Used for Each MCQ Item [PITH_FULL_IMAGE:figures/full_fig_p024_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 23 canonical work pages

  1. [1]

    Attention is All you Need

    Vaswani A, Shazeer N, Parmar N, et al. Attention is All you Need. In: Guyon I, Luxburg UV, Bengio S, et al., eds. Advances in Neural Information Processing Systems. Vol 30. Curran Associates, Inc.; 2017. https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053 c1c4a845aa-Paper.pdf

    work page 2017

  2. [2]

    ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations

    Dave T, Athaluri SA, Singh S. ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations. Frontiers in Artificial Intelligence. 2023;6. doi:10.3389/frai.2023.1169595

    arXiv 2023

  3. [3]

    Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models

    Kung TH, Cheatham M, Medenilla A, et al. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digital Health. 2023;2(2):1-12. doi:10.1371/journal.pdig.0000198

    doi:10.1371/journal.pdig.0000198 2023

  4. [4]

    Evaluating the Performance of ChatGPT in Ophthalmology: An Analysis of Its Successes and Shortcomings

    Antaki F, Touma S, Milad D, El-Khoury J, Duval R. Evaluating the Performance of ChatGPT in Ophthalmology: An Analysis of Its Successes and Shortcomings. Ophthalmology Science. 2023;3(4). doi:10.1016/j.xops.2023.100324

    arXiv 2023

  5. [5]

    Benchmarking large language models’ performances for myopia care: a comparative analysis of ChatGPT-3.5, ChatGPT- 4.0, and Google Bard

    Lim ZW, Pushpanathan K, Yew SME, et al. Benchmarking large language models’ performances for myopia care: a comparative analysis of ChatGPT-3.5, ChatGPT- 4.0, and Google Bard. eBioMedicine. 2023;95. doi:10.1016/j.ebiom.2023.104770

    arXiv 2023

  6. [6]

    Large language models and their impact in ophthalmology

    Betzler BK, Chen H, Cheng CY, et al. Large language models and their impact in ophthalmology. The Lancet Digital Health. 2023;5(12):e917-e924. doi:10.1016/S2589-7500(23)00201-7

    work page doi:10.1016/s2589-7500(23)00201-7 2023

  7. [7]

    Evaluation and mitigation of the limitations of large language models in clinical decision-making

    Hager P, Jungmann F, Holland R, et al. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nature Medicine. 2024;30(9):2613-2622. doi:10.1038/s41591-024-03097-1

    doi:10.1038/s41591-024-03097-1 2024

  8. [8]

    Evolution of Future Medical AI Models — From Task- Specific, Disease-Centric to Universal Health

    Ma W, Sheng B, Liu Y, et al. Evolution of Future Medical AI Models — From Task- Specific, Disease-Centric to Universal Health. NEJM AI. 2024;1(8):AIp2400289. doi:10.1056/AIp2400289

    work page doi:10.1056/aip2400289 2024

Show all 44 references
  1. [9]

    Radiology-Llama2: Best-in-Class Large Language Model for Radiology

    Liu Z, Li Y, Shu P, et al. Radiology-Llama2: Best-in-Class Large Language Model for Radiology. Published online 2023. https://arxiv.org/abs/2309.06419

    2023 arXiv

  2. [10]

    Towards a general-purpose foundation model for computational pathology

    Chen RJ, Ding T, Lu MY, et al. Towards a general-purpose foundation model for computational pathology. Nature Medicine. 2024;30(3):850-862. doi:10.1038/s41591-024-02857-3

    2024 doi

  3. [11]

    Embracing Large Language Models for Medical Applications: Opportunities and Challenges

    Karabacak M, Margetis K. Embracing Large Language Models for Medical Applications: Opportunities and Challenges. Cureus. 2023;15(5):e39305. doi:10.7759/cureus.39305

    2023 doi

  4. [12]

    Assessing the usefulness of a large language model to query and summarize unstructured medical notes in intensive care

    Madden MG, McNicholas BA, Laffey JG. Assessing the usefulness of a large language model to query and summarize unstructured medical notes in intensive care. Intensive Care Medicine. 2023;49(8):1018-1020. doi:10.1007/s00134-023- 07128-2

    2023 doi

  5. [13]

    Utilizing Large Language Models to Simplify Radiology Reports: a comparative analysis of ChatGPT3.5, ChatGPT4.0, Google Bard, and Microsoft Bing

    Doshi R, Amin K, Khosla P, Bajaj S, Chheang S, Forman HP. Utilizing Large Language Models to Simplify Radiology Reports: a comparative analysis of ChatGPT3.5, ChatGPT4.0, Google Bard, and Microsoft Bing. medRxiv. Published online 2023. doi:10.1101/2023.06.04.23290786

    2023 doi

  6. [14]

    The effect of using a large language model to respond to patient messages

    Chen S, Guevara M, Moningi S, et al. The effect of using a large language model to respond to patient messages. The Lancet Digital Health. 2024;6(6):e379-e381. doi:10.1016/S2589-7500(24)00060-8

    2024 doi

  7. [15]

    P717 Evaluating the performance of Large Language Models in responding to patients’ health queries: A comparative analysis with medical experts

    Yan Z, Lu S, Xu D, et al. P717 Evaluating the performance of Large Language Models in responding to patients’ health queries: A comparative analysis with medical experts. Journal of Crohn’s and Colitis. 2024;18(Supplement_1):i1348- i1349. doi:10.1093/ecco-jcc/jjad212.0847

    2024 doi

  8. [16]

    Integrated image-based deep learning and language models for primary diabetes care

    Li J, Guan Z, Wang J, et al. Integrated image-based deep learning and language models for primary diabetes care. Nature Medicine. Published online July 19, 2024. doi:10.1038/s41591-024-03139-8

    2024 doi

  9. [17]

    Large language models in medicine

    Thirunavukarasu AJ, Ting DSJ, Elangovan K, Gutierrez L, Tan TF, Ting DSW. Large language models in medicine. Nature Medicine. 2023;29(8):1930-1940. doi:10.1038/s41591-023-02448-8

    2023 doi

  10. [18]

    Using Large Language Models to Generate Educational Materials on Childhood Glaucoma

    Dihan Q, Chauhan MZ, Eleiwa TK, et al. Using Large Language Models to Generate Educational Materials on Childhood Glaucoma. American Journal of Ophthalmology. 2024;265:28-38. doi:10.1016/j.ajo.2024.04.004

    2024 doi

  11. [19]

    Exploring AI-chatbots’ capability to suggest surgical planning in ophthalmology: ChatGPT versus Google Gemini analysis of retinal detachment cases

    Carlà MM, Gambini G, Baldascino A, et al. Exploring AI-chatbots’ capability to suggest surgical planning in ophthalmology: ChatGPT versus Google Gemini analysis of retinal detachment cases. Br J Ophthalmol. 2024;108(10):1457. doi:10.1136/bjo-2023-325143

    2024 doi

  12. [20]

    Optimising vitrectomy operation note coding with machine learning

    Lee YM, Bacchi S, Sia D, Casson RJ, Chan W. Optimising vitrectomy operation note coding with machine learning. Clinical & Experimental Ophthalmology. 2023;51(6):577-584. doi:https://doi.org/10.1111/ceo.14257

    2023 doi

  13. [21]

    Predicting Glaucoma Before Onset Using a Large Language Model Chatbot

    Huang X, Raja H, Madadi Y, et al. Predicting Glaucoma Before Onset Using a Large Language Model Chatbot. American Journal of Ophthalmology. 2024;266:289-299. doi:10.1016/j.ajo.2024.05.022

    2024 doi

  14. [22]

    Detecting hallucinations in large language models using semantic entropy

    Farquhar S, Kossen J, Kuhn L, Gal Y. Detecting hallucinations in large language models using semantic entropy. Nature. 2024;630(8017):625-630. doi:10.1038/s41586-024-07421-0

    2024 doi

  15. [23]

    Survey of Hallucination in Natural Language Generation

    Ji Z, Lee N, Frieske R, et al. Survey of Hallucination in Natural Language Generation. ACM Computing Surveys. 2023;55(12):1-38. doi:10.1145/3571730

    2023 doi

  16. [24]

    Popular large language model chatbots’ accuracy, comprehensiveness, and self-awareness in answering ocular symptom queries

    Pushpanathan K, Lim ZW, Yew SME, et al. Popular large language model chatbots’ accuracy, comprehensiveness, and self-awareness in answering ocular symptom queries. iScience. 2023;26(11):108163. doi:https://doi.org/10.1016/j.isci.2023.108163

    2023

  17. [25]

    Language Enhanced Model for Eye (LEME): An Open- Source Ophthalmology-Specific Large Language Model

    Gilson A, Ai X, Xie Q, et al. Language Enhanced Model for Eye (LEME): An Open- Source Ophthalmology-Specific Large Language Model. Published online October 1, 2024. doi:10.48550/arXiv.2410.03740

    2024 doi

  18. [26]

    September 12, 2024

    Introducing OpenAI o1. September 12, 2024. Accessed November 25, 2024. https://openai.com/o1/

    2024

  19. [27]

    Accessed November 25, 2024

    OpenAI o1 System Card. Accessed November 25, 2024. https://openai.com/index/openai-o1-system-card/

    2024

  20. [28]

    Evaluation of OpenAI o1: Opportunities and Challenges of AGI

    Zhong T, Liu Z, Pan Y, et al. Evaluation of OpenAI o1: Opportunities and Challenges of AGI. Published online September 27, 2024. doi:10.48550/arXiv.2409.18486

    2024 doi

  21. [29]

    A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor? arXiv preprint arXiv:240915277

    Xie Y, Wu J, Tu H, et al. A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor? arXiv preprint arXiv:240915277. Published online 2024

    2024

  22. [30]

    From Medprompt to o1: Exploration of Run-Time Strategies for Medical Challenge Problems and Beyond

    Nori H, Usuyama N, King N, et al. From Medprompt to o1: Exploration of Run-Time Strategies for Medical Challenge Problems and Beyond. Published online 2024. https://arxiv.org/abs/2411.03590

    2024 arXiv

  23. [31]

    Towards Next-Generation Medical Agent: How o1 is Reshaping Decision-Making in Medical Scenarios

    Xu S, Zhou Y, Liu Z, et al. Towards Next-Generation Medical Agent: How o1 is Reshaping Decision-Making in Medical Scenarios. Published online November 16,

  24. [32]

    Medmcqa: A large-scale multi-subject multi- choice dataset for medical domain question answering

    Pal A, Umapathi LK, Sankarasubbu M. Medmcqa: A large-scale multi-subject multi- choice dataset for medical domain question answering. In: Conference on Health, Inference, and Learning. PMLR; 2022:248-260

    2022

  25. [33]

    How Does ChatGPT Perform on the United States Medical Licensing Examination? The Implications of Large Language Models for Medical Education and Knowledge Assessment

    Gilson A, Safranek CW, Huang T, et al. How Does ChatGPT Perform on the United States Medical Licensing Examination? The Implications of Large Language Models for Medical Education and Knowledge Assessment. JMIR Med Educ. 2023;9:e45312. doi:10.2196/45312

    2023 doi

  26. [34]

    On the Relation between Sensitivity and Accuracy in In-context Learning

    Chen Y, Zhao C, Yu Z, McKeown K, He H. On the Relation between Sensitivity and Accuracy in In-context Learning. Published online 2024. https://arxiv.org/abs/2209.07661

    2024 arXiv

  27. [35]

    A Toolbox for Surfacing Health Equity Harms and Biases in Large Language Models

    Pfohl SR, Cole-Lewis H, Sayres R, et al. A Toolbox for Surfacing Health Equity Harms and Biases in Large Language Models. Nat Med. 2024;30(12):3590-3600. doi:10.1038/s41591-024-03258-2

    2024 doi

  28. [36]

    ROUGE: A Package for Automatic Evaluation of Summaries

    Lin CY. ROUGE: A Package for Automatic Evaluation of Summaries. In: Text Summarization Branches Out. Association for Computational Linguistics; 2004:74-

    2004

  29. [37]

    BERTScore: Evaluating Text Generation with BERT

    Zhang T, Kishore V, Wu F, Weinberger KQ, Artzi Y. BERTScore: Evaluating Text Generation with BERT. Published online 2020. https://arxiv.org/abs/1904.09675

    2020 arXiv

  30. [38]

    BARTScore: Evaluating Generated Text as Text Generation

    Yuan W, Neubig G, Liu P. BARTScore: Evaluating Generated Text as Text Generation. Published online 2021. https://arxiv.org/abs/2106.11520

    2021 arXiv

  31. [39]

    Meteor: an automatic metric for MT evaluation with high levels of correlation with human judgments

    Lavie A, Agarwal A. Meteor: an automatic metric for MT evaluation with high levels of correlation with human judgments. In: Proceedings of the Second Workshop on Statistical Machine Translation. StatMT ’07. Association for Computational Linguistics; 2007:228-231

    2007

  32. [40]

    AlignScore: Evaluating Factual Consistency with a Unified Alignment Function

    Zha Y, Yang Y, Li R, Hu Z. AlignScore: Evaluating Factual Consistency with a Unified Alignment Function. Published online 2023. https://arxiv.org/abs/2305.16739

    2023 arXiv

  33. [41]

    A systematic evaluation of large language models for biomedical natural language processing: benchmarks, baselines, and recommendations

    Chen Q, Hu Y, Peng X, et al. A systematic evaluation of large language models for biomedical natural language processing: benchmarks, baselines, and recommendations. Published online 2024. https://arxiv.org/abs/2305.16326

    2024 arXiv

  34. [42]

    OphGLM: An ophthalmology large language-and- vision assistant

    Deng Z, Gao W, Chen C, et al. OphGLM: An ophthalmology large language-and- vision assistant. Artificial Intelligence in Medicine. 2024;157:103001. doi:https://doi.org/10.1016/j.artmed.2024.103001

    2024

  35. [81]

    https://aclanthology.org/W04-1013

  36. [2024]

    doi:10.48550/arXiv.2411.14461

    doi arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.