Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

OphthBench: A Comprehensive Benchmark for Evaluating Large Language Models in Chinese Ophthalmology

T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 591-question benchmark built around the Chinese eye-care workflow shows that today's large language models still fall short of clinical readiness.

desk verdict Useful Chinese ophthalmology benchmark, but the headline 'substantial gap' claim is an artifact of the per-task normalization to 90, not a measured fact. read the letter →

arxiv 2502.01243 v1 pith:AR4KE2RR submitted 2025-02-03 cs.CL cs.AI

Chengfeng Zhou , Ji Wang , Juanjuan Qin , Yining Wang , Ling Sun , Weiwei Dai This is my paper · ORCID
classification cs.CLcs.AI
keywords largelanguagemodelsophthalmologyChinesemedicalbenchmarkclinicalworkflowLLM-as-a-judgequestionansweringpromptengineering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

OphthBench is a benchmark for deciding whether large language models can be useful in Chinese ophthalmology, built by splitting a typical clinical workflow into education, triage, diagnosis, treatment, and prognosis. It contains 591 questions across nine tasks, mixing single-choice, multiple-choice, and open-ended formats, and it evaluates 39 popular LLMs under two prompting regimes. The paper's central finding is that even the best models reach only the mid-80s on a normalized scale, with most scoring near 70 percent, leaving a substantial gap between current capabilities and practical clinical requirements. A sympathetic reader would take this as evidence that the field needs both workflow-aligned evaluation and further model development before deployment.

What carries the argument

The central object is OphthBench itself: a five-scenario, nine-task question set that maps onto a typical ophthalmic clinical workflow and includes single-choice, multiple-choice, and open-ended questions. The load-bearing scoring machinery is a normalized, rule-calibrated multi-dimensional LLM-as-a-judge protocol, in which the CompassJudger-1-7B model scores open-ended answers against ophthalmologist-reviewed reference responses on seven dimensions, task scores are normalized by the best model's score, and scenario totals are computed with harmonic means. Two prompting regimes, a common prompt and an advanced prompt generated with the CO-STAR framework, are used to reduce prompt-sensitivity bias.

What would settle it

Select a random sample of open-ended responses from the triage, diagnosis, and treatment tasks, have three independent ophthalmologists score them without seeing the judge's scores, and measure agreement; if agreement is low, the benchmark's open-ended scenario scores are not trustworthy.

Watch

Extended reading notes

Core claim

OphthBench's central claim is that LLM performance in Chinese ophthalmology should be measured against a workflow-shaped yardstick rather than isolated exam questions, and that when such a yardstick is built, 39 popular models show a substantial gap between their scores and practical requirements. The paper constructs this yardstick from 591 questions organized into five scenarios and nine tasks, with answers reviewed by three experienced ophthalmologists, and evaluates models under both common and optimized prompts. The strongest observed models reach the mid-80s on the normalized scale while many models sit near or below 70 percent, which the paper interprets as evidence that current LLMs are not yet ready for real-world ophthalmic use.

Load-bearing premise

The load-bearing premise is that the automated judge's scores on open-ended answers agree with what expert ophthalmologists would consider clinically correct, but the paper reports no human validation or inter-rater agreement to demonstrate this.

Editorial extensions

If this is right

  • If OphthBench reflects real clinical utility, current LLMs, including the best commercial systems, are not yet dependable enough for unsupervised use in Chinese eye care because the observed ceiling sits near the mid-80s rather than near 100.
  • Medical-specific LLMs trained in Chinese did not outperform general-purpose models on this benchmark, suggesting that domain post-training alone does not translate into ophthalmic competence.
  • Prompt engineering can raise most models' scores substantially, so part of the capability gap is accessible through better instruction design rather than only through better models.
  • The prognosis scenario proved easiest and the education scenario hardest, giving developers a task-level map of where to focus future work.
  • The benchmark's workflow structure provides a concrete way to compare models on triage, diagnosis, and treatment skills rather than on general medical knowledge alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Our inference: the same five-scenario template could be reused for other Chinese clinical specialties, producing comparable cross-specialty readiness scores.
  • Our inference: because the benchmark contains only text, adding fundus photography and OCT interpretation would probably lower diagnostic scores further and better reflect real eye-care work.
  • Our inference: the judge-model scores could be converted into a training signal by collecting a small human-rated sample and using those preferences to fine-tune either the judge or the clinical model, something the paper does not attempt.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces OphthBench, a benchmark of 591 questions for evaluating LLMs in Chinese ophthalmology, organized into five clinical workflow scenarios (Education, Triage, Diagnosis, Treatment, Prognosis) and nine tasks, with single-choice, multiple-choice, and open-ended questions. The authors evaluate 39 LLMs under two prompting regimes (common and advanced) and report leaderboard scores. The central claim is that the results reveal a substantial gap between current model capabilities and practical clinical requirements in Chinese ophthalmology, with secondary findings that Chinese models outperform non-Chinese models, medical-specific LLMs do not surpass general LLMs, and prompt engineering substantially improves performance.

Significance. If properly validated, OphthBench would be a useful resource: it is one of the few benchmarks explicitly aligned with a clinical workflow in a non-English medical context, it combines three question formats, and the evaluation includes both common and advanced prompting protocols. The authors also include an external MMLU reference column, report a t-test for one subgroup comparison, and describe their model deployment and output-constraint methods, which aids reproducibility. However, the paper's headline conclusion of a "substantial gap" between model capabilities and practical requirements is not currently supported by the reported metrics, because the normalization in Eq. (1) removes absolute score information and no human expert, random-chance, or clinical-threshold baseline is provided. The benchmark construction and the multi-model evaluation are credible contributions, but the measurement validity needs to be demonstrated before the practical-utility conclusion can be accepted.

major comments (4)
  1. [Section 3.3, Eq. (1); Section 4] The normalization in Eq. (1) divides every task score by the best observed model score and multiplies by 90, so the reported leaderboard values are relative to the best model on each task; they contain no information about absolute correctness. The abstract and Section 4 conclude that there is a "substantial gap between current model capabilities and practical requirements," but no clinical threshold, human expert baseline, or random-chance baseline is defined anywhere in the manuscript. The "performance rate of approximately 70%" mentioned in Section 4 is not tied to the tables, and the text does not state whether it is raw accuracy; if it refers to normalized scores it is an artifact of the 90-point ceiling. Please report raw accuracy/F1 per task and scenario, add human expert performance on the same 591 questions and a random-chance baseline, and provide confidence intervals, so that the conclusion about a gap to practical requirements can actually be tested.
  2. [Sections 3.2 and 3.3, OEQ protocol] For OEQs, reference answers were drafted by GPT-4o and then reviewed by ophthalmologists, and responses are scored by CompassJudger-1-7B comparing them to those references on seven dimensions. The manuscript provides no calibration evidence for this judge on this benchmark: no human-scored subset, no inter-rater agreement, and no correlation between judge scores and expert opinion. Because GPT-4o was also used to refine the advanced OEQ prompts, the evaluation loop is self-referential: an LLM helped create the gold responses and an LLM judges the answers. If the judge's dimension preferences diverge from expert judgment, the Triage, Diagnosis, and Treatment scenario scores would be systematically biased. Please add a human-validation sub-study on a stratified random sample of responses and report agreement statistics (e.g., Cohen's kappa or correlation) before using the judge scores as the basis for task rankings.
  3. [Section 4, Tables 3 and 4, t-test] The harmonic-mean aggregation across tasks is not justified and mixes relative per-task normalized scores into the "All scenarios" column; moreover, no confidence intervals or significance tests are reported for leaderboard differences, so adjacent ranks (e.g., ranks 1-3 in Table 3) may not be statistically distinguishable. The t-test for Chinese vs. non-Chinese 6B-9B models gives P=0.096 and P=0.085, which is at best marginal, yet the text states that Chinese models "consistently outperformed" non-Chinese models. Provide bootstrap confidence intervals around the aggregated scores and report the t-test details (e.g., whether it is paired, the sample size, and the effect size); otherwise, these claims are not supported.
  4. [Section 5, Limitations] The stated limitations in Section 5 mention incomplete model coverage and limited dataset size, but do not acknowledge the three most consequential threats to the paper's central claim: the absence of a human expert baseline, the lack of calibration for the LLM-as-judge, and the fact that the normalization in Eq. (1) removes absolute score information. Since the conclusion of a "substantial gap" depends on these measurement properties, the limitations section should be extended accordingly.
minor comments (6)
  1. [Section 4, paragraph on medical LLMs] The text refers to "HuatuoGPT2-o1-7B" as being based on Qwen2.5-7B, but Tables 2-4 list "HuatuoGPT-o1-7B"; the naming is inconsistent and should be corrected.
  2. [Section 4, final paragraph of the first subsection] The sentence ending with "These advancements" is cut off in the preprint and should be completed.
  3. [Figure 2] The statistics panel in Figure 2 is not legible in the preprint; please provide a vector graphic with labeled axes and clearly readable text.
  4. [Tables 3 and 4, MMLU column] The MMLU column is not part of OphthBench; clarify in the caption or text whether these are external reference scores and how they were obtained, and state why they are included in an OphthBench leaderboard.
  5. [Section 3.2 vs. Abstract] The abstract says the benchmark was developed by "three experienced Chinese ophthalmologists," while Section 3.2 describes "three junior ophthalmologists" plus a senior reviewer; please make the count and seniority consistent.
  6. [Figure 5] Figure 5 reports a maximum improvement of 175.0% and a decline of 62.2%, but the text does not explain which tasks or models produced these extremes; please clarify.

Circularity Check

1 steps flagged · score 6.0 of 10

Headline 'gap' is relative by construction: Eq. (1) sets each task's best LLM to 90, and the paper reads the resulting ~70% as an absolute performance rate against practical requirements.

  1. self definitional [Section 3.3, Eq. (1); Section 4, Fig. 4; Conclusion]
    "Before aggregating task performance, the highest score among 39 LLMs for each task (on a scale of 100) is set to 90 and used for score normalization to eliminate the task difficulty bias. ... Score′task = Score task/Score⋆task × 90 ... where n is the number of tasks within a specific clinical scenario and Score⋆task denotes the highest score among the 39 LLMs with advanced reponses for the given task. ... As shown in Fig."

    Under Eq. (1), every Score′task is (raw score / raw best score) × 90, so the strongest LLM always scores 90 on each task and every other number encodes only the distance to that observed maximum. The paper then cites the resulting 'approximately 70%' as a 'performance rate' and uses it to claim a 'substantial gap between current model capabilities and practical requirements.' That gap is not anchored to raw accuracy, human expert performance, chance level, or any clinical threshold; it is by construction 90 − 90×(raw/best), i.e., the gap to the best LLM in the same 39-model pool.

full rationale

The only load-bearing circularity I can quote is the §3.3 Eq. (1) normalization: by construction it sets the top LLM on each task to 90, and the paper subsequently reads the normalized 'approximately 70%' as an absolute performance rate supporting the 'substantial gap' conclusion. No raw accuracy, expert baseline, or clinical threshold is reported, so that conclusion is definitionally relative to the best model rather than measured against practical requirements. I deliberately did not count the OEQ construction loop as circular: GPT-4o drafts reference answers, but the paper states they were 'thoroughly reviewed and refined by ophthalmologists,' and CompassJudger-1-7B is a separate judge model; the absence of human-judge calibration or inter-rater statistics is an external-validity problem, not a circular reduction. There is also no author self-citation chain: the judge model and prompting references are not by the OphthBench authors. The rankings, prompt-sensitivity results, and Chinese-vs-foreign model differences rest on raw comparisons and have independent content, which is why the paper is only partially circular rather than wholly self-referential.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the representativeness of the test set and the validity of the scoring pipeline; neither is established with external baselines or released artifacts.

free parameters (2)
  • Normalization ceiling (90) = 90
    Used in Eq. (1) to rescale each task's best LLM score to 90, which shapes absolute performance claims like the approximately 70% finding.
  • Number of judge dimensions = 7
    The CompassJudger LLM-as-a-judge scores OEQs on 7 dimensions (factuality, user satisfaction, safety, etc.), chosen without reported sensitivity analysis.
assumptions (4)
  • domain assumption The five scenarios (Education, Triage, Diagnosis, Treatment, Prognosis) faithfully represent the Chinese ophthalmic clinical workflow.
    Section 3.1 derives the taxonomy from tertiary eye center workflows; the completeness and representativeness are asserted, not empirically validated.
  • domain assumption The 591 questions and their authoritative answers are clinically accurate and uncontroversial.
    Section 3.2 states three junior ophthalmologists created questions and a senior ophthalmologist reviewed them; no inter-rater reliability or external validation is reported.
  • domain assumption LLM-as-a-judge (CompassJudger-1-7B) scores correlate with human expert judgment for Chinese ophthalmology OEQs.
    Section 3.3 adopts the rule-calibrated LLM-as-judge method without presenting human agreement statistics on this benchmark.
  • domain assumption Reference answers generated by GPT-4o and refined by ophthalmologists are valid ground truth for OEQs.
    Section 3.2 describes the OEQ answer generation; the degree of refinement and any disagreements are not quantified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OphthBench: A Comprehensive Benchmark for Evaluating Large Language Models in Chinese Ophthalmology." pith.science (2026). https://pith.science/paper/AR4KE2RR

@misc{pith2026250201243,
  author       = {Pith},
  title        = {Pith review of: OphthBench: A Comprehensive Benchmark for Evaluating Large Language Models in Chinese Ophthalmology},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AR4KE2RR}},
  note         = {Machine review of arXiv:2502.01243}
}
read the original abstract

Large language models (LLMs) have shown significant promise across various medical applications, with ophthalmology being a notable area of focus. Many ophthalmic tasks have shown substantial improvement through the integration of LLMs. However, before these models can be widely adopted in clinical practice, evaluating their capabilities and identifying their limitations is crucial. To address this research gap and support the real-world application of LLMs, we introduce the OphthBench, a specialized benchmark designed to assess LLM performance within the context of Chinese ophthalmic practices. This benchmark systematically divides a typical ophthalmic clinical workflow into five key scenarios: Education, Triage, Diagnosis, Treatment, and Prognosis. For each scenario, we developed multiple tasks featuring diverse question types, resulting in a comprehensive benchmark comprising 9 tasks and 591 questions. This comprehensive framework allows for a thorough assessment of LLMs' capabilities and provides insights into their practical application in Chinese ophthalmology. Using this benchmark, we conducted extensive experiments and analyzed the results from 39 popular LLMs. Our evaluation highlights the current gap between LLM development and its practical utility in clinical settings, providing a clear direction for future advancements. By bridging this gap, we aim to unlock the potential of LLMs and advance their development in ophthalmology.

Figures

Figures reproduced from arXiv: 2502.01243 by the authors.

Figure 1
Figure 1. An illustration of our proposed benchmark. OphthBench is a specialized benchmark designed to assess LLM capabilities in Chinese ophthalmic practices. It comprises 5 core ophthalmic scenarios, Education, Triage, Diagnosis, Treatment, and Prognosis, covering 9 distinct tasks with single-choice, multiple-choice, and open-ended question formats. This comprehensive structure ensures a thorough evaluation of LLM performan… view at source ↗
Figure 2
Figure 2. The statistics of the OphthBench dataset. With the help of three experienced Chinese ophthalmologists, OphthBench comprises 591 questions spanning 5 core ophthalmic scenarios and evaluates model performance across 9 distinct tasks. 3.3 Evaluation Protocol Numerous studies have highlighted that the sensitivity of large language models to prompts can lead to biased eval￾uations. To mitigate these effects, we report tw… view at source ↗
Figure 3
Figure 3. The prompts utilized for evaluation. Bold fonts indicate content that needs to be filled in. English translations in parentheses are not included during the evaluation. To evaluate the LLMs’ performance, we employ Accuracy (Acc.) for SCQs and F1 score for MCQs. Given that evaluating OEQs presents greater challenges, we adopt a rule-calibrated multi-dimensional point-wise LLM￾as-Judge method [48], which automatically… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: A statistic of scoring rate of 39 for each task. The results are collected with an advanced prompt to illustrate the practice challenge better. 4 Main Results A total of 39 recent LLMs, including both open-sourced and commercial, were selected to evaluate their clinica…
Figure 5
Figure 5. Figure 5: Impact of Prompts. We compared the responses generated by common and advanced prompts, reporting the performance enhancement across various tasks and models. Notably, the advanced prompts resulted in apparent performance improvements (≥ 10%) in most cases, with the hig…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. BEnchmarking LLMs for Ophthalmology (BELO) for Ophthalmological Knowledge and Reasoning

    cs.CL 2025-07 conditional novelty 6.0 of 10

    BELO is a new ophthalmology benchmark of 900 expert-checked multiple-choice questions with reasoning, used to evaluate six LLMs on accuracy and explanation quality.

Reference graph

Works this paper leans on

67 extracted references · 37 canonical work pages · cited by 1 Pith paper

  1. [1]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, et al. Language models are few-shot learners. Adv. Neural Inf. Process. Syst., 33:1877–1901, 2020

    work page 1901

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, et al. Gpt-4 technical report. arXiv, 2023

    work page 2023

  3. [3]

    Large language models encode clinical knowledge

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, et al. Large language models encode clinical knowledge. Nature, 620(7972):172–180, 2023

    work page 2023

  4. [4]

    Empowering biomedical discovery with ai agents

    Shanghua Gao, Ada Fang, Yepeng Huang, Valentina Giunchiglia, Ayush Noori, Jonathan Richard Schwarz, et al. Empowering biomedical discovery with ai agents. Cell, 187(22):6125–6151, 2024

    work page 2024

  5. [5]

    A survey on large language models for critical societal domains: finance, healthcare, and law

    Zhiyu Zoey Chen, Jing Ma, Xinlu Zhang, Nan Hao, An Yan, Armineh Nourbakhsh, et al. A survey on large language models for critical societal domains: finance, healthcare, and law. arXiv preprint arXiv:2405.01769, 2024

    arXiv 2024

  6. [6]

    Integrated image- based deep learning and language models for primary diabetes care

    Jiajia Li, Zhouyu Guan, Jing Wang, Carol Y Cheung, Yingfeng Zheng, Lee-Ling Lim, et al. Integrated image- based deep learning and language models for primary diabetes care. Nat. Med., pages 1–11, 2024

    work page 2024

  7. [7]

    Evaluating large language models on medical evidence summarization

    Liyan Tang, Zhaoyi Sun, Betina Idnay, Jordan G Nestor, Ali Soroush, Pierre A Elias, et al. Evaluating large language models on medical evidence summarization. NPJ Digit. Med., 6(1):158, 2023

    work page 2023

  8. [8]

    Adapted large language models can outperform medical experts in clinical text summarization

    Dave Van Veen, Cara Van Uden, Louis Blankemeier, Jean-Benoit Delbrouck, Asad Aali, Christian Bluethgen, et al. Adapted large language models can outperform medical experts in clinical text summarization. Nat. Med., 30(4):1134–1142, 2024

    work page 2024

Show all 67 references
  1. [9]

    A strategy for cost-effective large language model use at health system-scale

    Eyal Klang, Donald Apakama, Ethan E Abbott, Akhil Vaid, Joshua Lampert, Ankit Sakhuja, et al. A strategy for cost-effective large language model use at health system-scale. NPJ Digit. Med., 7(1):320, 2024

    2024

  2. [10]

    Matching patients to clinical trials with large language models

    Qiao Jin, Zifeng Wang, Charalampos S Floudas, Fangyuan Chen, Changlin Gong, Dara Bracken-Clarke, et al. Matching patients to clinical trials with large language models. Nat. Commun., 15(1):9074, 2024

    2024

  3. [11]

    Scaling clinical trial matching using large language models: a case study in oncology

    Cliff Wong, Sheng Zhang, Yu Gu, Christine Moung, Jacob Abel, Naoto Usuyama, et al. Scaling clinical trial matching using large language models: a case study in oncology. In Proc. Mach. Learn. Healthc. Conf., volume 219, pages 846–862, 2023

    2023

  4. [12]

    Chatgpt and other large language models are double-edged swords, 2023

    Yiqiu Shen, Laura Heacock, Jonathan Elias, Keith D Hentel, Beatriu Reig, George Shih, et al. Chatgpt and other large language models are double-edged swords, 2023

    2023

  5. [13]

    Ethics of large language models in medicine and medical research

    Hanzhou Li, John T Moon, Saptarshi Purkayastha, Leo Anthony Celi, Hari Trivedi, and Judy W Gichoya. Ethics of large language models in medicine and medical research. Lancet Digit. Health, 5(6):e333–e335, 2023

    2023

  6. [14]

    The ethics of chatgpt in medicine and healthcare: a systematic review on large language models (llms)

    Joschka Haltaufderheide and Robert Ranisch. The ethics of chatgpt in medicine and healthcare: a systematic review on large language models (llms). NPJ Digit. Med., 7(1):183, 2024

    2024

  7. [15]

    Clinical large language models with misplaced focus

    Zining Luo, Haowei Ma, Zhiwu Li, Yuquan Chen, Yixin Sun, Aimin Hu, et al. Clinical large language models with misplaced focus. Nat. Mach. Intell., pages 1–2, 2024

    2024

  8. [16]

    Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conf. Health Infer. Learn. , pages 248–260, 2022

    2022

  9. [17]

    Medcalc- bench: Evaluating large language models for medical calculations

    Nikhil Khandekar, Qiao Jin, Guangzhi Xiong, Soren Dunn, Serina S Applebaum, Zain Anwar, et al. Medcalc- bench: Evaluating large language models for medical calculations. arXiv preprint arXiv:2406.12036, 2024

    2024 arXiv

  10. [18]

    Longhealth: A question answering benchmark with long clinical documents

    Lisa Adams, Felix Busch, Tianyu Han, Jean-Baptiste Excoffier, Matthieu Ortala, Alexander Löser, et al. Longhealth: A question answering benchmark with long clinical documents. arXiv preprint arXiv:2401.14490, 2024

    2024 arXiv

  11. [19]

    Cblue: A chinese biomedical language understanding evaluation benchmark.arXiv preprint arXiv:2106.08087, 2021

    Ningyu Zhang, Mosha Chen, Zhen Bi, Xiaozhuan Liang, Lei Li, Xin Shang, Kangping Yin, Chuanqi Tan, Jian Xu, Fei Huang, et al. Cblue: A chinese biomedical language understanding evaluation benchmark.arXiv preprint arXiv:2106.08087, 2021

    2021 arXiv

  12. [20]

    case of the month

    Pokhraj P Suthar, Avin Kounsal, Lavanya Chhetri, Divya Saini, and Sumeet G Dua. Artificial intelligence (ai) in radiology: a deep dive into chatgpt 4.0’s accuracy with the american journal of neuroradiology’s (ajnr)" case of the month". Cureus, 15(8), 2023

    2023

  13. [21]

    Accuracy and reliability of chatbot responses to physician questions

    Rachel S Goodman, J Randall Patrinely, Cosby A Stone, Eli Zimmerman, Rebecca R Donald, Sam S Chang, et al. Accuracy and reliability of chatbot responses to physician questions. JAMA Netw. Open, 6(10):e2336483– e2336483, 2023. 12 arXiv Template A PREPRINT

    2023

  14. [22]

    Capabilities of gpt-4 in ophthalmology: an analysis of model entropy and progress towards human-level medical question answering

    Fares Antaki, Daniel Milad, Mark A Chia, Charles-Édouard Giguère, Samir Touma, Jonathan El-Khoury, Pearse A Keane, and Renaud Duval. Capabilities of gpt-4 in ophthalmology: an analysis of model entropy and progress towards human-level medical question answering. British Journa...

    2023

  15. [23]

    Comparison of ophthalmologist and large language model chatbot responses to online patient eye care questions

    Isaac A Bernstein, Youchen Zhang, Devendra Govil, Iyad Majid, Robert T Chang, Yang Sun, Ann Shue, Jonathan C Chou, Emily Schehlein, Karen L Christopher, et al. Comparison of ophthalmologist and large language model chatbot responses to online patient eye care questions. JAMA N...

    2023

  16. [24]

    Eval- uation and mitigation of the limitations of large language models in clinical decision-making

    Paul Hager, Friederike Jungmann, Robbie Holland, Kunal Bhagat, Inga Hubrecht, Manuel Knauer, et al. Eval- uation and mitigation of the limitations of large language models in clinical decision-making. Nat. Med. , 30(9):2613–2622, 2024

    2024

  17. [25]

    Assessing the utility of chatgpt throughout the entire clinical workflow: development and usability study.J

    Arya Rao, Michael Pang, John Kim, Meghana Kamineni, Winston Lie, Anoop K Prasad, et al. Assessing the utility of chatgpt throughout the entire clinical workflow: development and usability study.J. Med. Internet Res., 25:e48659, 2023

    2023

  18. [26]

    Benchmarking large language models’ performances for myopia care: a comparative analysis of chatgpt- 3.5, chatgpt-4.0, and google bard

    Zhi Wei Lim, Krithi Pushpanathan, Samantha Min Er Yew, Yien Lai, Chen-Hsin Sun, Janice Sing Harn Lam, et al. Benchmarking large language models’ performances for myopia care: a comparative analysis of chatgpt- 3.5, chatgpt-4.0, and google bard. EBioMed., 95, 2023

    2023

  19. [27]

    Sarah W Li, Matthew W Kemp, Susan JS Logan, Pooja Sharma Dimri, Navkaran Singh, Citra NZ Mattar, Pradip Dashraath, Harshaana Ramlal, Aniza P Mahyuddin, Suren Kanayan, et al. Chatgpt outscored human candidates in a virtual objective structured clinical examination in obstetrics...

    2023

  20. [28]

    Performance of large language models on medical oncology examination questions

    Jack B Longwell, Ian Hirsch, Fernando Binder, Galileo Arturo Gonzalez Conchas, Daniel Mau, Raymond Jang, et al. Performance of large language models on medical oncology examination questions. JAMA Netw. Open, 7(6):e2417641–e2417641, 2024

    2024

  21. [29]

    Assessment of a large lan- guage model’s responses to questions and cases about glaucoma and retina management

    Andy S Huang, Kyle Hirabayashi, Laura Barna, Deep Parikh, and Louis R Pasquale. Assessment of a large lan- guage model’s responses to questions and cases about glaucoma and retina management. JAMA ophthalmology, 142(4):371–375, 2024

    2024

  22. [30]

    What disease does this patient have? A large-scale open domain question answering dataset from medical exams

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? A large-scale open domain question answering dataset from medical exams. CoRR, abs/2009.13081, 2020

    2009 arXiv

  23. [31]

    Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Gerardo Flores, George H. Chen, Tom J. Pollard, Joyce C. Ho, and Tristan Naumann, editors, Conference on Health, ...

    2022

  24. [32]

    Benchmarking large language models on cmexam-a comprehensive chinese medical exam dataset

    Junling Liu, Peilin Zhou, Yining Hua, Dading Chong, Zhongyu Tian, Andrew Liu, Helin Wang, Chenyu You, Zhenhua Guo, Lei Zhu, et al. Benchmarking large language models on cmexam-a comprehensive chinese medical exam dataset. Advances in Neural Information Processing Systems, 36, 2024

    2024

  25. [33]

    Medbench: A large- scale chinese benchmark for evaluating medical large language models

    Yan Cai, Linlin Wang, Ye Wang, Gerard de Melo, Ya Zhang, Yanfeng Wang, and Liang He. Medbench: A large- scale chinese benchmark for evaluating medical large language models. In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Inno...

    2024

  26. [34]

    Jmedbench: A benchmark for evaluating japanese biomedical large language models

    Junfeng Jiang, Jiahao Huang, and Akiko Aizawa. Jmedbench: A benchmark for evaluating japanese biomedical large language models. arXiv preprint arXiv:2409.13317, 2024

    2024 arXiv

  27. [35]

    Xuanzhong Chen, Xiaohao Mao, Qihan Guo, Lun Wang, Shuyang Zhang, and Ting Chen. Rarebench: Can llms serve as rare diseases specialists? In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2024, Barcelona, Spain, August 25-29, 2024, page...

    2024

  28. [36]

    Tcmbench: A comprehensive benchmark for evaluating large language models in traditional chinese medicine

    Wenjing Yue, Xiaoling Wang, Wei Zhu, Ming Guan, Huanran Zheng, Pengfei Wang, Changzhi Sun, and Xin Ma. Tcmbench: A comprehensive benchmark for evaluating large language models in traditional chinese medicine. CoRR, abs/2406.01126, 2024

    2024 arXiv

  29. [37]

    Prosa: Assessing and understanding the prompt sensitivity of llms

    Jingming Zhuo, Songyang Zhang, Xinyu Fang, Haodong Duan, Dahua Lin, and Kai Chen. Prosa: Assessing and understanding the prompt sensitivity of llms. arXiv preprint arXiv:2410.12405, 2024. 13 arXiv Template A PREPRINT

    2024 arXiv

  30. [38]

    Generalization or memorization: Data contamination and trustworthy evaluation for large language models

    Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. Generalization or memorization: Data contamination and trustworthy evaluation for large language models. arXiv preprint arXiv:2402.15938, 2024

    2024 arXiv

  31. [39]

    Promptcblue: A chinese prompt tuning benchmark for the medical domain

    Wei Zhu, Xiaoling Wang, and Huanran Zheng. Promptcblue: A chinese prompt tuning benchmark for the medical domain. urlhttps://github.com/michael-wzhu/PromptCBLUE, 2023

    2023

  32. [40]

    Cmb: A comprehensive medical benchmark in chinese

    Xidong Wang, Guiming Hardy Chen, Dingjie Song, Zhiyi Zhang, Zhihong Chen, Qingying Xiao, Feng Jiang, Jianquan Li, Xiang Wan, Benyou Wang, et al. Cmb: A comprehensive medical benchmark in chinese. arXiv preprint arXiv:2308.08833, 2023

    2023 arXiv

  33. [41]

    Llm-as-a-judge & reward model: What they can and cannot do

    Guijin Son, Hyunwoo Ko, Hoyoung Lee, Yewon Kim, and Seunghyeok Hong. Llm-as-a-judge & reward model: What they can and cannot do. arXiv preprint arXiv:2409.11239, 2024

    2024 arXiv

  34. [42]

    Compassjudger- 1: All-in-one judge model helps model evaluation and evolution

    Maosong Cao, Alexander Lam, Haodong Duan, Hongwei Liu, Songyang Zhang, and Kai Chen. Compassjudger- 1: All-in-one judge model helps model evaluation and evolution. arXiv preprint arXiv:2410.16256, 2024

    2024 arXiv

  35. [43]

    Wangyue Li, Liangzhi Li, Tong Xiang, Xiao Liu, Wei Deng, and Noa Garcia. Can multiple-choice questions really be useful in detecting the abilities of llms? In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation...

    2024

  36. [44]

    Pedi- abench: A comprehensive chinese pediatric dataset for benchmarking large language models

    Qian Zhang, Panfeng Chen, Jiali Li, Linkun Feng, Shuyu Liu, Mei Chen, Hui Li, and Yanhao Wang. Pedi- abench: A comprehensive chinese pediatric dataset for benchmarking large language models. arXiv preprint arXiv:2412.06287, 2024

    2024 arXiv

  37. [45]

    Diagnostic reasoning prompts reveal the potential for large language model interpretability in medicine.NPJ Digital Medicine, 7(1):20, 2024

    Thomas Savage, Ashwin Nayak, Robert Gallo, Ekanath Rangan, and Jonathan H Chen. Diagnostic reasoning prompts reveal the potential for large language model interpretability in medicine.NPJ Digital Medicine, 7(1):20, 2024

    2024

  38. [46]

    Large language models and their impact in ophthalmology

    Bjorn Kaijun Betzler, Haichao Chen, Ching-Yu Cheng, Cecilia S Lee, Guochen Ning, Su Jeong Song, et al. Large language models and their impact in ophthalmology. Lancet Digit. Health, 5(12):e917–e924, 2023

    2023

  39. [47]

    How i won singapore’s gpt-4 prompt engineering competition

    Sheila Teo. How i won singapore’s gpt-4 prompt engineering competition. Towards Data Science, Medium, 29, 2023

    2023

  40. [48]

    Alignbench: Benchmarking chinese alignment of large language models

    Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Andrew Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, Xiaohan Zhang, Lichao Sun, Xiaotao Gu, Hongning Wang, Jing Zhang, Minlie Huang, Yuxiao Dong, and Jie Tang. Alignbench: Benchmarking chinese alignment of large l...

    2024

  41. [49]

    Baichuan 2: Open large-scale language models

    Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305, 2023

    2023 arXiv

  42. [50]

    Huatuogpt, towards taming language model to be a doctor

    Hongbo Zhang, Junying Chen, Feng Jiang, Fei Yu, Zhihong Chen, Jianquan Li, Guiming Chen, Xiangbo Wu, Zhiyi Zhang, Qingying Xiao, et al. Huatuogpt, towards taming language model to be a doctor. arXiv preprint arXiv:2305.15075, 2023

    2023 arXiv

  43. [51]

    Huatuogpt-o1, towards medical complex reasoning with llms

    Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. Huatuogpt-o1, towards medical complex reasoning with llms. arXiv preprint arXiv:2412.18925, 2024

    2024 arXiv

  44. [52]

    T \" ulu 3: Pushing frontiers in open language model post-training

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T \" ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024

    2024 arXiv

  45. [53]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  46. [54]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    arXiv

  47. [55]

    Pulse: Pretrained and unified language service engine

    Shaoting Zhang Xiaofan Zhang, Kui Xue. Pulse: Pretrained and unified language service engine. 2023

    2023

  48. [56]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024. 14 ar...

    2024 arXiv

  49. [57]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  50. [58]

    Sunsimiao: Chinese medicine llm

    Dong Xue* Xin Yan. Sunsimiao: Chinese medicine llm. https://github.com/thomas-yanxin/Sunsimiao, 2023

    2023

  51. [59]

    Yi: Open foundation models by 01

    Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652, 2024

    2024 arXiv

  52. [60]

    Deepseek llm: Scaling open-source language models with longtermism

    DeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024

    2024 arXiv

  53. [61]

    Gemma 2: Improving open language models at a practical size

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024

    2024 arXiv

  54. [62]

    Granite 3.0 language models, 2024

    IBM Granite Team. Granite 3.0 language models, 2024

    2024

  55. [63]

    Chatglm: A family of large language models from glm-130b to glm-4 all tools

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793, 2024

    2024 arXiv

  56. [64]

    Internlm2 technical report

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internlm2 technical report. arXiv preprint arXiv:2403.17297, 2024

    2024 arXiv

  57. [65]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024

    2024 arXiv

  58. [66]

    Hunyuan-large: An open-source moe model with 52 billion activated parameters by tencent

    Xingwu Sun, Yanfeng Chen, Yiqing Huang, Ruobing Xie, Jiaqi Zhu, Kai Zhang, Shuaipeng Li, Zhen Yang, Jonny Han, Xiaobo Shu, et al. Hunyuan-large: An open-source moe model with 52 billion activated parameters by tencent. arXiv preprint arXiv:2411.02265, 2024

    2024 arXiv

  59. [67]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. 15

    2024 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.