Pith. sign in

REVIEW 2 major objections 4 minor 3 cited by

This literature survey argues that medical LLMs have moved from information retrieval to multi-step clinical reasoning, and that Chain-of-Thought prompting, specialized medical models, multi-agent collaboration, prompt optimization, and rei

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A literature review of reasoning-LLM techniques for medicine, from CoT prompting to RL-trained medical models, with no new experiments and several placeholder citations.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Reads like a reasonable orientation to medical reasoning LLMs, but the bibliography contains admitted placeholder references, which undercuts the 'comprehensive roadmap' claim and makes it unready for refereeing. the 2 major comments →

arxiv 2508.19097 v1 pith:JLUQYQ5S submitted 2025-08-26 cs.AI

Reasoning LLMs in the Medical Domain: A Literature Survey

Armin Berger , Sarthak Khanna , David Berghaus , Rafet Sifa This is my paper
classification cs.AI
keywords Medical AIreasoning LLMschain-of-thought promptingreinforcement learningmulti-agent systemsclinical decision supportevaluation benchmarksAI safety
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This literature survey argues that medical LLMs have crossed a threshold: they are no longer just retrieving facts but performing multi-step clinical reasoning, and the techniques that made this possible—Chain-of-Thought prompting, specialized medical models, multi-agent collaboration, automated prompt optimization, and reinforcement learning—now form a recognizable research program. The survey's central evidence includes Med-PaLM 2's clinician-preferred answers, MDTeamGPT's multi-agent consultation results, AutoMedPrompt's prompt optimization, and Med-R1's reinforcement-learned medical vision-language reasoning. If this reading of the field is right, the remaining obstacles to clinical deployment are not model scale but evaluation, bias, safety, multimodal integration, and longitudinal reasoning—problems the survey lays out as a future research agenda.

Core claim

On its own terms, the survey claims that reasoning LLMs in medicine have evolved from information retrieval systems into sophisticated reasoning architectures, and that this evolution is driven by a coordinated set of techniques: Chain-of-Thought prompting and its layered and domain-structured variants; purpose-built medical models such as Med-PaLM 2, DiagnosisGPT, and BioMedQ&A; multi-agent frameworks like MDTeamGPT, KG4Diagnosis, and MedAide; automated prompt optimization such as AutoMedPrompt; and reinforcement learning approaches exemplified by DeepSeek-R1 and Med-R1. The survey further asserts that these techniques make reasoning more transparent and verifiable, that resource-efficient

What carries the argument

Chain-of-Thought (CoT) prompting is the central mechanism: prompting the model to externalize intermediate reasoning steps, with variants such as Layered-CoT, Chain-of-Diagnosis, and MedCoT that make steps inspectable or domain-structured. Reinforcement learning, specifically Group Relative Policy Optimization (GRPO) as used in DeepSeek-R1 and Med-R1, is the second mechanism, letting models learn reasoning from rule-based rewards without pre-annotated rationales. Multi-agent frameworks and textual-gradient prompt optimization extend these mechanisms by simulating team deliberation and automatically refining prompts.

Load-bearing premise

The survey's roadmap is only as solid as the primary papers it summarizes: it assumes the cited studies are real and their reported benchmark numbers (e.g., 90.1% MedQA for MDTeamGPT, 87.1% PubMedQA for Gyan) are accurate as stated; the reference list itself contains placeholder entries [48] and [49], so some citations cannot be independently verified.

What would settle it

A reader could check the reference list for entries [48] and [49]; if a literature search finds no traceable publication matching either, the survey's claim that these 'future works' exist collapses. More broadly, independently rerunning the cited benchmarks (e.g., MedQA with MDTeamGPT, PubMedQA with Gyan) under a common evaluation harness would settle whether the reported numbers hold.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If CoT variants and RL training continue to mature, medical LLMs can provide step-by-step rationales that clinicians can audit, supporting adoption in high-stakes settings.
  • Specialized architectures like Gyan and BioMedQ&A suggest explainability and resource efficiency can be achieved without scaling to hundreds of billions of parameters.
  • MDTeamGPT-style multi-agent systems imply that simulating multidisciplinary team deliberation can beat single-model accuracy on benchmarks like MedQA.
  • HealthBench's result—worst-case performance dropping by roughly a third despite doubled average scores—implies that evaluation must shift from average accuracy to reliability in critical scenarios.
  • RL approaches like Med-R1 indicate that small models can outperform much larger SFT-trained VLMs on medical imaging tasks, lowering deployment barriers.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The survey treats the cited primary papers as mutually reinforcing, but many are preprints not peer-reviewed; the roadmap's strength depends on those results replicating in independent evaluations.
  • The reference list contains placeholder entries ([48] 'A. Authors' and [49] 'placeholder for conceptual future works'), so some cited 'future works' are not traceable publications; readers should verify each cited benchmark before relying on it.
  • A testable extension: run the same prompt-optimization and multi-agent techniques on a standardized, clinician-annotated set of real clinical cases rather than exam-style benchmarks, to see whether transparency gains survive in messy, longitudinal records.
  • The HealthBench fragility result suggests a concrete additional benchmark axis: worst-case performance over criticality tiers, not just average score.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. This survey reviews reasoning LLMs in medicine, structuring the field into foundational concepts, Chain-of-Thought prompting and variants (Layered-CoT, MC-CoT), specialized models (Med-PaLM, DiagnosisGPT, BioMedQ&A), multi-agent systems (MDTeamGPT, MedAgents, KG4Diagnosis, MedAide), prompt optimization (AutoMedPrompt, MedCoT, OpenMedLM), evaluation frameworks (MultiMedQA, HealthBench), resource-constrained reasoning (MHMKI, Gyan), reinforcement learning (DeepSeek-R1, Med-R1), and a set of open challenges including interpretability, bias, safety, multimodality, longitudinal reasoning, and clinical workflow integration. The stated aim is to establish a roadmap toward reliable medical reasoning systems.

Significance. If its synthesis is reliable, the survey would be a useful roadmap for researchers entering this rapidly moving area. Its strengths are its broad organization, the inclusion of very recent work such as Med-R1 and HealthBench, and a concrete list of open problems. The paper contains no original experiments or derivations, so its value depends entirely on the accuracy and completeness of the cited literature. That dependence makes the reference-integrity and methodology problems described below load-bearing: until they are fixed, the 'comprehensive examination' claim is not substantiated. With a corrected bibliography and a transparent selection methodology, this could become a valuable survey.

major comments (2)
  1. [References [48], [49]; Sections X.D-E] The bibliography contains admitted non-publications. Reference [48] lists 'A. Authors' as the author, and reference [49] is explicitly labeled 'placeholder for conceptual future works.' Sections X.D and X.E rely on these entries to motivate future research directions. A survey whose central claim is a comprehensive, grounded roadmap cannot rest on placeholder citations. In addition, references [21], [31], and [39] are the authors' own under-review submissions cited as support; they should be clearly marked as such or removed. The authors should audit the entire reference list and replace every placeholder with a real, verifiable source.
  2. [Section I; Sections V.A, VI.A, VIII.B] The paper claims a 'comprehensive examination' but provides no methodology: no search strategy, databases, inclusion/exclusion criteria, time span, or quality appraisal. Specific quantitative claims are repeated as established facts from arXiv preprints, e.g., MDTeamGPT 90.1% MedQA and 83.9% PubMedQA (V.A), AutoMedPrompt 82.6% PubMedQA (VI.A), and Gyan 87.1% PubMedQA (VIII.B). These are author-reported numbers, not independently verified, and the survey does not flag this. Add a methodology section or temper the comprehensiveness claim, and clearly label all reported figures as unverified preprint numbers.
minor comments (4)
  1. [Fig. 1] The figure is difficult to decode; the A/B/C/D labels are not explained in the caption and the relationships between the boxes are unclear.
  2. [Section IV.A] Typo: 'Med-PaLM 2: . Med-PaLM 2, released in 2023...' contains an extra period and a duplicated phrase.
  3. [References [8], [16], [20]] Several entries lack complete bibliographic metadata: [8] gives only a book title, and [16] and [20] give no venue or arXiv identifier. Also, reference [17] misspells 'Chain-of-Thought' as 'Chain-of-though'.
  4. [Section VII.C] The HealthBench discussion is useful, but the claims about 'frontier models doubling average scores' and 'worst-case performance drops by a third' should be accompanied by the specific models and evaluation dates from [38].

Circularity Check

0 steps flagged

No circular derivation: the survey is an external literature review; its self-citations are minor and non-load-bearing, while placeholder references are an integrity issue, not a circularity.

full rationale

This paper is a literature survey, not a derivation or prediction pipeline. Its claims are summaries of external papers, and it contains no equations, fitted parameters, or first-principles results that could reduce to its inputs by construction. The only self-citations are [21], [31], and [39], all listed as under-review submissions by the authors. Each appears in a supporting or motivational role: [21] follows a general statement that medical-domain intricacies drive specialized models; [31] is cited alongside the independent OpenMedLM study [30]; [39] supports the importance of resource-constrained reasoning before citing the concrete MHMKI and Gyan systems. Removing these self-citations would not alter any reported benchmark, framework description, or roadmap step, so they are not load-bearing. The placeholder references [48] and [49] and the unverified preprint numbers repeated from external sources are citation-integrity and verifiability concerns, but they do not make the survey's own synthesis equivalent to its inputs. No specific circular step can be quoted from the paper, so the appropriate finding is no significant circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

No free parameters or invented entities because the paper makes no new measurements or proposals. The central claims rest on unverified external literature and on an implicit sampling of that literature.

axioms (2)
  • domain assumption Cited papers exist and report benchmark numbers accurately as described.
    The survey repeats accuracy figures from cited preprints (e.g., Section V.A, Section VIII.B) without independent verification; refs [48] and [49] are placeholders, so this assumption is partially violated.
  • domain assumption The authors' selection of papers is representative of the field.
    No search protocol, inclusion criteria, or coverage statistics are given; the abstract's claim that the survey is 'thorough' depends on this selection being unbiased.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Reasoning LLMs in the Medical Domain: A Literature Survey." pith.science (2026). https://pith.science/paper/JLUQYQ5S

@misc{pith2026250819097,
  author       = {Pith},
  title        = {Pith review of: Reasoning LLMs in the Medical Domain: A Literature Survey},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JLUQYQ5S}},
  note         = {Machine review of arXiv:2508.19097}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

The emergence of advanced reasoning capabilities in Large Language Models (LLMs) marks a transformative development in healthcare applications. Beyond merely expanding functional capabilities, these reasoning mechanisms enhance decision transparency and explainability-critical requirements in medical contexts. This survey examines the transformation of medical LLMs from basic information retrieval tools to sophisticated clinical reasoning systems capable of supporting complex healthcare decisions. We provide a thorough analysis of the enabling technological foundations, with a particular focus on specialized prompting techniques like Chain-of-Thought and recent breakthroughs in Reinforcement Learning exemplified by DeepSeek-R1. Our investigation evaluates purpose-built medical frameworks while also examining emerging paradigms such as multi-agent collaborative systems and innovative prompting architectures. The survey critically assesses current evaluation methodologies for medical validation and addresses persistent challenges in field interpretation limitations, bias mitigation strategies, patient safety frameworks, and integration of multimodal clinical data. Through this survey, we seek to establish a roadmap for developing reliable LLMs that can serve as effective partners in clinical practice and medical research.

Figures

Figures reproduced from arXiv: 2508.19097 by Armin Berger, David Berghaus, Rafet Sifa, Sarthak Khanna.

Figure 1
Figure 1. Figure 1: The schematic structure of Reasoning LLMs in the Medical Domain. must manage this uncertainty, akin to how clinicians operate with probabilities and differential diagnoses. • Patient-Specificity: Medical decisions are highly indi￾vidualized, needing to account for unique patient his￾tories, genetic predispositions, comorbidities, and socio￾economic factors. • High Stakes and Safety Criticality: Errors in m… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Reasoning4Sciences: Bridging Reasoning Language Models to All Scientific Branches

    cs.AI 2026-05 unverdicted novelty 6.0

    Survey of RLM adoption in 28 disciplines reveals maturity disparities via a new assessment framework, with focus on development, evaluation, and public resources.

  2. Reasoning4Sciences: Bridging Reasoning Language Models to All Scientific Branches

    cs.AI 2026-05 unverdicted novelty 6.0

    A survey of RLM use in 28 disciplines reveals uneven adoption and introduces a maturity assessment framework showing larger gaps when limited to public resources.

  3. Reasoning4Sciences: Bridging Reasoning Language Models to All Scientific Branches

    cs.AI 2026-05 unverdicted novelty 4.0

    A survey of reasoning language model adoption across 28 ERC scientific disciplines finds large maturity gaps, especially when only public resources are counted.

Reference graph

Works this paper leans on

51 extracted references · 26 canonical work pages · cited by 1 Pith paper · 3 internal anchors

  1. [1]

    Large language models in medicine,

    A. J. Thirunavukarasu, D. S. J. Ting, K. Elangovan, L. Gutierrez, T. F. Tan, and D. S. W. Ting, “Large language models in medicine,” Nature medicine, vol. 29, no. 8, pp. 1930–1940, 2023

    1930

  2. [2]

    Foundation models for generalist medical artificial intelligence,

    M. Moor, O. Banerjee, Z. S. H. Abad, H. M. Krumholz, J. Leskovec, E. J. Topol, and P. Rajpurkar, “Foundation models for generalist medical artificial intelligence,” Nature, vol. 616, no. 7956, pp. 259–265, 2023

    work page 2023

  3. [3]

    Large language models in health care: Development, applications, and challenges,

    R. Yang, T. F. Tan, W. Lu, A. J. Thirunavukarasu, D. S. W. Ting, and N. Liu, “Large language models in health care: Development, applications, and challenges,” Health Care Science , vol. 2, no. 4, pp. 255–263, 2023

    work page 2023

  4. [4]

    Llms for relational reasoning: How far are we?

    Z. Li, Y . Cao, X. Xu, J. Jiang, X. Liu, Y . S. Teo, S.-W. Lin, and Y . Liu, “Llms for relational reasoning: How far are we?” in Proceedings of the 1st International Workshop on Large Language Models for Code , 2024, p. 119–126

    work page 2024

  5. [5]

    Large language models for medicine: a survey,

    Y . Zheng, W. Gan, Z. Chen, Z. Qi, Q. Liang, and P. S. Yu, “Large language models for medicine: a survey,” International Journal of Machine Learning and Cybernetics, vol. 16, no. 2, pp. 1015–1040, 2025

    work page 2025

  6. [6]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi et al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” arXiv preprint arXiv:2501.12948 , 2025

    Pith/arXiv arXiv 2025

  7. [7]

    Med-r1: Reinforcement learning for generalizable medical reasoning in vision-language models,

    Y . Lai, J. Zhong, M. Li, S. Zhao, and X. Yang, “Med-r1: Reinforcement learning for generalizable medical reasoning in vision-language models,” arXiv preprint arXiv:2503.13939 , 2025

    arXiv 2025

  8. [8]

    Understanding reasoning in large language models,

    S. Raschka, “Understanding reasoning in large language models,” in Build a Large Language Model , 2025

    work page 2025

  9. [9]

    Biobert: a pre-trained biomedical language representation model for biomedical text mining,

    J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, and J. Kang, “Biobert: a pre-trained biomedical language representation model for biomedical text mining,” Bioinformatics, vol. 36, no. 4, pp. 1234–1240, 2020

    work page 2020

  10. [10]

    A pubmedbert-based classifier with data augmentation strategy for detecting medication mentions in tweets,

    Q. Han, S. Tian, and J. Zhang, “A pubmedbert-based classifier with data augmentation strategy for detecting medication mentions in tweets,” arXiv preprint arXiv:2112.02998 , 2021

    Pith/arXiv arXiv 2021

  11. [11]

    Large language models encode clinical knowledge,

    K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl et al., “Large language models encode clinical knowledge,” Nature, vol. 620, no. 7972, pp. 172– 180, 2023

    work page 2023

  12. [12]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou et al. , “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems , vol. 35, pp. 24 824–24 837, 2022

    2022

  13. [13]

    Large lan- guage models are zero-shot reasoners,

    T. Kojima, S. S. Gu, M. Reid, Y . Matsuo, and Y . Iwasawa, “Large lan- guage models are zero-shot reasoners,” Advances in neural information processing systems, vol. 35, pp. 22 199–22 213, 2022

    2022

  14. [14]

    Toward expert-level medical question answering with large language models,

    K. Singhal, T. Tu, J. Gottweis, R. Sayres, E. Wulczyn, M. Amin, L. Hou, K. Clark, S. R. Pfohl, H. Cole-Lewis et al., “Toward expert-level medical question answering with large language models,” Nature Medicine, pp. 1–8, 2025

    work page 2025

  15. [15]

    Layered chain-of-thought prompting for multi-agent llm systems: A comprehensive approach to explainable large language models,

    M. Sanwal, “Layered chain-of-thought prompting for multi-agent llm systems: A comprehensive approach to explainable large language models,” arXiv preprint arXiv:2501.18645 , 2025

    Pith/arXiv arXiv 2025

  16. [16]

    Improving medical chatbot with chain-of-thought and document retrieval techniques,

    Z. Wu, A. Liu, Z. Liu, S. Wang, L. Shu, T. M. Sun, Z. Mcinnis, and H. Ji, “Improving medical chatbot with chain-of-thought and document retrieval techniques,” 2024

    work page 2024

  17. [17]

    Chain-of-though (cot) prompting strategies for medical error detection and correction,

    Z. Wu, A. Hasan, J. Wu, Y . Kim, J. P. Cheung, T. Zhang, and H. Wu, “Chain-of-though (cot) prompting strategies for medical error detection and correction,” arXiv preprint arXiv:2406.09103 , 2024

    Pith/arXiv arXiv 2024

  18. [18]

    Mc-cot: A modular collaborative cot framework for zero-shot medical-vqa with llm and mllm integration,

    L. Wei, W. Wang, X. Shen, Y . Xie, Z. Fan, X. Zhang, Z. Wei, and W. Chen, “Mc-cot: A modular collaborative cot framework for zero-shot medical-vqa with llm and mllm integration,” arXiv preprint arXiv:2410.04521, 2024

    Pith/arXiv arXiv 2024

  19. [19]

    J. Lee, A. Chen, and R. Smith, “Automated pathologic tn classification prediction and rationale generation from lung cancer surgical pathology reports using a large language model fine-tuned with chain-of-thought,” JMIR Med Inform , vol. 12, 2024

    work page 2024

  20. [20]

    Medreason: Eliciting factual medical reasoning steps in llms via knowledge graphs,

    W. Zhang, X. Li, and Q. Wang, “Medreason: Eliciting factual medical reasoning steps in llms via knowledge graphs,” 2025

    work page 2025

  21. [21]

    Advancing personalized medicine: A scal- able llm-based recommender system for patient matching,

    A. Berger, D. Berghaus, A. H. Bashir, L. Grigull, L. Fendrich, T. A. Lagones, H. H ¨ogl, G. Ernst, R. Schmidt, D. Bascom, T. Deußer, T. Bell, M. L ¨ubbering, and R. Sifa, “Advancing personalized medicine: A scal- able llm-based recommender system for patient matching,” Submitted to Big Data 2024 , 2024, under review

    work page 2024

  22. [22]

    Cod, towards an interpretable medical agent using chain of diagnosis,

    J. Chen, C. Gui, A. Gao, K. Ji, X. Wang, X. Wan, and B. Wang, “Cod, towards an interpretable medical agent using chain of diagnosis,” arXiv preprint arXiv:2407.13301, 2024

    Pith/arXiv arXiv 2024

  23. [23]

    Biomedq&a: An intelligent biogpt-powered transformer model for accurate biomedical answer retrieval from medquad,

    T. Khatoon, “Biomedq&a: An intelligent biogpt-powered transformer model for accurate biomedical answer retrieval from medquad,” Journal of Information Systems Engineering and Management , vol. 10, pp. 380– 394, 03 2025

    work page 2025

  24. [24]

    Mdteamgpt: A self-evolving llm-based multi-agent framework for multi-disciplinary team medical consultation,

    K. Chen, X. Li, T. Yang, H. Wang, W. Dong, and Y . Gao, “Mdteamgpt: A self-evolving llm-based multi-agent framework for multi-disciplinary team medical consultation,” arXiv preprint arXiv:2503.13856 , 2025

    Pith/arXiv arXiv 2025

  25. [25]

    Medagents: Large language models as collaborators for zero-shot medical reasoning,

    X. Tang, A. Zou, Z. Zhang, Z. Li, Y . Zhao, X. Zhang, A. Cohan, and M. Gerstein, “Medagents: Large language models as collaborators for zero-shot medical reasoning,” 01 2024, pp. 599–621

    work page 2024

  26. [26]

    Kg4diagnosis: A hierarchical multi-agent llm framework with knowledge graph enhance- ment for medical diagnosis,

    C. Liu, Y . Wang, L. Zhang, M. Chen, and J. Zhou, “Kg4diagnosis: A hierarchical multi-agent llm framework with knowledge graph enhance- ment for medical diagnosis,” arXiv preprint arXiv:2412.16833 , 2025

    Pith/arXiv arXiv 2025

  27. [27]

    Medaide: Towards an omni medical aide via specialized llm-based multi-agent collaboration,

    Q. Zhao, Z. Liu, Y . Wu, X. Zhang, and H. Li, “Medaide: Towards an omni medical aide via specialized llm-based multi-agent collaboration,” arXiv preprint arXiv:2410.12532 , 2024

    Pith/arXiv arXiv 2024

  28. [28]

    Automedprompt: A new framework for optimizing llm medical prompts using textual gradients,

    S. Wu, M. Koo, F. Scalzo, and I. Kurtz, “Automedprompt: A new framework for optimizing llm medical prompts using textual gradients,” arXiv preprint arXiv:2502.15944 , 2025

    Pith/arXiv arXiv 2025

  29. [29]

    Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment

    Y . Zhao, L. Wang, M. Zhang, X. Liu, and W. Chen, “Medcot: Medi- cal chain-of-thought prompting for multi-diagnosis clinical reasoning,” arXiv preprint arXiv:2503.12847 , 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  30. [30]

    Openmedlm: prompt engineering can out-perform fine-tuning in medical question- answering with open-source large language models,

    J. Maharjan, A. Garikipati, N. P. Singh, L. Cyrus, M. Sharma, M. Ciobanu, G. Barnes, R. Thapa, Q. Mao, and R. Das, “Openmedlm: prompt engineering can out-perform fine-tuning in medical question- answering with open-source large language models,” Scientific Reports, vol. 14, no. 1, p. 14156, 2024

    work page 2024

  31. [31]

    Tackling data sparsity and combinatorial challenges in rare disease matching with medical informed machine learning,

    A. Berger, T. A. Lagones, L. Grigull, L. Fendrich, T. Bell, H. H ¨ogl, G. Ernst, R. Schmidt, D. Bascom, R. Sifa, and M. L ¨ubbering, “Tackling data sparsity and combinatorial challenges in rare disease matching with medical informed machine learning,” Submitted to Big Data 2024, 2024, under review

    work page 2024

  32. [32]

    Pubmedqa: A dataset for biomedical research question answering,

    Q. Jin, B. Dhingra, Z. Liu, W. Cohen, and X. Lu, “Pubmedqa: A dataset for biomedical research question answering,” 09 2019

    work page 2019

  33. [33]

    Holistic evaluation of language models,

    P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y . Zhang, D. Narayanan, Y . Wu, A. Kumar et al., “Holistic evaluation of language models,” arXiv preprint arXiv:2211.09110 , 2022

    Pith/arXiv arXiv 2022

  34. [34]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models,

    A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso et al., “Beyond the imitation game: Quantifying and extrapolating the capabilities of language models,” arXiv preprint arXiv:2206.04615 , 2022

    Pith/arXiv arXiv 2022

  35. [35]

    A survey on medical large language models: Technology, application, trustworthiness, and future directions,

    L. Liu, X. Yang, J. Lei, X. Liu, Y . Shen, Z. Zhang, P. Wei, J. Gu, Z. Chu, Z. Qin et al. , “A survey on medical large language models: Technology, application, trustworthiness, and future directions,” arXiv preprint arXiv:2406.03712, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    Performance of chatgpt on usmle: potential for ai-assisted medical education using large language models,

    T. H. Kung, M. Cheatham, A. Medenilla, C. Sillos, L. De Leon, C. Elepa ˜no, M. Madriaga, R. Aggabao, G. Diaz-Candido, J. Maningo et al. , “Performance of chatgpt on usmle: potential for ai-assisted medical education using large language models,” PLoS digital health , vol. 2, no. 2, p. e0000198, 2023

    work page 2023

  37. [37]

    Capabilities of gpt-4 on medical challenge problems,

    H. Nori, N. King, S. M. McKinney, D. Carignan, and E. Horvitz, “Capabilities of gpt-4 on medical challenge problems,” arXiv preprint arXiv:2303.13375, 2023

    Pith/arXiv arXiv 2023

  38. [38]

    Healthbench: Evaluating large language models towards improved human health,

    R. K. Arora, J. Wei, R. S. Hicks, P. Bowman, J. Qui ˜nonero-Candela, F. Tsimpourlas, M. Sharman, M. Shah, A. Vallone, A. Beutel, J. Hei- decke, and K. Singhal, “Healthbench: Evaluating large language models towards improved human health,” arXiv preprint arXiv:2505.08775 , 2025

    Pith/arXiv arXiv 2025

  39. [39]

    Optimizing rare disease patient matching with large language models,

    A. Berger, A. H. Bashir, D. Berghaus, Mowmita, N. Afsan, L. Grigull, L. Fendrich, H. H ¨ogl, G. Ernst, R. Schmidt, D. Bascom, T. A. Lagones, T. Deußer, T. Bell, M. L ¨ubbering, and R. Sifa, “Optimizing rare disease patient matching with large language models,” Submitted to Big Data 2024, 2024, under review

    work page 2024

  40. [40]

    Infusing multi-hop medi- cal knowledge into smaller language models for biomedical question answering,

    J. Chen, Z. Wei, W. Shen, and R. Shang, “Infusing multi-hop medi- cal knowledge into smaller language models for biomedical question answering,” IEEE Journal of Biomedical and Health Informatics , 2025

    work page 2025

  41. [41]

    On the Performance of an Explainable Language Model on PubMedQA

    V . Srinivasan, V . Jatav, A. Chandrababu, and G. Sharma, “On the performance of an explainable language model on pubmedqa,” arXiv preprint arXiv:2504.05074, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  42. [42]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models,

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y . Li, Y . Wuet al., “Deepseekmath: Pushing the limits of mathematical reasoning in open language models,” arXiv preprint arXiv:2402.03300 , 2024

    Pith/arXiv arXiv 2024

  43. [43]

    Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning,

    T. Xie, Z. Gao, Q. Ren, H. Luo, Y . Hong, B. Dai, J. Zhou, K. Qiu, Z. Wu, and C. Luo, “Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning,” arXiv preprint arXiv:2502.14768 , 2025

    Pith/arXiv arXiv 2025

  44. [44]

    Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution,

    Y . Wei, O. Duchenne, J. Copet, Q. Carbonneaux, L. Zhang, D. Fried, G. Synnaeve, R. Singh, and S. I. Wang, “Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution,” arXiv preprint arXiv:2502.18449, 2025

    Pith/arXiv arXiv 2025

  45. [45]

    Challenges in Ensuring AI Safety in DeepSeek-R1 Models: The Shortcomings of Reinforcement Learning Strategies

    M. Parmar and Y . Govindarajulu, “Challenges in ensuring ai safety in deepseek-r1 models: The shortcomings of reinforcement learning strategies,” arXiv preprint arXiv:2501.17030 , 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  46. [46]

    Taxonomy of risks posed by language models,

    L. Weidinger, J. Uesato, M. Rauh, C. Griffin, P.-S. Huang, J. Mellor, A. Glaese, M. Cheng, B. Balle, A. Kasirzadeh et al., “Taxonomy of risks posed by language models,” in Proceedings of the 2022 ACM conference on fairness, accountability, and transparency , 2022, pp. 214–229

    work page 2022

  47. [47]

    Truthfulqa: Measuring how models mimic human falsehoods,

    S. Lin, J. Hilton, and O. Evans, “Truthfulqa: Measuring how models mimic human falsehoods,” arXiv preprint arXiv:2109.07958 , 2021

    Pith/arXiv arXiv 2021

  48. [48]

    Timer: Temporal instruction modeling and evaluation for longitudinal clinical records,

    A. Authors, “Timer: Temporal instruction modeling and evaluation for longitudinal clinical records,” arXiv preprint arXiv:2503.04176 , 2025

    Pith/arXiv arXiv 2025

  49. [49]

    Anticipated advancements in medical llm reasoning (placeholder a),

    F. R. et al., “Anticipated advancements in medical llm reasoning (placeholder a),” Represents anticipated future work or placeholder arXiv IDs (e.g., arXiv:2409.19487, arXiv:2411.13599, arXiv:2502.17132, arXiv:2409.08523) mentioned in the source document survey., 2024- 2025, placeholder for conceptual future works mentioned in original survey text

    Pith/arXiv arXiv 2024

  50. [50]

    Gpt-4o system card,

    OpenAI, A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford et al., “Gpt-4o system card,” 2024

    work page 2024

  51. [51]

    Gemini: A family of highly capable multimodal models,

    Gemini Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican et al. , “Gemini: A family of highly capable multimodal models,” 2025

    work page 2025

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.