Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

Training LLMs for EHR-Based Reasoning Tasks via Reinforcement Learning

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 3B model beats GPT-4 on clinical reasoning via RLVR

desk verdict A practical 3B-parameter RLVR recipe for EHR reasoning that shows real gains on three benchmarks, but the SFT trace provenance and missing error bars mean the headline superiority claims should be read with caution. read the letter →

arxiv 2505.24105 v1 pith:5DJPUUQQ submitted 2025-05-30 cs.CL

Jiacheng Lin , Zhenbang Wu , Jimeng Sun This is my paper
classification cs.CL
keywords reinforcementlearningwithverifiablerewardselectronichealthrecordsclinicalreasoningGRPOsupervisedfine-tuningwarm-upPass@kmedicalcalculationpatient-trialmatching
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EHRMIND claims that reinforcement learning with verifiable rewards (RLVR), where only the final answer's correctness is rewarded, can train small open LLMs to reason over electronic health records, and that a brief supervised warm-up fixes the one case where pure RL fails. On the MedCalc medical-calculation benchmark, a 3B LLaMA-3 model trained with RLVR alone reaches 41.26% exact-match accuracy, above GPT-4's 37.92%, and adding an SFT warm-up on about 2,000 reasoning examples raises it to 51.96%, above o3-mini and DeepSeek-R1. The same recipe improves patient-trial matching and disease diagnosis, and the paper identifies when warm-up is needed via Pass@k: when the base model rarely produces a correct answer, RLVR itself cannot discover the missing knowledge. The practical upshot is that outcome-level feedback can unlock clinical reasoning in models small enough for local deployment, provided the training data covers the relevant medical knowledge.

What carries the argument

The machinery is Group Relative Policy Optimization (GRPO) with a rule-based reward: for each query the model samples twelve responses, each is scored by exact-match or label accuracy against ground truth, and advantages are normalized within the group to update the policy. Before that, a lightweight SFT warm-up trains on roughly two thousand GPT-4o-generated step-by-step reasoning traces to supply missing domain knowledge and stabilize RL. The paper also introduces Pass@k, and a stricter Reliable Pass@k for classification, as a pre-training diagnostic: Pass@12 of the base model on the training set correlates with RL-induced gain at R²=0.91 on MedCalc, telling practitioners when the warm-up is required.

What would settle it

Regenerate the SFT warm-up traces without giving GPT-4o the ground-truth label, retrain EHRMIND-SFT-RLVR, and compare on the MedCalc test set; if the 51.96% accuracy or the 30-to-40 point gains on trial matching collapse, the warm-up's effect was label leakage rather than knowledge injection. A stronger test would also evaluate the trained model on a held-out hospital's EHRs.

Watch

Extended reading notes

Core claim

The paper's central claim is that a two-stage recipe, lightweight supervised fine-tuning (SFT) warm-up followed by GRPO-based RLVR, lets a 3B model outperform all evaluated open-source and proprietary baselines on EHR-based reasoning. The headline evidence is MedCalc: EHRMIND-SFT-RLVR achieves 51.96% overall accuracy versus 48.13% for DeepSeek-R1, 46.42% for o3-mini, 41.18% for Claude-3.5 Sonnet, and 37.92% for GPT-4, and it does so by injecting knowledge via roughly two thousand reasoning traces and then reinforcing correct outcomes. RLVR alone already lifts the same 3B backbone from 9.74% to 41.26%, but only in categories where the model has relevant knowledge (high Pass@12); when knowledge is absent, reward is sparse and training fails, which motivates the SFT warm-up. On TREC patient-trial matching, the full recipe reaches 63.14% balanced accuracy and 44.70% macro F1, and on four EHRSHOT diagnosis tasks it achieves the best or competitive accuracy while producing more structured, clinically interpretable rationales.

Load-bearing premise

The claim that SFT warm-up injects missing medical knowledge assumes the GPT-4o-generated reasoning traces are medically correct and do not leak the ground-truth label; if they merely rationalize a label through spurious correlations, the reported gains could be overfitting to a teacher style rather than genuine clinical learning.

Editorial extensions

If this is right

  • A 3B model trained with this recipe can exceed DeepSeek-R1 and GPT-4 on MedCalc and outperform all baselines on patient-trial matching, suggesting model scale is not the binding constraint for these clinical reasoning tasks.
  • Pass@k computed on a few hundred training examples in about one hour can predict whether SFT warm-up is needed before committing to full RLVR training, saving compute in deployment pipelines.
  • The SFT warm-up prevents reasoning collapse: pure RLVR on classification tasks tends to omit reasoning, while SFT-RLVR produces longer, structured rationales that GPT-4o prefers in pairwise evaluation.
  • RLVR-based training transfers across diagnosis tasks better than SFT-only training, so outcome optimization encourages reusable clinical reasoning patterns.
  • Success depends on training data covering the underlying medical knowledge; performance on unseen formulas and concepts stays low, so dataset diversity is a practical requirement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Left implicit is that the same Pass@k diagnostic could serve as a general RLVR readiness test for other specialized domains such as legal, financial, or scientific reasoning where verifiable rewards exist but knowledge coverage varies.
  • Because the warm-up traces were generated by GPT-4o with ground-truth labels in the prompt, part of the SFT benefit may be imitation of GPT-4o's reasoning style; testing with label-agnostic or human-written traces would separate genuine knowledge injection from style transfer.
  • If the gains replicate with human-verified traces and on held-out hospital data, the recipe offers a path for privacy-preserving clinical model adaptation from a small, curated example set rather than large annotated corpora.
  • The rule-based reward requirement restricts the recipe to tasks with verifiable outputs; extending it to narrative medical generation would need a reliable automatic clinical-quality judge.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes EHRMIND, a two-stage recipe for adapting a 3B LLaMA-3 model to EHR-based reasoning tasks via GRPO-based RLVR, optionally preceded by a lightweight SFT warm-up. On MedCalc, RLVR alone raises exact-match accuracy from 9.74% to 41.26%, and SFT warm-up on about 2k reasoning examples followed by RLVR reaches 51.96%, which the authors report as outperforming GPT-4, o3-mini, and DeepSeek-R1. The paper also reports large gains on TREC Clinical Trials patient-trial matching and EHRSHOT diagnosis prediction, and introduces Pass@k / Reliable Pass@k as a diagnostic for deciding when SFT warm-up is necessary. The method is clearly described, and the appendices disclose prompt templates, data construction, and hyperparameters.

Significance. The central empirical claim—that outcome-only RLVR can produce large absolute gains on a 3B model for a clinical calculation benchmark, and that a small warm-up set can unlock sparse-reward categories—is timely and practically relevant for resource-constrained clinical NLP. The paper is transparent about its data-construction pipeline and includes a limitations section, which is commendable. If the claims survive re-evaluation with error bars, prospectively validated Pass@k thresholds, and a check that the SFT traces inject genuine medical knowledge rather than rationalization artifacts, the recipe would be a useful contribution. As it stands, the headline "state-of-the-art" and "cross-task generalization" claims are not yet statistically supported, and there is a concrete internal inconsistency in the reported test-set sizes.

major comments (5)
  1. [§3.2, Table 1] The headline result that EHRMIND-SFT-RLVR outperforms all baselines is based on a single training run with no error bars, confidence intervals, or significance tests. The overall gap over DeepSeek-R1 is 51.96% vs. 48.13% on n=1047; a two-proportion z-test gives z≈1.8 (p≈0.08 two-sided), so the difference is within sampling noise. The per-category test sets are very small (Dosage n=40, Severity n=60), and the checkpoint is selected on a validation set, which further inflates apparent gains. Please report multiple seeds and paired significance tests (e.g., McNemar) or confidence intervals, and qualify the state-of-the-art claim accordingly. The same absence of uncertainty quantification affects Tables 2 and 3.
  2. [§3.3, Figure 2] The Pass@k diagnostic is validated on the same seven MedCalc categories used to motivate it: the R²=0.91 correlation is a post-hoc fit, not a prospective test. No threshold is defined for what counts as "low" Pass@k before deciding to use SFT, and the Reliable Pass@k definition in Appendix E introduces ad-hoc constants (τp = ⌈k/C⌉+2, τe = 0.8 log C) with no sensitivity analysis. As presented, the heuristic is a plausible hypothesis, but the paper does not demonstrate that it reliably predicts SFT necessity on held-out tasks or categories.
  3. [§3.2, Table 7] Table 7 is internally inconsistent with Table 5: Table 5 reports a MedCalc test set of 1,047 examples, while Table 7 reports category totals of 19, 12, 3, 16, 12, 4, and 3, summing to 69. If the seen/unseen analysis in Figure 1 is based on these 69 examples, the unseen subsets contain only 4 (Lab), 8 (Risk), and 3 (Severity) examples, making the accuracy estimates extremely noisy—a single correct answer changes the rate by roughly 12–33 percentage points. If the analysis uses the full test set, the table is wrong. The paper must reconcile these numbers and report the seen/unseen analysis with exact counts and intervals.
  4. [Appendices F.2.2 and F.3.3] The SFT warm-up traces for TREC Clinical Trials and EHRSHOT are generated by GPT-4o from prompts that include the ground-truth label (Tables 8 and 10), and the only filter is GPT-4o's self-reported confidence. No medical validation or label-leakage check is reported. Since GRPO trains on these full traces, the "missing knowledge injection" story is not distinguished from overfitting to GPT-4o's rationalization style or to label-correlated shortcuts. The GPT-4o-as-judge pairwise evaluation in Appendix F.3.4 uses the same model family that generated the traces, so it does not provide independent evidence that the learned reasoning is clinically meaningful. Please add human or independent validation of a sample of traces, an analysis of shortcut reliance, or re-frame the transfer and interpretability claims to acknowledge this confound.
  5. [§4.4, Figure 4] The cross-task generalization claim rests on training on Hyperlipidemia only and evaluating on three other diagnosis tasks, with no error bars and no control for per-task class balance or difficulty. The differences in Figure 4 (e.g., 15.38 vs. 22.71 F1 on Acute MI) are reported without uncertainty quantification, and the conclusion that "RL-based optimization generalizes better" is not statistically supported. Please provide repeated runs or confidence intervals, and state whether the compared training setups use identical data sizes and hyperparameters.
minor comments (5)
  1. [Appendix F.1.3] The text says "PPO parameters via the VeRL codebase" but the method uses GRPO; please correct this to avoid confusion.
  2. [§4.2] The description of training for TREC and EHRSHOT says class imbalance is handled by downsampling, but the exact downsampling ratios and the resulting class distributions are not reported; please add them.
  3. [Figure 3(b)] The bar chart for the GPT-4o pairwise evaluation does not show the actual win/tie percentages, only the qualitative pattern; please include the numeric values so readers can assess the effect size.
  4. [§4.4, Finding 1] The phrase "reasoning supervision during pretraining" is imprecise; the supervision occurs in the SFT warm-up phase of fine-tuning, not during pretraining.
  5. [Appendix E] The definition of Reliable Pass@k estimates the label-space size C by enumerating unique ground-truth values in the training data, but for Diagnosis and Severity the paper sets C=7 and for Risk C=21; it would help to state explicitly how sensitive the reported Pass@k values are to these choices.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central results are empirically self-contained, and the main caveats are data-quality and statistical risks rather than circular reasoning.

full rationale

The paper's headline claims are empirical comparisons on held-out test sets of MedCalc, TREC Clinical Trials, and EHRSHOT. Training uses GRPO with rule-based rewards derived from ground-truth labels, plus an SFT warm-up; neither the reward nor the SFT objective is defined in terms of the reported test metric, and the test results are not used to construct the training signal. The Pass@12 analysis (Section 3.3, Figure 2) is an in-sample correlation between the base model's pass rate on the training set and the RL-induced accuracy gain on the test set. It is a post-hoc diagnostic rather than a fitted parameter renamed as a prediction: no regression coefficient from this correlation is used to produce the reported gains, and the gains themselves come from actual rollouts on held-out data. The SFT traces for TREC and EHRSHOT are generated by GPT-4o with the ground-truth label included in the prompt (Appendices F.2.2 and F.3.3), which is a genuine data-quality limitation: the traces may rationalize the label rather than teach transferable clinical knowledge, and the paper does not control for this. However, this is not circularity in the required sense, because the supervised labels are legitimate training targets and the paper does not derive its generalization claims from the label-conditioned traces by construction. The GPT-4o-as-judge comparison (Appendix F.3.4) raises an evaluator-bias risk rather than a circularity. Citations to the authors' prior work (e.g., [17]) are methodological references, not load-bearing uniqueness theorems. No equation reduces to its own input, and no fitted parameter is renamed as a prediction; therefore the appropriate circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central method introduces no new entities; it relies on three domain assumptions (base model adequacy, reward fidelity, and quality of LLM-generated supervision) and on hand-chosen threshold constants for the new diagnostic. The 'Reliable Pass@k' metric is a new construct, but it is a definition rather than an invented physical entity.

free parameters (3)
  • Reliable Pass@k threshold constants = tau_p = ceil(k/C)+2; tau_e = 0.8*log(C)
    Hand-chosen constants in Appendix E define when a sample counts as a 'confident pass'. Changing them changes the diagnostic and therefore the SFT decision.
  • GRPO training hyperparameters = lr=1e-6, KL=0.001, batch=128, k=12, temp=0.6
    Reported in Appendix F.1.3. Standard values but necessary for reproduction.
  • Implicit Pass@k cutoff for SFT necessity = not specified
    The paper recommends SFT when Pass@k is 'low' but never defines a concrete cutoff, making the rule post hoc.
assumptions (3)
  • domain assumption LLaMA-3-3B-Instruct is an adequate base model for EHR reasoning
    All results are on this backbone; no other base model is tested.
  • domain assumption Rule-based rewards (exact match, accuracy) faithfully measure clinical correctness
    Section 2.2.1 states this; Section 4.4 acknowledges reward hacking in binary classification, partially violating it.
  • domain assumption GPT-4o-generated reasoning traces are medically sound and leakage-free
    Used to build SFT sets for TREC and EHRSHOT (Appendices F.2.2, F.3.3) without human expert review.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Training LLMs for EHR-Based Reasoning Tasks via Reinforcement Learning." pith.science (2026). https://pith.science/paper/5DJPUUQQ

@misc{pith2026250524105,
  author       = {Pith},
  title        = {Pith review of: Training LLMs for EHR-Based Reasoning Tasks via Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5DJPUUQQ}},
  note         = {Machine review of arXiv:2505.24105}
}
read the original abstract

We present EHRMIND, a practical recipe for adapting large language models (LLMs) to complex clinical reasoning tasks using reinforcement learning with verifiable rewards (RLVR). While RLVR has succeeded in mathematics and coding, its application to healthcare contexts presents unique challenges due to the specialized knowledge and reasoning required for electronic health record (EHR) interpretation. Our pilot study on the MEDCALC benchmark reveals two key failure modes: (1) misapplied knowledge, where models possess relevant medical knowledge but apply it incorrectly, and (2) missing knowledge, where models lack essential domain knowledge. To address these cases, EHRMIND applies a two-stage solution: a lightweight supervised fine-tuning (SFT) warm-up that injects missing domain knowledge, stabilizes subsequent training, and encourages structured, interpretable outputs; followed by RLVR, which reinforces outcome correctness and refines the model's decision-making. We demonstrate the effectiveness of our method across diverse clinical applications, including medical calculations (MEDCALC), patient-trial matching (TREC CLINICAL TRIALS), and disease diagnosis (EHRSHOT). EHRMIND delivers consistent gains in accuracy, interpretability, and cross-task generalization. These findings offer practical guidance for applying RLVR to enhance LLM capabilities in healthcare settings.

Figures

Figures reproduced from arXiv: 2505.24105 by the authors.

Figure 1
Figure 1. Comparison of accuracy on seen vs. unseen [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Relationship between Pass@k on RLVR training set and RL-induced test set improvement. Each point corresponds to one category from MEDCALC. The x-axis shows the Pass@12 of LLaMA-3-3B on the RLVR training set, and the y-axis shows the resulting accuracy gain on the test set after applying EHRMIND-RLVR. A strong cor￾relation (R2 = 0.91) suggests that low ini￾tial Pass@12 predicts minimal RL improve￾ment—highlighting wh… view at source ↗
Figure 3
Figure 3. Quantitative and qualitative analysis of reasoning improvements from SFT warm-up on [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Comparison of cross-task general￾ization performance. Each model is trained exclusively on Hyperlipidemia data and eval￾uated on out-of-distribution diagnosis tasks. RLVR generalizes better than SFT models, suggesting reinforcement learning promotes transferable reason…
Figure 5
Figure 5. Figure 5: Average number of events per patient by event type in the [PITH_FULL_IMAGE:figures/full_fig_p020_5.png]
Figure 6
Figure 6. Figure 6: AUROC of XGBoost classifiers trained on individual event types across the four diag [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Comprehensive Survey of Electronic Health Record Modeling: From Deep Learning Approaches to Large Language Models

    cs.LG 2025-07 reject novelty 4.0 of 10

    A survey that taxonomizes EHR modeling research into data-centric, architectural, learning-focused, multimodal, and LLM-based categories, with datasets and metrics.

Reference graph

Works this paper leans on

81 extracted references · 34 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Claude 3 haiku: our fastest model yet, March 2024

    Anthropic. Claude 3 haiku: our fastest model yet, March 2024. Accessed: 2025-05-01

    2024

  3. [3]

    Claude 3.5 sonnet, June 2024

    Anthropic. Claude 3.5 sonnet, June 2024. Accessed: 2025-05-01

    work page 2024

  4. [4]

    Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, ...

    2022

  5. [5]

    Rm-r1: Reward modeling as reasoning.arXiv preprint arXiv:2505.02387, 2025

    Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru Wang, Yu Zhang, Denghui Zhang, Tong Zhang, et al. Rm-r1: Reward modeling as reasoning.arXiv preprint arXiv:2505.02387, 2025

    arXiv 2025

  6. [6]

    Timer: Temporal instruction modeling and evaluation for longitudinal clinical records.arXiv preprint arXiv:2503.04176, 2025

    Hejie Cui, Alyssa Unell, Bowen Chen, Jason Alan Fries, Emily Alsentzer, Sanmi Koyejo, and Nigam Shah. Timer: Temporal instruction modeling and evaluation for longitudinal clinical records.arXiv preprint arXiv:2503.04176, 2025

    arXiv 2025

  7. [7]

    A new paradigm for accelerating clinical data science at stanford medicine, 2020

    Somalee Datta, Jose Posada, Garrick Olson, Wencheng Li, Ciaran O’Reilly, Deepa Balraj, Joseph Mesterhazy, Joseph Pallas, Priyamvada Desai, and Nigam Shah. A new paradigm for accelerating clinical data science at stanford medicine, 2020

    work page 2020

  8. [8]

    Electronic health records: then, now, and in the future.Yearbook of medical informatics, 25(S 01):S48–S61, 2016

    R Scott Evans. Electronic health records: then, now, and in the future.Yearbook of medical informatics, 25(S 01):S48–S61, 2016

    2016

Show all 81 references
  1. [9]

    Fleming, Alejandro Lozano, William J

    Scott L. Fleming, Alejandro Lozano, William J. Haberkorn, Jenelle A. Jindal, Eduardo P. Reis, Rahul Thapa, Louis Blankemeier, Julian Z. Genkins, Ethan Steinberg, Ashwin Nayak, Birju S. Patel, Chia-Chun Chiang, Alison Callahan, Zepeng Huo, Sergios Gatidis, Scott J. Adams, Oluse...

    2023

  2. [10]

    Metrics for multi-class classification: An overview.stat, 1050:13, 2020

    Margherita Grandini, CRIF SpA, Enrico Bagli, and Giorgio Visani. Metrics for multi-class classification: An overview.stat, 1050:13, 2020

    2020

  3. [11]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  4. [12]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  5. [13]

    Kale, Greg Ver Steeg, and Aram Galstyan

    Hrayr Harutyunyan, Hrant Khachatrian, David C. Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data.Scientific Data, 6(1):96, 2019

    2019

  6. [14]

    Gencls++: Pushing the boundaries of generative classification in llms through comprehensive sft and rl studies across diverse datasets.arXiv preprint arXiv:2504.19898, 2025

    Mingqian He, Fei Zhao, Chonggang Lu, Ziyan Liu, Yue Wang, and Haofu Qian. Gencls++: Pushing the boundaries of generative classification in llms through comprehensive sft and rl studies across diverse datasets.arXiv preprint arXiv:2504.19898, 2025

    2025 arXiv

  7. [15]

    Measuring massive multitask language understanding, 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. 10

    2021

  8. [16]

    Deepretrieval: Hacking real search engines and retrievers with large language models via reinforcement learning.arXiv preprint arXiv:2503.00223, 2025

    Pengcheng Jiang, Jiacheng Lin, Lang Cao, Runchu Tian, SeongKu Kang, Zifeng Wang, Jimeng Sun, and Jiawei Han. Deepretrieval: Hacking real search engines and retrievers with large language models via reinforcement learning.arXiv preprint arXiv:2503.00223, 2025

    2025 arXiv

  9. [17]

    Reasoning-enhanced healthcare predictions with knowledge graph community retrieval.arXiv preprint arXiv:2410.04585, 2024

    Pengcheng Jiang, Cao Xiao, Minhao Jiang, Parminder Bhatia, Taha Kass-Hout, Jimeng Sun, and Jiawei Han. Reasoning-enhanced healthcare predictions with knowledge graph community retrieval.arXiv preprint arXiv:2410.04585, 2024

    2024 arXiv

  10. [18]

    Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516, 2025

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Za- mani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516, 2025

    2025 arXiv

  11. [19]

    What disease does this patient have? a large-scale open domain question answering dataset from medical exams.arXiv preprint arXiv:2009.13081, 2020

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams.arXiv preprint arXiv:2009.13081, 2020

    2009 arXiv

  12. [20]

    Medcalc-bench: Evaluating large language models for medical calculations.Advances in Neural Information Processing Systems, 37:84730–84745, 2024

    Nikhil Khandekar, Qiao Jin, Guangzhi Xiong, Soren Dunn, Serina Applebaum, Zain Anwar, Maame Sarfo-Gyamfi, Conrad Safranek, Abid Anwar, Andrew Zhang, et al. Medcalc-bench: Evaluating large language models for medical calculations.Advances in Neural Information Processing System...

    2024

  13. [21]

    Enhancing llms’ clinical reasoning with real-world data from a nationwide sepsis registry.arXiv preprint arXiv:2505.02722, 2025

    Junu Kim, Chaeeun Shim, Sungjin Park, Su Yeon Lee, Gee Young Suh, Chae-Man Lim, Seong Jin Choi, Song Mi Moon, Kyoung-Ho Song, Eu Suk Kim, et al. Enhancing llms’ clinical reasoning with real-world data from a nationwide sepsis registry.arXiv preprint arXiv:2505.02722, 2025

    2025 arXiv

  14. [22]

    Med-r1: Reinforce- ment learning for generalizable medical reasoning in vision-language models.arXiv preprint arXiv:2503.13939, 2025

    Yuxiang Lai, Jike Zhong, Ming Li, Shitian Zhao, and Xiaofeng Yang. Med-r1: Reinforce- ment learning for generalizable medical reasoning in vision-language models.arXiv preprint arXiv:2503.13939, 2025

    2025

  15. [23]

    Clinicalgpt-r1: Pushing reasoning capability of generalist disease diagnosis with large language model.arXiv preprint arXiv:2504.09421, 2025

    Wuyang Lan, Wenzheng Wang, Changwei Ji, Guoxing Yang, Yongbo Zhang, Xiaohong Liu, Song Wu, and Guangyu Wang. Clinicalgpt-r1: Pushing reasoning capability of generalist disease diagnosis with large language model.arXiv preprint arXiv:2504.09421, 2025

    2025 arXiv

  16. [24]

    Rlaif vs

    Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback, 2024

    2024

  17. [25]

    A scoping review of using large language models (llms) to investigate electronic health records (ehrs).arXiv preprint arXiv:2405.03066, 2024

    Lingyao Li, Jiayan Zhou, Zhenxiang Gao, Wenyue Hua, Lizhou Fan, Huizi Yu, Loni Hagen, Yongfeng Zhang, Themistocles L Assimes, Libby Hemphill, et al. A scoping review of using large language models (llms) to investigate electronic health records (ehrs).arXiv preprint arXiv:2405...

    2024 arXiv

  18. [26]

    Cls-rl: Image classifica- tion with rule-based reinforcement learning.arXiv preprint arXiv:2503.16188, 2025

    Ming Li, Shitian Zhao, Jike Zhong, Yuxiang Lai, and Kaipeng Zhang. Cls-rl: Image classifica- tion with rule-based reinforcement learning.arXiv preprint arXiv:2503.16188, 2025

    2025

  19. [27]

    Rec-r1: Bridging generative large language mod- els and user-centric recommendation systems via reinforcement learning.arXiv preprint arXiv:2503.24289, 2025

    Jiacheng Lin, Tian Wang, and Kun Qian. Rec-r1: Bridging generative large language mod- els and user-centric recommendation systems via reinforcement learning.arXiv preprint arXiv:2503.24289, 2025

    2025

  20. [28]

    Panacea: A foundation model for clinical trial search, summarization, design, and recruitment.medRxiv, pages 2024–06, 2024

    Jiacheng Lin, Hanwen Xu, Zifeng Wang, Sheng Wang, and Jimeng Sun. Panacea: A foundation model for clinical trial search, summarization, design, and recruitment.medRxiv, pages 2024–06, 2024

    2024

  21. [29]

    Pisces: A cross-modal contrastive learning approach to synergistic drug combination prediction

    Jiacheng Lin, Hanwen Xu, Addie Woicik, and Jianzhu Ma. Pisces: A cross-modal contrastive learning approach to synergistic drug combination prediction. InResearch in Computational Molecular Biology, page 268, 2023

    2023

  22. [30]

    Reft: Reasoning with reinforced fine-tuning.arXiv preprint arXiv:2401.08967, 3, 2024

    Trung Quoc Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reasoning with reinforced fine-tuning.arXiv preprint arXiv:2401.08967, 3, 2024

    2024 arXiv

  23. [31]

    Exploring the limit of outcome reward for learning mathematical reasoning.arXiv preprint arXiv:2502.06781, 2025

    Chengqi Lyu, Songyang Gao, Yuzhe Gu, Wenwei Zhang, Jianfei Gao, Kuikun Liu, Ziyi Wang, Shuaibin Li, Qian Zhao, Haian Huang, et al. Exploring the limit of outcome reward for learning mathematical reasoning.arXiv preprint arXiv:2502.06781, 2025. 11

    2025 arXiv

  24. [32]

    Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning.arXiv preprint arXiv:2503.07365, 2025

    Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Tiancheng Han, Botian Shi, Wenhai Wang, Junjun He, et al. Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning.arXiv preprint arXiv:2503.07365, 2025

    2025 arXiv

  25. [33]

    Can generalist foundation models outcompete special-purpose tuning? case study in medicine.Medicine, 84(88.3):77–3, 2023

    Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolo Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu, et al. Can generalist foundation models outcompete special-purpose tuning? case study in medicine.Medicine, 84(88.3):77–3, 2023

    2023

  26. [34]

    Openai o3-mini

    OpenAI. Openai o3-mini. Accessed: 2025-05-01

    2025

  27. [35]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022

  28. [36]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730...

    2022

  29. [37]

    Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. InConference on health, inference, and learning, pages 248–260. PMLR, 2022

    2022

  30. [38]

    Medvlm-r1: Incentivizing medical reasoning capability of vision-language models (vlms) via reinforcement learning.arXiv preprint arXiv:2502.19634, 2025

    Jiazhen Pan, Che Liu, Junde Wu, Fenglin Liu, Jiayuan Zhu, Hongwei Bran Li, Chen Chen, Cheng Ouyang, and Daniel Rueckert. Medvlm-r1: Incentivizing medical reasoning capability of vision-language models (vlms) via reinforcement learning.arXiv preprint arXiv:2502.19634, 2025

    2025 arXiv

  31. [39]

    Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl.arXiv preprint arXiv:2503.07536, 2025

    Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl.arXiv preprint arXiv:2503.07536, 2025

    2025 arXiv

  32. [40]

    Open-medical-r1: How to choose data for rlvr training at medicine domain.arXiv preprint arXiv:2504.13950, 2025

    Zhongxi Qiu, Zhang Zhang, Yan Hu, Heng Li, and Jiang Liu. Open-medical-r1: How to choose data for rlvr training at medicine domain.arXiv preprint arXiv:2504.13950, 2025

    2025 arXiv

  33. [41]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2024

    2024

  34. [42]

    Overview of the trec 2021 clinical trials track

    Kirk Roberts, Dina Demner-Fushman, Ellen M V oorhees, Steven Bedrick, and Willian R Hersh. Overview of the trec 2021 clinical trials track. InProceedings of the Thirtieth Text REtrieval Conference (TREC 2021), 2021

    2021

  35. [43]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  36. [44]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  37. [45]

    Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256, 2024

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256, 2024

    2024 arXiv

  38. [46]

    Large language models encode clinical knowledge.Nature, 620(7972):172–180, 2023

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge.Nature, 620(7972):172–180, 2023

    2023

  39. [47]

    Toward expert-level medical question answering with large language models.Nature Medicine, pages 1–8, 2025

    Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, et al. Toward expert-level medical question answering with large language models.Nature Medicine, pages 1–8, 2025. 12

    2025

  40. [48]

    R1-searcher: Incentivizing the search capability in llms via reinforcement learning.arXiv preprint arXiv:2503.05592, 2025

    Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. R1-searcher: Incentivizing the search capability in llms via reinforcement learning.arXiv preprint arXiv:2503.05592, 2025

    2025 arXiv

  41. [49]

    Gmai-vl-r1: Harnessing reinforcement learning for multimodal medical reasoning.arXiv preprint arXiv:2504.01886, 2025

    Yanzhou Su, Tianbin Li, Jiyao Liu, Chenglong Ma, Junzhi Ning, Cheng Tang, Sibo Ju, Jin Ye, Pengcheng Chen, Ming Hu, et al. Gmai-vl-r1: Harnessing reinforcement learning for multimodal medical reasoning.arXiv preprint arXiv:2504.01886, 2025

    2025 arXiv

  42. [50]

    Expanding rl with verifiable rewards across diverse domains.arXiv preprint arXiv:2503.23829, 2025

    Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. Expanding rl with verifiable rewards across diverse domains.arXiv preprint arXiv:2503.23829, 2025

    2025 arXiv

  43. [51]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599, 2025

    2025 arXiv

  44. [52]

    Yet another ICU benchmark: A flexible multi-center framework for clinical ML

    Robin van de Water, Hendrik Nils Aurel Schmidt, Paul Elbers, Patrick Thoral, Bert Arnrich, and Patrick Rockenschaub. Yet another ICU benchmark: A flexible multi-center framework for clinical ML. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  45. [53]

    Drg-llama: tuning llama model to predict diagnosis-related group for hospitalized patients.npj Digital Medicine, 7(1):16, 2024

    Hanyin Wang, Chufan Gao, Christopher Dantona, Bryan Hull, and Jimeng Sun. Drg-llama: tuning llama model to predict diagnosis-related group for hospitalized patients.npj Digital Medicine, 7(1):16, 2024

    2024

  46. [54]

    Reinforcement learning for reasoning in large language models with one training example.arXiv preprint arXiv:2504.20571, 2025

    Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Lucas Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, et al. Reinforcement learning for reasoning in large language models with one training example.arXiv preprint arXiv:2504.20571, 2025

    2025 arXiv

  47. [55]

    Ehrshot: An ehr benchmark for few-shot evaluation of foundation models.Advances in Neural Information Processing Systems, 36:67125–67137, 2023

    Michael Wornow, Rahul Thapa, Ethan Steinberg, Jason Fries, and Nigam Shah. Ehrshot: An ehr benchmark for few-shot evaluation of foundation models.Advances in Neural Information Processing Systems, 36:67125–67137, 2023

    2023

  48. [56]

    The shaky foundations of clinical foundation models: A survey of large language models and foundation models for emrs.arXiv preprint arXiv:2303.12961, 2023

    Michael Wornow, Yizhe Xu, Rahul Thapa, Birju Patel, Ethan Steinberg, Scott Fleming, Michael A Pfeffer, Jason Fries, and Nigam H Shah. The shaky foundations of clinical foundation models: A survey of large language models and foundation models for emrs.arXiv preprint arXiv:2303...

    2023 arXiv

  49. [57]

    Pathvlm-r1: A reinforcement learning-driven reasoning model for pathology visual-language tasks.arXiv preprint arXiv:2504.09258, 2025

    Jianyu Wu, Hao Yang, Xinhua Zeng, Guibing He, Zhiyu Chen, Zihui Li, Xiaochuan Zhang, Yangyang Ma, Run Fang, and Yang Liu. Pathvlm-r1: A reinforcement learning-driven reasoning model for pathology visual-language tasks.arXiv preprint arXiv:2504.09258, 2025

    2025 arXiv

  50. [58]

    Sailing ai by the stars: A survey of learning from rewards in post-training and test-time scaling of large language models.arXiv preprint arXiv:2505.02686, 2025

    Xiaobao Wu. Sailing ai by the stars: A survey of learning from rewards in post-training and test-time scaling of large language models.arXiv preprint arXiv:2505.02686, 2025

    2025 arXiv

  51. [59]

    Instruction tuning large language models to understand electronic health records

    Zhenbang Wu, Anant Dadu, Michael Nalls, Faraz Faghri, and Jimeng Sun. Instruction tuning large language models to understand electronic health records. InThe Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024

    2024

  52. [60]

    Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning.arXiv preprint arXiv:2502.14768, 2025

    Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning.arXiv preprint arXiv:2502.14768, 2025

    2025 arXiv

  53. [61]

    Qwen3 technical report, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, et al. Qwen3 technical report, 2025

    2025

  54. [62]

    Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    2025 arXiv

  55. [63]

    Med-rlvr: Emerging medical reasoning from a 3b base model via reinforcement learning.arXiv preprint arXiv:2502.19655, 2025

    Sheng Zhang, Qianchu Liu, Guanghui Qin, Tristan Naumann, and Hoifung Poon. Med-rlvr: Emerging medical reasoning from a 3b base model via reinforcement learning.arXiv preprint arXiv:2502.19655, 2025

    2025 arXiv

  56. [64]

    Rank- r1: Enhancing reasoning in llm-based document rerankers via reinforcement learning.arXiv preprint arXiv:2503.06034, 2025

    Shengyao Zhuang, Xueguang Ma, Bevan Koopman, Jimmy Lin, and Guido Zuccon. Rank- r1: Enhancing reasoning in llm-based document rerankers via reinforcement learning.arXiv preprint arXiv:2503.06034, 2025. 13 A Ethics, Broader Impacts, and Limitations EthicsIn this study, we evalu...

    2025 arXiv

  57. [65]

    Details are provided in Section F

    We employ LLMs to assist in constructing supervised warm-up datasets for both the Patient-Trial Matching and EHRSHOT tasks. Details are provided in Section F

  58. [66]

    We include LLMs as baseline models for evaluation on all three benchmarks: MedCalc-Bench, TREC Clinical Trials, and EHRSHOT

  59. [67]

    See Section F.3.4

    We use LLMs as evaluators to compare the quality of model-generated reasoning chains in the EHRSHOT Clinical Event Prediction task. See Section F.3.4. All usage was conducted responsibly and within the scope of model licenses and data-sharing agreements. D Related Work Reinfor...

  60. [68]

    Given a dose of Drug A, what is the equivalent dose of Drug B?

    or multimodal clinical VQA [22]. LLMs for Clinical Reasoning.LLMs are increasingly being explored for a range of clinical tasks, including summarization, question answering, and diagnosis [46, 33]. Many of these systems rely on prompt engineering or supervised fine-tuning usin...

    2021

  61. [69]

    Irrelevant (patient does not have sufficient information to qualify for the trial)

  62. [70]

    Excluded (patient meets inclusion criteria, but is excluded on the grounds of the trial’s exclusion criteria); and

  63. [71]

    Key Considerations: - Carefully **evaluate each inclusion and exclusion criterion individually**

    Eligible (patient meets inclusion criteria and exclusion criteria do not apply). Key Considerations: - Carefully **evaluate each inclusion and exclusion criterion individually**. - For each criterion, determine whether the patient **clearly satisfies**, **clearly violates**, o...

  64. [72]

    **Patient and Clincial Trial Overview**: Go over the key information in the patient’s EHR context and the clinical trial criteria, with the **Key Considerations** from the task description in mind

  65. [74]

    The reasoning should be comprehensive, medically sound, and clearly explain how the patient’s information leads to the predicted outcome

    **Conclusion**: Summarize the reasoning and state the prediction without mentioning the ground truth label. The reasoning should be comprehensive, medically sound, and clearly explain how the patient’s information leads to the predicted outcome. **Important Notes:** - **Do not...

  66. [75]

    **Patient and Clincial Trial Overview**: [YOUR OUTPUT]

  67. [77]

    Very Confident

    **Conclusion**: [YOUR OUTPUT] # Confidence # [CONFIDENCE (choose one: "Very Confident", "Confident", "Neutral", "Not Confident", "Very Not Confident")] 19 EHRSHOT defines 15 tasks, broadly categorized into the following four groups: (1) Operational Outcomes, (2) Anticipating L...

    2020

  68. [78]

    **Patient Overview**: Go over the key information in the patient’s EHR context

  69. [79]

    **Reasoning Towards Prediction**: Integrate the above information to logically reason towards the predicted outcome

  70. [80]

    The reasoning should be comprehensive, medically sound, and clearly explain how the patient’s information leads to the predicted outcome

    **Conclusion**: Summarize the reasoning and state the prediction without mentioning the ground truth label. The reasoning should be comprehensive, medically sound, and clearly explain how the patient’s information leads to the predicted outcome. **Important Notes:** - **Do not...

  71. [81]

    **Patient Overview**: [YOUR OUTPUT]

  72. [82]

    **Reasoning Towards Prediction**: [YOUR OUTPUT]

  73. [83]

    Very Confident

    **Conclusion**: [YOUR OUTPUT] # Confidence # [CONFIDENCE (choose one: "Very Confident", "Confident", "Neutral", "Not Confident", "Very Not Confident")] F.3.4 Assessing Reasoning Quality via GPT-4o Evaluation To assess the quality of the generated reasoning chains beyond accura...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.