Pith. sign in

REVIEW 4 major objections 5 minor 6 cited by

Beyond Distillation: Pushing the Limits of Medical LLM Reasoning with Minimalist Rule-Based RL

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Reasoning in a medical LLM can be induced by simple right-or-wrong reinforcement on multiple-choice questions, without supervised fine-tuning on distilled chain-of-thought data.

desk verdict Solid empirical results on rule-based RL for medical QA, but the 'reasoning emerges' claim is not supported without a base-model control, and the abstract oversells it. read the letter →

arxiv 2505.17952 v1 pith:PMI7K22Q submitted 2025-05-23 cs.CL cs.AI

Che Liu , Haozhe Wang , Jiazhen Pan , Zhongwei Wan , Yong Dai , Fangzhen Lin , Wenjia Bai , Daniel Rueckert
show 1 more author
Rossella Arcucci
This is my paper
classification cs.CLcs.AI
keywords medicalLLMrule-basedreinforcementlearningchain-of-thoughtmultiple-choiceQAemergentreasoningdatainformativenessGRPO
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that a medical language model can learn to reason from nothing but right-or-wrong feedback on public multiple-choice exam questions, with no supervised fine-tuning on step-by-step explanations and no reasoning traces distilled from a closed-source model. The authors build AlphaMed, train it with a minimalist reinforcement-learning reward that only checks whether the final boxed letter matches the answer key, and report state-of-the-art scores on six medical QA benchmarks. The point that matters is economic: if correct, high-performing clinical reasoning no longer requires the costly distillation or annotation of chain-of-thought data. The paper also argues that the quality, difficulty mix, and informativeness of the training questions, not just their quantity, determine whether such reasoning emerges.

What carries the argument

The engine is group-relative policy optimization (GRPO), a policy-gradient update that normalizes rewards within a batch of candidate answers, paired with a rule-based reward: each question generates eight candidate answers, a reward of 1 is given when the response ends with a correctly formatted boxed answer and 0 otherwise, and advantages are normalized within the group to drive the policy update. The supporting mechanism is a data-curation procedure that labels every training question by difficulty level (L1 to L6) based on how often a base Llama-3.1-8B-Instruct model answers it correctly across five tries, and uses question length as a proxy for informativeness. This lets the authors construct training sets balanced by difficulty and compare easy, medium, and hard data, isolating which question properties cause reasoning to appear.

What would settle it

Run the untrained Llama-3.1-8B-Instruct backbone on the same six evaluation benchmarks with the same "reason step by step" prompt and inspect its outputs. If the base model already emits Step 1/Step 2 traces at a comparable rate, or if it scores within a few points of AlphaMed without any training, then the claim that reward-only RL produced the reasoning collapses; a cleaner test is to train on the same questions with randomly shuffled answer keys and see whether accuracy gains and reasoning traces persist.

Watch

Extended reading notes

Core claim

The central discovery claimed is that step-by-step reasoning, visible as "Step 1, Step 2" traces in the model's answers, emerges as a by-product of reinforcement learning whose only reward is a binary check of the final answer against the ground truth. AlphaMed, trained this way on roughly 19,000 questions from MedQA and MedMCQA, outperforms medical models trained with supervised fine-tuning on distilled chain-of-thought data and reinforcement learning with learned reward models on all six benchmarks, and at the 70B scale it is reported to surpass GPT-4o, Claude-3.5-Sonnet, and DeepSeek-V3 on the hardest benchmark, MedXpert. The authors interpret this as evidence that reasoning capability can be incentivized rather than taught, and that the content of the training set, especially its informativeness and difficulty balance, is the main lever.

Load-bearing premise

The load-bearing premise is that the base Llama-3.1-8B-Instruct model does not already produce step-by-step reasoning under the training and evaluation prompts, so the traces and accuracy gains the paper shows can be attributed to reinforcement learning rather than to behavior the model had before training.

Editorial extensions

If this is right

  • Expensive SFT on distilled CoT data may be unnecessary for medical multiple-choice reasoning; a right-or-wrong signal suffices.
  • Curating training data by informativeness and difficulty becomes as important as model or algorithm choice, since low-quality data can actively hurt RL training.
  • Mixing difficulty levels, rather than training only on hard questions, is needed for generalizable gains across benchmarks.
  • High scores on existing medical QA benchmarks can be reached with relatively easy training data, which the paper reads as a signal that current benchmarks underestimate what reasoning models can do and need harder, reasoning-oriented items.
  • The same rule-based RL recipe transfers to other backbones, including smaller Qwen2.5 models, suggesting the effect is not tied to one base model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the strongest form of the claim, that the observed step-by-step traces emerged from RL, would need a control in which the base Llama-3.1-8B-Instruct is prompted and evaluated under identical conditions before training; the paper does not provide that baseline.
  • Editorial inference: using question length as a proxy for informativeness risks conflating verbosity with educational value; a direct test would hold question length fixed while varying reasoning demands.
  • Editorial inference: if benchmark gains come largely from answer memorization or format adaptation rather than new reasoning, the same recipe should be tried on open-ended clinical questions with human grading.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces AlphaMed, a medical LLM trained with minimalist rule-based reinforcement learning (GRPO) using a binary reward on public multiple-choice medical QA datasets (MedQA, MedMCQA, and pilot subsets from PubMedQA), initialized from Llama-3.1-8B/70B-Instruct. The authors report state-of-the-art accuracy on six medical QA benchmarks, including out-of-domain ones, and claim that step-by-step reasoning 'emerges' purely through RL without SFT on distilled chain-of-thought data. A data-centric analysis examines how dataset quantity, diversity, difficulty, and a proposed 'informativeness' proxy (question length) affect reasoning performance. The paper also applies the pipeline to Qwen backbones and provides qualitative examples of step-by-step outputs.

Significance. If the central claim is substantiated, the work would be a significant cost-reduction result for medical LLM reasoning: it would show that expensive distilled CoT data and SFT are unnecessary, and that a simple accuracy reward on public MC-QA data can induce interpretable step-by-step behavior. The paper has notable strengths: a clean in-domain/OOD split for the final models, use of purely public datasets, a transparent binary reward (Eq. 2), extensive pilot experiments across difficulty levels, and a promise to release code and weights. The data-centric findings on informativeness and difficulty mixing are potentially useful to practitioners. However, the 'emergence' claim is not yet established because the paper never shows what the base instruction-tuned model already produces under the same prompt, and the difficulty/informativeness analyses contain circular or post-hoc elements.

major comments (4)
  1. [Section 5.1, Section 5.2, Figs. 11-13] The headline claim that reasoning 'emerges purely through reinforcement learning' lacks a control on the base model's pre-existing chain-of-thought behavior. The paper's own difficulty annotation in Section 5.1 uses Llama-3.1-8B-Instruct with the prompt 'Please reason step by step...', and the base model is an instruction-tuned assistant, so it likely already emits multi-step reasoning traces under the training/evaluation prompt. Without reporting base-model outputs under the exact prompt, decoding settings, and answer-format requirements used in RL and evaluation, the step-by-step traces in Figs. 11-13 cannot be attributed to RL, and the benchmark gains could partly reflect pre-existing reasoning plus learning of the boxed-answer format. Please provide a quantitative comparison of the base model's outputs (e.g., frequency and quality of CoT traces, formatting compliance, and accuracy) before and after training. In addition, the abstract's 'without relying on SFT' is too strong because the initialization is an instruction-tuned model; the defensible claim is 'without SFT on distilled medical CoT.'
  2. [Table 1, Figs. 1-6]
  3. [Section 5.1, Finding 1.2, Fig. 5]
  4. [Fig. 8 and Section 5.5]
minor comments (5)
  1. [Section 5.5]
  2. [Section 5.1]
  3. [Finding 3.2]
  4. [Eq. (2) and Section 4.1]
  5. [Table 1]

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the RL reward, the benchmark evaluations, and the claimed emergent reasoning are not equal by construction; the main weaknesses are missing controls and post hoc data selection, which are validity concerns rather than circular reductions.

full rationale

The paper's central derivation is not circular. The rule-based reward (Eq. 2) rewards only the correctness of the final boxed answer, while the headline results are evaluated on six medical QA benchmarks, several of which (PubMedQA, MMLU-ProM, GPQA-M, MedXpert) are held out from the final training set. Optimizing GRPO (Eq. 1) with this binary reward does not by construction guarantee out-of-domain accuracy, nor does it force the step-by-step traces shown in Figs. 11-13; those traces are an empirical observation. The difficulty labels in Section 5.1 are computed with Llama-3.1-8B-Instruct, the same backbone later trained, and the 'informativeness' proxy (question length) is inferred from the same pilot experiments that motivate final data selection; these are post hoc and selection-on-results choices that could weaken generalization claims, but they do not make the reported benchmark performance equal to the training inputs by construction. The self-citations ([16], [18], [23]) are used for methodology and context, not as a load-bearing uniqueness or existence argument. The absence of a base-model control for step-by-step output is a missing-control validity issue, not a circularity: the claimed emergence is underdetermined, but the paper does not define the target result in terms of the training signal. No equation-level reduction or fitted-parameter-renamed-as-prediction was found.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim does not introduce new physical or conceptual entities. The free parameters are data-selection choices and the difficulty-labeling scheme.

free parameters (3)
  • Pilot subset size per difficulty level = 200
    Each of the three pilot subsets contains 200 samples per difficulty level (L1-L6), chosen by hand; no sensitivity analysis is reported.
  • Final MedMCQA sampling count per difficulty level = 1,600
    The final training set samples 1,600 QA pairs per difficulty level from MedMCQA, matched to MedQA's scale by hand; this choice affects the reported results.
  • Difficulty thresholds (five-generation accuracy) = L1-L6 based on 0,1,2,3,4,5 correct out of 5
    Questions are labeled by the accuracy of Llama-3.1-8B-Instruct over five generations; the bin boundaries are ad hoc and depend on the base model's own behavior.
assumptions (3)
  • domain assumption Binary correctness of the final boxed answer is a valid reward for eliciting reasoning.
    Eq. 2 assigns reward solely on the formatting and correctness of the final answer; the paper assumes this signal shapes reasoning rather than only answer memorization.
  • domain assumption MedQA and MedMCQA training splits do not overlap with the test splits of the six evaluation benchmarks.
    The paper treats MedQA/MedMCQA as in-domain and PubMedQA, MMLU-ProM, GPQA-M, MedXpert as OOD; contamination is not tested.
  • ad hoc to paper Question length is a practical proxy for dataset informativeness.
    Finding 1.2 elevates question length to a proxy for informativeness post hoc, after observing that MedQA (longest) leads to the best performance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Distillation: Pushing the Limits of Medical LLM Reasoning with Minimalist Rule-Based RL." pith.science (2026). https://pith.science/paper/PMI7K22Q

@misc{pith2026250517952,
  author       = {Pith},
  title        = {Pith review of: Beyond Distillation: Pushing the Limits of Medical LLM Reasoning with Minimalist Rule-Based RL},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PMI7K22Q}},
  note         = {Machine review of arXiv:2505.17952}
}
read the original abstract

Improving performance on complex tasks and enabling interpretable decision making in large language models (LLMs), especially for clinical applications, requires effective reasoning. Yet this remains challenging without supervised fine-tuning (SFT) on costly chain-of-thought (CoT) data distilled from closed-source models (e.g., GPT-4o). In this work, we present AlphaMed, the first medical LLM to show that reasoning capability can emerge purely through reinforcement learning (RL), using minimalist rule-based rewards on public multiple-choice QA datasets, without relying on SFT or distilled CoT data. AlphaMed achieves state-of-the-art results on six medical QA benchmarks, outperforming models trained with conventional SFT+RL pipelines. On challenging benchmarks (e.g., MedXpert), AlphaMed even surpasses larger or closed-source models such as DeepSeek-V3-671B and Claude-3.5-Sonnet. To understand the factors behind this success, we conduct a comprehensive data-centric analysis guided by three questions: (i) Can minimalist rule-based RL incentivize reasoning without distilled CoT supervision? (ii) How do dataset quantity and diversity impact reasoning? (iii) How does question difficulty shape the emergence and generalization of reasoning? Our findings show that dataset informativeness is a key driver of reasoning performance, and that minimalist RL on informative, multiple-choice QA data is effective at inducing reasoning without CoT supervision. We also observe divergent trends across benchmarks, underscoring limitations in current evaluation and the need for more challenging, reasoning-oriented medical QA benchmarks.

Figures

Figures reproduced from arXiv: 2505.17952 by the authors.

Figure 1
Figure 1. Performance comparison on six medical QA benchmarks. Our models are initialized with Llama3.1-8B-Instruct [45] and trained using minimalist rule-based RL on one of three balanced subsets: MedQA-Sub, MedMCQA-Sub, or PubMedQA-Sub (shown as blue, green, and orange bars, respectively). Despite using only 1,200 examples per subset, all variants of our model achieve substantial improvements over the base Llama3.1-8B-Instr… view at source ↗
Figure 2
Figure 2. Dataset analysis and training dynamics. Left: Ratio of effective queries over training steps; each curve corresponds to models trained on a specific subset. Middle: Training reward per step for models trained on each subset. Right: Distribution of question lengths (number of tokens) in MedQA, MedMCQA, and PubMedQA [43, 44, 34]. 5.2 RQ1: Can Minimalist RL Incentivize Medical Reasoning Without Distilled-CoT SFT? To in… view at source ↗
Figure 4
Figure 4. Effect of data diversity. Aver￾age accuracy across six medical QA bench￾marks when models are trained individu￾ally on single or combined subsets. Adding MedMCQA-Sub to MedQA-Sub boosts perfor￾mance, while further adding PubMedQA-Sub reduces it, suggesting that less informative data can negate the benefits of increased di￾versity. PubMedQA-Sub. To understand this variation, we explore the training dynamics of models… view at source ↗
Figures from the paper (8 more)
Figure 5
Figure 5. Figure 5: Performance on six benchmarks when training on subsets with increasing difficulty levels [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Performance on six benchmarks when training with distinct difficulty groups: easy [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Comparison of AlphaMed(8B) with prior models on MMLU-ProM [35] and MedXpert [37]. Despite its smaller scale and use of minimalist RL, AlphaMed(8B) outper￾forms the larger model QwQ-32B [48] and other baselines [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 9
Figure 9. Figure 9: Performance comparison across six medical QA benchmarks. AlphaMed(7B) is initialized [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Performance comparison across six medical QA benchmarks. AlphaMed(3B) is initial [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Question and answer pair for Case 1. Cyan text highlights the final predicted choices. Green highlight are used to emphasize reasoning steps and key clinically key information. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: Question and answer pair for Case 2. Cyan text highlights the final predicted choices. Green highlight are used to emphasize reasoning steps and key clinically key information. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Question and answer pair for Case 3. Cyan text highlights the final predicted choices. Green highlight are used to emphasize reasoning steps and key clinically key information. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Credit Cards, Confusion, Computation, and Consequences: What Can We Uncover About Language Model Reasoning?

    cs.CL 2026-07 conditional novelty 6.5 of 10

    CreditCardQA shows LLMs err mainly on credit-card contractual conditions and comparisons, not arithmetic, with Program-of-Thought narrowing open–closed model gaps.

  2. AdaThink-Med: Optimizing Inference-Time Compute for Medical Reasoning via Uncertainty Quantification

    cs.CL 2025-09 conditional novelty 6.0 of 10

    An entropy-guided reward that shortens answers to easy, confident questions and extends reasoning on hard ones reduces medical QA tokens by 4.7x to 6.4x with minimal accuracy loss.

  3. Psyche-R1: Towards Reliable Psychological LLMs through Unified Empathy, Expertise, and Reasoning

    cs.CL 2025-08 conditional novelty 6.0 of 10

    A 7B Chinese psychological LLM trained with curated reasoning questions and empathetic dialogues via SFT plus GRPO approaches DeepSeek-R1's exam accuracy at a fraction of the size.

  4. Med-R$^3$: Enhancing Medical Retrieval-Augmented Reasoning of LLMs via Progressive Reinforcement Learning

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A progressive three-stage RL framework with medical-specific rewards improves retrieval-augmented reasoning on medical QA benchmarks, reportedly surpassing GPT-4o-mini with an 8B model.

  5. InfiMed-ORBIT: Aligning LLMs on Open-Ended Complex Tasks via Rubric-Based Incremental Training

    cs.CL 2025-10 conditional novelty 5.0 of 10

    Rubric-based incremental RL with LLM-generated case-specific checklists lifts Qwen3-4B's HealthBench-Hard score from 7.0 to 27.5 with 2k samples, and improves InfoBench instruction-following from 42.0 to 82.9.

  6. Baichuan-M2: Scaling Medical Capability with Large Verifier System

    cs.LG 2025-09 conditional novelty 5.0 of 10

    Baichuan-M2, a 32B medical LLM trained with a patient simulator and a clinical rubric generator as RL verifiers, reports state-of-the-art HealthBench scores (60.1 overall, 34.7 hard), ahead of all open-source models.

Reference graph

Works this paper leans on

50 extracted references · 15 canonical work pages · cited by 6 Pith papers

  1. [1]

    O1 replication journey: A strategic progress report–part 1,

    Y . Qin, X. Li, H. Zou, Y . Liu, S. Xia, Z. Huang, Y . Ye, W. Yuan, H. Liu, Y . Liet al., “O1 replication journey: A strategic progress report–part 1,”arXiv preprint arXiv:2410.18982, 2024

    arXiv 2024

  2. [2]

    Scaling of search and learning: A roadmap to reproduce o1 from reinforcement learning perspective,

    Z. Zeng, Q. Cheng, Z. Yin, B. Wang, S. Li, Y . Zhou, Q. Guo, X. Huang, and X. Qiu, “Scaling of search and learning: A roadmap to reproduce o1 from reinforcement learning perspective,”arXiv preprint arXiv:2412.14135, 2024

    arXiv 2024

  3. [3]

    Openr: An open source framework for advanced reasoning with large language models,

    J. Wang, M. Fang, Z. Wan, M. Wen, J. Zhu, A. Liu, Z. Gong, Y . Song, L. Chen, L. M. Niet al., “Openr: An open source framework for advanced reasoning with large language models,”arXiv preprint arXiv:2410.09671, 2024

    arXiv 2024

  4. [4]

    Deliberative alignment: Reasoning enables safer language models,

    M. Y . Guan, M. Joglekar, E. Wallace, S. Jain, B. Barak, A. Heylar, R. Dias, A. Vallone, H. Ren, J. Wei, H. W. Chung, S. Toyer, J. Heidecke, A. Beutel, and A. Glaese, “Deliberative alignment: Reasoning enables safer language models,”OpenAI Blog, 2024. [Online]. Available: https://openai.com/index/deliberative-alignment/

    work page 2024

  5. [5]

    Capabilities of gemini models in medicine,

    K. Saab, T. Tu, W.-H. Weng, R. Tanno, D. Stutz, E. Wulczyn, F. Zhang, T. Strother, C. Park, E. Vedadi et al., “Capabilities of gemini models in medicine,”arXiv preprint arXiv:2404.18416, 2024

    arXiv 2024

  6. [6]

    Cod, towards an interpretable medical agent using chain of diagnosis,

    J. Chen, C. Gui, A. Gao, K. Ji, X. Wang, X. Wan, and B. Wang, “Cod, towards an interpretable medical agent using chain of diagnosis,”arXiv preprint arXiv:2407.13301, 2024

    arXiv 2024

  7. [7]

    Thinking and reasoning in medicine,

    V . L. Patel, J. F. Arocha, and J. Zhang, “Thinking and reasoning in medicine,”The Cambridge handbook of thinking and reasoning, vol. 14, pp. 727–750, 2005

    work page 2005

  8. [8]

    Towards next- generation medical agent: How o1 is reshaping decision-making in medical scenarios,

    S. Xu, Y . Zhou, Z. Liu, Z. Wu, T. Zhong, H. Zhao, Y . Li, H. Jiang, Y . Pan, J. Chenet al., “Towards next- generation medical agent: How o1 is reshaping decision-making in medical scenarios,”arXiv preprint arXiv:2411.14461, 2024

    arXiv 2024

Show all 50 references
  1. [9]

    Openai o1-preview vs. chatgpt in healthcare: A new frontier in medical ai reasoning,

    M.-H. Temsah, A. Jamal, K. Alhasan, A. A. Temsah, and K. H. Malki, “Openai o1-preview vs. chatgpt in healthcare: A new frontier in medical ai reasoning,”Cureus, vol. 16, no. 10, p. e70640, 2024

    2024

  2. [10]

    A preliminary study of o1 in medicine: Are we closer to an ai doctor?

    Y . Xie, J. Wu, H. Tu, S. Yang, B. Zhao, Y . Zong, Q. Jin, C. Xie, and Y . Zhou, “A preliminary study of o1 in medicine: Are we closer to an ai doctor?”arXiv preprint arXiv:2409.15277, 2024

    2024 arXiv

  3. [11]

    Huatuogpt- ii, one-stage training for medical adaption of llms,

    J. Chen, X. Wang, K. Ji, A. Gao, F. Jiang, S. Chen, H. Zhang, D. Song, W. Xie, C. Konget al., “Huatuogpt- ii, one-stage training for medical adaption of llms,”arXiv preprint arXiv:2311.09774, 2023

    2023 arXiv

  4. [12]

    Chain- of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V . Le, and D. Zhou, “Chain- of-thought prompting elicits reasoning in large language models,”Advances in Neural Information Pro- cessing Systems, vol. 35, pp. 24 824–24 837, 2022

    2022

  5. [13]

    Scaling instruction-finetuned language models,

    H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y . Tay, W. Fedus, Y . Li, X. Wang, J. Weiet al., “Scaling instruction-finetuned language models,”arXiv preprint arXiv:2210.11416, 2022

    2022 arXiv

  6. [14]

    Star: Bootstrapping reasoning with reasoning,

    E. Zelikman, Y . Wu, J. Mu, and N. D. Goodman, “Star: Bootstrapping reasoning with reasoning,”Ad- vances in Neural Information Processing Systems, vol. 35, pp. 15 476–15 488, 2022

    2022

  7. [15]

    Sft memorizes, rl generalizes: A comparative study of foundation model post-training,

    T. Chu, Y . Zhai, J. Yang, S. Tong, S. Xie, D. Schuurmans, Q. V . Le, S. Levine, and Y . Ma, “Sft memorizes, rl generalizes: A comparative study of foundation model post-training,”arXiv preprint arXiv:2501.17161, 2025

    2025 arXiv

  8. [16]

    Vl-rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning,

    H. Wang, C. Qu, Z. Huang, W. Chu, F. Lin, and W. Chen, “Vl-rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning,”arXiv preprint arXiv:2504.08837, 2025

    2025 arXiv

  9. [17]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Rayet al., “Training language models to follow instructions with human feedback,”Advances in Neural Information Processing Systems, vol. 35, pp. 27 730–27 744, 2022

    2022

  10. [18]

    Learning autonomous code integration for math language models,

    H. Wang, L. Li, C. Qu, F. Zhu, W. Xu, W. Chu, and F. Lin, “Learning autonomous code integration for math language models,”arXiv preprint arXiv:2502.00691, 2025

    2025 arXiv

  11. [19]

    Direct preference opti- mization: Your language model is secretly a reward model,

    R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference opti- mization: Your language model is secretly a reward model,”Advances in Neural Information Processing Systems, vol. 36, 2023

    2023

  12. [20]

    Openbiollm-70b: Advancing open-source biomedical llms with direct preference optimization,

    A. Ura, “Openbiollm-70b: Advancing open-source biomedical llms with direct preference optimization,” Hugging Face Blog, 2024, available at https://huggingface.co/blog/aaditya/openbiollm. 11

    2024

  13. [21]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Biet al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,”arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  14. [22]

    Acecoder: Acing coder rl via automated test-case synthesis,

    H. Zeng, D. Jiang, H. Wang, P. Nie, X. Chen, and W. Chen, “Acecoder: Acing coder rl via automated test-case synthesis,”arXiv preprint arXiv:2502.01718, 2025

    2025 arXiv

  15. [23]

    Medvlm-r1: Incentivizing medical reasoning capability of vision-language models (vlms) via reinforcement learning,

    J. Pan, C. Liu, J. Wu, F. Liu, J. Zhu, H. B. Li, C. Chen, C. Ouyang, and D. Rueckert, “Medvlm-r1: Incentivizing medical reasoning capability of vision-language models (vlms) via reinforcement learning,” arXiv preprint arXiv:2502.19634, 2025

    2025 arXiv

  16. [24]

    Huatuogpt, towards taming language model to be a doctor,

    H. Zhang, J. Chen, F. Jiang, F. Yu, Z. Chen, J. Li, G. Chen, X. Wu, Z. Zhang, Q. Xiaoet al., “Huatuogpt, towards taming language model to be a doctor,”arXiv preprint arXiv:2305.15075, 2023

    2023 arXiv

  17. [25]

    Biomistral: A collection of open-source pretrained large language models for medical domains,

    Y . Labrak, A. Bazoge, E. Morin, P.-A. Gourraud, M. Rouvier, and R. Dufour, “Biomistral: A collection of open-source pretrained large language models for medical domains,”arXiv preprint arXiv:2402.10373, 2024

    2024 arXiv

  18. [26]

    Continual pre-training of language models,

    Z. Ke, Y . Shao, H. Lin, T. Konishi, G. Kim, and B. Liu, “Continual pre-training of language models,” arXiv preprint arXiv:2302.03241, 2023

    2023 arXiv

  19. [27]

    Ultramedical: Building specialized generalists in biomedicine,

    K. Zhang, S. Zeng, E. Hua, N. Ding, Z.-R. Chen, Z. Ma, H. Li, G. Cui, B. Qi, X. Zhuet al., “Ultramedical: Building specialized generalists in biomedicine,”Advances in Neural Information Processing Systems, vol. 37, pp. 26 045–26 081, 2024

    2024

  20. [29]

    m1: Unleash the potential of test-time scaling for medical reasoning with large language models,

    X. Huang, J. Wu, H. Liu, X. Tang, and Y . Zhou, “m1: Unleash the potential of test-time scaling for medical reasoning with large language models,”arXiv preprint arXiv:2504.00869, 2025

    2025

  21. [30]

    Proximal policy optimization algo- rithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algo- rithms,”arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  22. [31]

    Hybridflow: A flexible and efficient rlhf framework,

    G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y . Peng, H. Lin, and C. Wu, “Hybridflow: A flexible and efficient rlhf framework,”arXiv preprint arXiv:2409.19256, 2024

    2024 arXiv

  23. [33]

    Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering,

    A. Pal, L. K. Umapathi, and M. Sankarasubbu, “Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering,” inConference on health, inference, and learning. PMLR, 2022, pp. 248–260

    2022

  24. [34]

    Pubmedqa: A dataset for biomedical research question answering,

    Q. Jin, B. Dhingra, Z. Liu, W. W. Cohen, and X. Lu, “Pubmedqa: A dataset for biomedical research question answering,”arXiv preprint arXiv:1909.06146, 2019

    1909 arXiv

  25. [35]

    Mmlu- pro: A more robust and challenging multi-task language understanding benchmark,

    Y . Wang, X. Ma, G. Zhang, Y . Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jianget al., “Mmlu- pro: A more robust and challenging multi-task language understanding benchmark,” inThe Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchma...

    2024

  26. [36]

    Gpqa: A graduate-level google-proof q&a benchmark,

    D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y . Pang, J. Dirani, J. Michael, and S. R. Bowman, “Gpqa: A graduate-level google-proof q&a benchmark,” inFirst Conference on Language Modeling, 2024

    2024

  27. [37]

    Medxpertqa: Bench- marking expert-level medical reasoning and understanding,

    Y . Zuo, S. Qu, Y . Li, Z. Chen, X. Zhu, E. Hua, K. Zhang, N. Ding, and B. Zhou, “Medxpertqa: Bench- marking expert-level medical reasoning and understanding,”arXiv preprint arXiv:2501.18362, 2025

    2025 arXiv

  28. [38]

    Medagents- bench: Benchmarking thinking models and agent frameworks for complex medical reasoning,

    X. Tang, D. Shao, J. Sohn, J. Chen, J. Zhang, J. Xiang, F. Wu, Y . Zhao, C. Wu, W. Shiet al., “Medagents- bench: Benchmarking thinking models and agent frameworks for complex medical reasoning,”arXiv preprint arXiv:2503.07459, 2025

    2025 arXiv

  29. [39]

    Mmlu- pro: A more robust and challenging multi-task language understanding benchmark,

    Y . Wang, X. Ma, G. Zhang, Y . Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jianget al., “Mmlu- pro: A more robust and challenging multi-task language understanding benchmark,”arXiv preprint arXiv:2406.01574, 2024

    2024 arXiv

  30. [40]

    Openbiollms: Advancing open-source large language models for healthcare and life sciences,

    M. S. A. Pal and M. Sankarasubbu, “Openbiollms: Advancing open-source large language models for healthcare and life sciences,” 2024. 12

    2024

  31. [41]

    Towards building multilin- gual language model for medicine,

    P. Qiu, C. Wu, X. Zhang, W. Lin, H. Wang, Y . Zhang, Y . Wang, and W. Xie, “Towards building multilin- gual language model for medicine,”Nature Communications, vol. 15, no. 1, p. 8384, 2024

    2024

  32. [42]

    Med42–evaluating fine-tuning strategies for medical llms: Full-parameter vs. parameter-efficient approaches,

    C. Christophe, P. K. Kanithi, P. Munjal, T. Raha, N. Hayat, R. Rajan, A. Al-Mahrooqi, A. Gupta, M. U. Salman, G. Gosalet al., “Med42–evaluating fine-tuning strategies for medical llms: Full-parameter vs. parameter-efficient approaches,”arXiv preprint arXiv:2404.14779, 2024

    2024 arXiv

  33. [43]

    What disease does this patient have? a large-scale open domain question answering dataset from medical exams,

    D. Jin, E. Pan, N. Oufattole, W.-H. Weng, H. Fang, and P. Szolovits, “What disease does this patient have? a large-scale open domain question answering dataset from medical exams,”Applied Sciences, vol. 11, no. 14, p. 6421, 2021

    2021

  34. [44]

    Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering,

    A. Pal, L. K. Umapathi, and M. Sankarasubbu, “Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering,” inConference on Health, Inference, and Learning. PMLR, 2022, pp. 248–260

    2022

  35. [45]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fanet al., “The llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  36. [46]

    Huatuogpt-o1, towards medical complex reasoning with llms,

    J. Chen, Z. Cai, K. Ji, X. Wang, W. Liu, R. Wang, J. Hou, and B. Wang, “Huatuogpt-o1, towards medical complex reasoning with llms,”arXiv preprint arXiv:2412.18925, 2024

    2024 arXiv

  37. [47]

    Gpt-4o system card,

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radfordet al., “Gpt-4o system card,”arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  38. [48]

    Qwq: Reflect deeply on the boundaries of the unknown,

    Q. Team, “Qwq: Reflect deeply on the boundaries of the unknown,” November 2024. [Online]. Available: https://qwenlm.github.io/blog/qwq-32b-preview/

    2024

  39. [49]

    The claude 3 model family: Opus, sonnet, haiku,

    Anthropic, “The claude 3 model family: Opus, sonnet, haiku,” https://www-cdn.anthropic.com/ de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf, 2024

    2024

  40. [50]

    Deepseek- v3 technical report,

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruanet al., “Deepseek- v3 technical report,”arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  41. [51]

    Gpqa: A graduate-level google-proof q&a benchmark,

    D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y . Pang, J. Dirani, J. Michael, and S. R. Bowman, “Gpqa: A graduate-level google-proof q&a benchmark,”arXiv preprint arXiv:2311.12022, 2023

    2023 arXiv

  42. [52]

    Qwen2.5: A party of foundation models,

    Q. Team, “Qwen2.5: A party of foundation models,” September 2024. [Online]. Available: https://qwenlm.github.io/blog/qwen2.5/ 13 Limitations and Future Work Although AlphaMed achieves impressive results on multiple-choice QA tasks, its capabilities remain constrained by the cl...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.