{"id":"084b8432-4714-4773-b3fd-48a6c48d6572","arxiv_id":"2608.08623","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A knowledge-guided reward framework combining formula verification with a hybrid soft-hard accuracy reward improves medical calculation accuracy for small language models on MedCalc-Bench.","lead":"This paper proposes a reward framework for training small language models to solve medical calculation problems, adding a formula-checking step and a combined strict-plus-gradual accuracy reward. The authors report that a 3B model trained this way beats larger open models on the MedCalc-Bench benchmark.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Knowledge reward depends on an unvalidated LLM judge; without gold-standard accuracy or bias analysis, the central mechanism is not established.","rationale":"The reader's weakest assumption points to the unvalidated knowledge verifier, and my reading of the paper reinforces that this is the most load-bearing concern. The paper's central contribution is not merely a score on MedCalc-Bench but a claim that knowledge-guided formula verification, combined with a hybrid soft-hard reward, drives the improvement. The final accuracy numbers could be correct while the mechanism attributed to them is wrong; the absence of any gold-standard evaluation of the frozen judge makes the mechanism unverifiable as reported. Ablations and final-policy formula-recall measurements are informative but do not close this gap. I find no need to manufacture additional objections: the tolerance-reward criticism, the RL algorithm comparison, and the generalization results are all secondary to this core issue. The reader's conditional verdict is the right level of confidence; the concern does not by itself invalidate the reported accuracies, but it does mean the framework's central mechanism should not be accepted without a direct verifier-calibration study. A concrete gold-label verifier evaluation, plus a causal retraining check with gold labels, would settle whether the concern actually lands.","tokens_in":19581,"tokens_out":5258,"duration_ms":62813,"concrete_test":"Build a gold-labeled verification set from MedCalc-Bench: for each sampled question, include the ground-truth formula as positive and several plausible wrong formulas as negatives (correct equation with wrong patient variables, wrong equation for the task, equation from a different calculator). Run the exact Qwen2.5-14B-Instruct verifier prompt used in training on roughly 300 positives and 300 negatives; compute overall accuracy and per-task precision/recall. If accuracy is below 95% or false-positive/false-negative rates differ materially across tasks, the knowledge reward is unreliable. Then rerun GRPO training for the 3B model with judge decisions replaced by gold labels on the same sampled rollouts and compare the final MedCalc-Bench score; a shift of more than about one point confirms the noisy judge is causally load-bearing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 (Eq. 3) delegates the entire knowledge signal to a frozen LLM judge (Qwen2.5-14B-Instruct) that returns a binary +1/−1 reward, but the paper never measures this judge's accuracy, precision, recall, or systematic bias against gold-standard formulas. The only supporting evidence is an ablation showing that removing the knowledge reward degrades performance by 4.49 points (Table 2) and Appendix F's GPT-4o-based formula-recall evaluation of the final policy. Neither validates the judge: the ablation shows the reward component matters, not that it is semantically correct; Appendix F evaluates the policy's output formulas, not the judge's decisions. Because the judge's output directly enters the GRPO advantage signal, systematic false positives would reward hallucinated formulas and false negatives would suppress valid ones. Figure 4's interpretation that the 32B verifier is 'too strict' is inference from final accuracy, not a measurement of verifier correctness. The paper's own Limitation section acknowledges the verifier may bottleneck on complex logic, but an acknowledgment is not a substitute for calibration. The central claim attributes the gain to knowledge-guided verification; that attribution is currently unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MEDCALC-R1, a two-stage SFT+RL framework for medical mathematical reasoning on MedCalc-Bench. The RL stage uses GRPO with a composite reward: a format reward, a knowledge verification reward from a frozen LLM judge (Qwen2.5-14B-Instruct) that checks whether the generated formula matches clinical guidelines, and an answer reward combining a hard tolerance interval with a soft exponential precision term. Experiments report that a 3B model achieves 51.34 average accuracy, outperforming zero-shot open-source baselines and SFT baselines. The authors attribute gains to the knowledge-guided and hybrid soft-hard reward design.","tokens_in":19857,"tokens_out":6504,"duration_ms":62926,"significance":"If the claims hold, the work would offer a practical recipe for RLVR with continuous outputs, combining process-level formula verification with safety-aware rewards. The two-stage design and the hybrid soft-hard answer reward are reasonable and the experimental gains over SFT are non-trivial. However, the core attribution to the knowledge verifier is currently unsupported because the verifier itself is never validated, and the missing tolerance-based RLVR baseline weakens the comparison with the standard approach.","major_comments":[{"comment":"The knowledge verifier's correctness is never evaluated. The ablation in Table 2 shows only that removing this reward component changes performance, which does not establish that the reward signal is semantically correct. Appendix F evaluates the final policy's formula recall using GPT-4o, not the judge's decisions. Provide a direct evaluation of the verifier against gold-standard formulas (e.g., precision, recall, and error analysis), and examine how judge errors affect the reward signal and training.","section":"Section 3.3, Eq. (3)"},{"comment":"The main experiments lack a direct baseline with a plain tolerance-based RLVR reward. The paper motivates the framework by the failure of tolerance-based rewards, but never trains such a model on MedCalc-Bench under the same SFT+GRPO pipeline. Add a baseline with R_answer set to the binary tolerance reward (same SFT init, same GRPO, same rollout settings) and report it in Table 1 (and ideally also in the ablations). Appendix C's Table 3 uses a small subset and a slightly different protocol; it does not substitute for the main comparison.","section":"Section 4.4, Table 1"},{"comment":"The \"w/o HyRe\" ablation removes both the hard and soft components together, so the individual contributions of the hard safety constraint and the soft precision reward are not isolated. Add ablations that remove only the hard component and only the soft component. Additionally, the reward weights alpha, beta, gamma are all set to 1.0 with no sensitivity analysis; since these are free parameters, please report at least a small sweep or justify the choice.","section":"Section 4.5, Table 2"},{"comment":"The per-category train task counts sum to 41, not 38 as stated in the \"Overall\" row, while the test sum is 57 as stated. This inconsistency is important because the number of unseen test tasks underpins the generalization claims. Please correct the table and clarify the exact split of unseen tasks, including how Figure 6 defines \"unseen tasks.\"","section":"Section 4.1, Table 4"},{"comment":"No error bars, confidence intervals, or multiple seeds are reported for any experiment. Given the stochastic nature of RL with a group size of 5, single-run results are not sufficient to support the accuracy differences claimed. Report means and standard deviations over at least 3 seeds for the main results and ablations.","section":"Section 4.4 and 4.5 (reproducibility)"}],"minor_comments":[{"comment":"The Introduction describes the verifier as an \"external solver,\" but Section 3.3 uses a frozen LLM judge; align the terminology to avoid overstating determinism.","section":"Section 1 vs Section 3.3"},{"comment":"The phrase \"Inequation-based tasks\" should be corrected to \"In equation-based tasks.\"","section":"Section 4.4"},{"comment":"The x-axis labels are missing and the \"unseen tasks\" subset is not clearly defined; specify which tasks are considered unseen and add axis labels.","section":"Figure 6"},{"comment":"Define the abbreviations \"KnoRe\" and \"HyRe\" in the caption for readability.","section":"Table 2 caption"},{"comment":"The x-axis label \"Sample /glyph1197umber\" contains a rendering artifact; it should read \"Sample Number.\"","section":"Figure 5"},{"comment":"Some rows have numbers concatenated without spacing (e.g., the DeepSeek-R1-Distill-Qwen-14B row); please format the table consistently.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper compares against zero-shot baselines and its own SFT, but not against other RL-trained medical calculators (e.g., Med-R1 or tool-based agents). The \"SOTA among open-weight models\" claim should be qualified accordingly. The work is a reasonable empirical contribution, but the core mechanism needs stronger validation before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi [name],\n\nQuick take: this paper deserves a serious look. It applies GRPO-based RLVR with a formula-verification reward and a hybrid soft-hard answer reward to MedCalc-Bench, and the empirical gains are real: the 3B model goes from 12.49 to 51.34 average accuracy, beating much larger open-weight models and nearly closing the gap with GPT-4o. The ablations are clean and show both reward components matter; the appendix on tolerance thresholds (Table 3, Figure 8) is a nice controlled demonstration that the hybrid reward helps stability.\n\nWhat's actually new is the specific combination for clinical calculations: forcing explicit formula generation, judging those formulas with a frozen LLM, and mixing a hard safety interval with a soft exponential precision reward. That's an extension of existing RLVR ideas, not a conceptual breakthrough, but the paper is honest about that.\n\nThe soft spots are the usual ones for this kind of paper. Most importantly, the knowledge verifier (Qwen2.5-14B-Instruct) is never validated. The paper knows this—the limitations section admits the verifier might bottleneck—but an acknowledgment isn't calibration. The ablation shows the reward component changes behavior, not that the judge's +1/−1 signal is semantically correct. A systematic evaluation against gold-standard formulas, or at least a bias analysis, would firm up the attribution. Second, there's no plain tolerance-based RLVR baseline in the main table. Appendix C partially fills this gap, but it's on a subset with a single epsilon. Third, no error bars or multiple seeds anywhere; for a 5-epoch RL run, that's a legitimate ask. And the train-task count in Table 4 (38) doesn't match the sum of its rows (41)—a minor reporting error, but the kind that makes a referee twitch.\n\nI disagree with the stress-test note's stronger reading: the final accuracy numbers don't collapse if the verifier is noisy; the method still works in the aggregate. But the mechanism story is indeed under-supported. That's a fixable weakness, not a fatal flaw.\n\nWho's this for: people working on reward design for RLVR, especially in safety-critical domains, and anyone building medical reasoning models. I'd cite it if I were in that space.\n\nRecommendation: send it to peer review. Require the verifier evaluation and error bars; the core empirical result likely survives.","headline":"MedCalc-R1 is a competent RLVR-for-medical-reasoning paper with a real empirical gain, but the knowledge-verifier reward is asserted more than validated.","tokens_in":20366,"tokens_out":3006,"would_cite":true,"duration_ms":29239,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A small model with formula-checked rewards can beat far larger open models on clinical calculations.","keywords":["medical mathematical reasoning","reinforcement learning with verifiable rewards","reward design","formula verification","soft-hard reward","MedCalc-Bench","GRPO"],"falsifier":"Take a gold-standard sample of formulas from MedCalc-Bench, label each as valid or invalid by clinical experts, run the Qwen2.5-14B-Instruct verifier on the same sample, and compare its binary verdicts with the expert labels; if agreement is low or biased toward rejecting valid formulas, the knowledge reward's contribution to MedCalc-R1's gains is not the mechanism the paper claims.","tokens_in":19422,"feed_emoji":"🧮","tokens_out":6813,"duration_ms":68242,"temperature":0.7,"pith_summary":"Reinforcement learning for clinical calculations normally rewards a prediction only when it falls inside a fixed tolerance window around the true number; the authors argue this creates a calibration-stability dilemma and can reward lucky approximations. Their proposed framework instead makes the model emit an explicit formula and a reasoning trace, verifies the formula with a frozen LLM judge, and scores the numeric answer with a hard safety-interval reward plus a continuous soft reward. On MedCalc-Bench, the trained 3-billion-parameter model scores 51.34, the best result among open-weight models and close to much larger closed-source systems. The thesis is that reward design, not scale, is what small models need for reliable clinical calculation.","feed_headline":"Formula-verified rewards lift 3B medical math model to open-weight top","feed_subtitle":"MedCalc-R1 3B scores 51.34 on MedCalc-Bench, beating 32B rivals and closing in on closed-source models.","key_machinery":"The load-bearing object is the reward composition $R=\\alpha R_f+\\beta R_k+\\gamma R_a$ in Eq. (2), with the knowledge reward $R_k$ and the hybrid answer reward $R_a$ doing the new work. $R_k$ uses a stronger frozen language model as an external judge to enforce that the generated formula is semantically aligned with medical guidelines, preventing formula hallucination and knowledge forgetting; $R_a$ combines a hard clinical safety threshold with a soft exponential reward so the optimization landscape is coarse-to-fine rather than sparsely binary. These rewards supply the group-relative advantages in GRPO, so the entire argument depends on whether this composite signal, not the base model's scale, drives the accuracy gains.","core_discovery":"MedCalc-R1 claims that a knowledge-guided hybrid reward makes a 1.5B or 3B model reason reliably about clinical calculations. Training starts with SFT on 9,765 MedCalc-Bench instances, then GRPO optimizes a weighted sum of three rewards: format reward for the required formula/reasoning/answer structure, knowledge reward $R_k\\in\\{1.0,-1.0\\}$ from a frozen Qwen2.5-14B-Instruct verifier that judges whether the formula matches the task's valid formula set, and answer reward $R_a=R_{\\mathrm{hard}}+R_{\\mathrm{soft}}$ where $R_{\\mathrm{hard}}$ is binary on the clinical interval $[L,U]$ and $R_{\\mathrm{soft}}=\\exp(-|v-v^*|/\\tau)$. The resulting MedCalc-R1 3B reaches 51.34 average accuracy on MedCalc-Bench, ahead of all open-weight baselines including Qwen2.5-32B-Instruct at 39.03, and ablations show the two reward components are complementary: removing both costs 11.74 points.","pith_inferences":["The same reward decomposition is a natural fit for other numerical safety-critical tasks, such as drug dose adjustments, lab-value flags, or ventilator settings, wherever a verifiable formula or rubric exists; the paper only tests clinical calculators.","Because $R_{\\mathrm{soft}}$ uses absolute error symmetrically, it implicitly assigns equal cost to over- and under-estimation; an asymmetric penalty aligned with clinical risk, such as different costs for overdose versus underdose, would be a straightforward extension the authors do not explore.","If the frozen judge is swapped for a deterministic symbolic evaluator of the emitted formula, the framework would gain a formal guarantee and remove the unmeasured judge-error risk; the paper's results do not establish that an LLM judge is necessary for the gains.","The larger gain on GPQA than on GSM8K hints that process-level reward for formula fidelity may teach general constraint-following rather than medical memorization; this is an inference, since the paper provides only two open-domain benchmarks."],"forward_implications":["A 3B model trained this way outperforms all open-weight baselines on MedCalc-Bench (51.34 vs 39.03 for Qwen2.5-32B-Instruct) and substantially narrows the gap to o1-mini and DeepSeek-R1.","Ablation results attribute 11.74 average points to the two reward components together, with equation-based tasks relying mainly on formula verification and rule-based tasks on the hybrid reward.","SFT alone can hurt unseen-task performance, while SFT plus RL improves generalization to new medical subtasks and to the open-domain GSM8K and GPQA benchmarks.","Formula recall improves from 12.02 in the base model to 57.92 in MedCalc-R1, and final accuracy rises by 10.69 points over the SFT baseline, showing the reward closes the gap between knowing a formula and applying it correctly."],"supporting_citations":[{"why":"Supplies the MedCalc-Bench dataset, the 55 clinical tasks, and the tolerance interval [L, U] that the framework is built around.","marker":"Khandekar et al. (2024)"},{"why":"Provides the GRPO algorithm used in the RL stage and the DeepSeek-R1 baselines.","marker":"Guo et al. (2025)"},{"why":"Provides the Qwen2.5-Instruct backbone models and the verifier model used for knowledge verification.","marker":"Yang et al. (2024a)"},{"why":"Supplies the HuatuoGPT-o1 domain-specific medical baseline.","marker":"Chen et al. (2024)"},{"why":"Establishes the o1-mini closed-source upper-bound comparison.","marker":"Jaech et al. (2024)"},{"why":"Supplies the CalcQA benchmark used for testing generalization to a different rule-based medical calculation task.","marker":"Zhu et al. (2025)"},{"why":"Provides the GSM8K open-domain numerical reasoning benchmark for the generalization analysis.","marker":"Cobbe et al. (2021)"},{"why":"Provides the GPQA benchmark that MedCalc-R1 uses to demonstrate cross-domain reasoning transfer.","marker":"Rein et al. (2024)"}],"fun_headline_variants":["3B medical math model beats 32B with formula-verified hybrid rewards","Small model, big clinical math: MedCalc-R1 3B outperforms 32B rivals","Formula verification plus safety intervals: why MedCalc-R1 3B wins","Knowledge-guided rewards lift 3B MedCalc-R1 to top of open-weight roster","MedCalc-R1: formula check and safety thresholds boost 3B clinical math"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The formula-verification reward assumes the frozen judge LLM correctly decides whether a generated formula matches medical guidelines and the patient context, yet the paper never measures the judge's own error rate against a gold standard.","fun_headline_variants_meta":{"raw":{"variants":["3B medical math model beats 32B with formula-verified hybrid rewards","Small model, big clinical math: MedCalc-R1 3B outperforms 32B rivals","Formula verification plus safety intervals: why MedCalc-R1 3B wins","Knowledge-guided rewards lift 3B MedCalc-R1 to top of open-weight roster","MedCalc-R1: formula check and safety thresholds boost 3B clinical math"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000884,"raw_usage":{"total_tokens":3815,"prompt_tokens":938,"completion_tokens":2877,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":554,"completion_tokens_details":{"reasoning_tokens":2768}},"tokens_in":554,"tokens_out":2877,"duration_ms":20735,"temperature":1.0,"reasoning_tokens":2768,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:28:57.131851+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a gold-standard sample of formulas from MedCalc-Bench, label each as valid or invalid by clinical experts, run the Qwen2.5-14B-Instruct verifier on the same sample, and compare its binary verdicts with the expert labels; if agreement is low or biased toward rejecting valid formulas, the knowledge reward's contribution to MedCalc-R1's gains is not the mechanism the paper claims.","supporting_citations":[{"cited_title":"Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future","cited_arxiv_id":null,"evidence_quote":"Supplies the HuatuoGPT-o1 domain-specific medical baseline."}],"review_version":1}