{"id":"dbb58009-d26f-4d94-a5f6-31d19d8f8a07","arxiv_id":"2608.10827","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A two-stage pipeline, MCTS-based supervised fine-tuning plus GRPO reinforcement learning with a validation-gated reflection memory, makes an 8B medical vision-language model both more accurate and more selective in tool use.","lead":"MIRA trains an 8-billion-parameter medical image assistant to decide when to zoom, crop, point, measure, rotate, or search the web, and then to check whether the returned evidence actually supports its diagnosis. The system improves its base model by 7.4 points on average across nine medical question-answering benchmarks and cuts harmful tool use from 8.9% to 1.6%.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Validation-gated reflection memory uses an unspecified D_val; if it overlaps the nine test splits, the headline +7.4 gain is inflated by selection on the evaluation data.","rationale":"The reader's weakest assumption is exactly the concern I consider most load-bearing: the validation set D_val that gates reflection memory acceptance is unspecified, and the paper's assertion that test splits are never used cannot be audited from the text. The central claim is the +7.4 average improvement; the only component of the pipeline that performs explicit selection of model behavior against a held-out set is the memory gate in Eq. 14. If D_val overlaps the evaluation benchmarks, the accepted memory has been chosen to improve those benchmarks, which directly inflates the headline number. This is not an internal inconsistency, but an unverified external precondition, and the manuscript itself promises released recipes and an audit trail, so the requested check is feasible. I do not see a stronger objection: the MCTS data engine, reward composition, and GRPO setup are described in enough detail to be internally coherent, and the ablations are consistent with the claimed contributions. The secondary issue about MedImgRU versus MedLesion naming in Section 4.4 is real but does not change the numerical central claim. Because the reader already marked the paper CONDITIONAL on the same validation-set issue, my read leaves the verdict unchanged.","tokens_in":36987,"tokens_out":3979,"duration_ms":45839,"concrete_test":"Request the exact D_val sample identifiers from the authors, relying on the promised released data recipes and the audit trail described in Appendix E, and compute the intersection with the test splits of SLAKE, PMC-VQA, OmniMedVQA, MedLesionVQA, MedLesionMCQ, VQA-RAD, PathVQA, MedXpertQA-MM, and MMMU-Medical. If the intersection is non-empty, recompute Table 1 after removing the leaked samples from evaluation or after re-running memory selection on a cleanly disjoint D_val; if the average gain drops materially, the headline is inflated. If the intersection is empty, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing condition for the central claim (Section 4.3: MIRA-VL-8B improves the average from 57.29 to 64.73) is that the reflection memory accepted through Eq. 14 generalizes because D_val is a genuine held-out set disjoint from all evaluation splits. The paper never specifies how D_val is constructed, its size, or its overlap with the test splits of the nine benchmarks in Table 1. Sections 1 and 2.2 assert that test splits are never used for validation or checkpoint selection, but that is a statement, not auditable documentation. This matters because accepted memory rules are injected into the system prompt at inference (Section 3.3 and Appendix E), so every acceptance decision is a form of model selection. If D_val contains samples from the evaluation benchmarks, the memory has been selected on those benchmarks and the reported gain is not an unbiased estimate. The Table 4 ablation isolates the memory component on the same benchmarks, so it cannot resolve the leakage question. A secondary inconsistency also weakens the surrounding narrative: the tool-use necessity analysis (Figure 4, Table A2) labels the lesion benchmarks MedImgRU-QA and MedImgRU-MCQ, while Table 1 reports MedLesionVQA and MedLesionMCQ, undermining the claim of consistency across all nine benchmarks; however, the primary numerical claim rests on Table 1 and D_val.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MIRA, a two-stage medical visual diagnostic agent built on Qwen3-VL-8B. Stage one uses a tool-augmented Monte Carlo Tree Search engine to synthesize golden tool-use and reflection trajectories for SFT. Stage two applies GRPO with a composite reward (format, accuracy, consistency) and an online reflection memory that is updated only when a candidate memory improves rollout reward on a held-out validation set. The central empirical claim is that MIRA-VL-8B raises the average accuracy across nine medical VQA benchmarks from 57.29 to 64.73 (Table 1), with especially large gains on lesion-level benchmarks, while also increasing useful tool use from 56.2% to 73.8% and reducing harmful tool use from 8.9% to 1.6% (Section 4.4). The paper also provides ablations isolating SFT data components and RL rewards, fine-grained task analyses, training curves, and qualitative case studies.","tokens_in":37222,"tokens_out":5117,"duration_ms":60740,"significance":"If the headline evaluation result is unbiased, the contribution is significant for medical visual agent research: it demonstrates that a compact 8B model can approach or exceed much larger generalist and medical-specific systems by combining active tool use with explicit verification and memory-guided reflection. The paper has concrete strengths: the central numbers in Table 1 are internally consistent (I re-computed the averages and deltas), the two-stage training recipe is described in enough detail to be reproducible, and the authors state that trajectory data, tool environment, training recipes, and checkpoints will be released. The core improvement is measured on external public benchmarks, so the result is not circular by construction. However, the validity of the headline gain depends on whether the validation set used for reflection-memory selection is genuinely disjoint from the nine evaluation test splits; that condition is asserted but not documented, and one benchmark appears under two different names in the paper.","major_comments":[{"comment":"The validation-gated reflection memory is a load-bearing selection step: Eq. (14) accepts a candidate memory only when S_t(˜m) > S_t(m) on a held-out validation set D_val, and the accepted memory is injected into the system prompt at inference (Section 3.3 and Appendix E). The paper never specifies how D_val is constructed, its size, or which data sources it draws from, and it does not provide documentation showing D_val is disjoint from the test splits of the nine benchmarks in Table 1. Sections 1 and 2.2 assert that test splits are never used for validation or checkpoint selection, but an assertion is not an auditable dataset statement. Because every memory acceptance is a form of model selection, any overlap between D_val and the evaluation splits would inflate the reported +7.44 average gain and the per-benchmark gains. The Table 4 ablation cannot resolve this issue because it evaluates the memory component on the same benchmarks. The authors should specify D_val's composition and size, release it or describe its construction from training-only sources, and verify disjointness from all nine test splits; otherwise the headline improvement is not independently verifiable.","section":"§3.3, Eq. (14); §1; §2.2; Appendix E"},{"comment":"The tool-use necessity analysis labels two benchmarks as MedImgRU-QA and MedImgRU-MCQ, while Table 1 and Section 4.2 report the same two benchmarks as MedLesionVQA and MedLesionMCQ. The paper should clarify whether these are the same datasets with inconsistent naming or two different benchmark families. If they are the same, the names should be unified throughout the paper and the appendix; if they are different, the claim of evaluating 'all nine benchmarks' and the aggregated useful/harmful tool-use percentages in Section 4.4 need to be recomputed and restated. As written, the inconsistency undermines the claim that the tool-use necessity improvement is consistent across the same nine benchmarks used for the main accuracy result.","section":"§4.4, Figure 4, Table A2 vs. §4.2, Table 1"},{"comment":"The consistency reward used in Eq. (13) is computed by a judge that receives the current reflection memory as the 'injected skill' and is asked to penalize trajectories that contradict that skill. Consequently, the validation score S_t(m) in Eq. (14) is not a pure measure of diagnostic improvement: it includes a self-referential component in which the memory being evaluated is also the standard against which trajectory consistency is judged. A candidate memory that is more prescriptive can mechanically raise the consistency reward even if it does not improve answer correctness, although the result-reward gating in Eq. (13) limits the impact on the final benchmark accuracies. The paper should either adjust the validation score to use a memory-independent measure of trajectory quality or explicitly acknowledge and quantify this coupling when presenting the memory acceptance criterion as a held-out reward improvement.","section":"§3.2, Eq. (13); §3.3, Eq. (14); Appendix G"}],"minor_comments":[{"comment":"The evaluation section states that all benchmarks are evaluated with the 'MedEvalKit testing suite [46]', but reference [46] is the Lingshu paper, not a MedEvalKit citation; please provide the correct reference for MedEvalKit or cite the appropriate software/benchmark resource.","section":"Appendix G"},{"comment":"The benchmark names MedLesionVQA and MedLesionMCQ appear with inconsistent capitalization (e.g., 'Medlesion VQA', 'Medlesion MCQ') across tables and prose; unify the spelling to match Section 4.2.","section":"Table 1; Table 3; Table 4; §4.2"},{"comment":"The row label 'MIRA-RL w/o Tools' refers to evaluation with tool execution disabled, while the model was trained with tools; consider renaming the row to 'MIRA-RL, tools disabled at inference' to avoid ambiguity about training versus inference-time tool use.","section":"Table 4"},{"comment":"The RL training curves report accuracy reward and consistency reward values but do not state whether these curves are computed on a held-out validation set or on the on-policy rollout batches; please specify the data source for Figure 5 so readers can interpret the curves as training diagnostics rather than evaluation results.","section":"§4.4 and Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The central empirical claim is plausible and internally consistent, but the unspecified D_val is a genuine validity concern that must be resolved before the headline numbers can be trusted: validation-guided memory selection with an undisclosed validation split is precisely the kind of leakage risk that reviewers cannot audit from the current text. I also note that MedLesionVQA/MedLesionMCQ [48] shares authors with the MIRA paper; this does not by itself invalidate the result, but it increases the importance of documenting that the SFT data construction used only training splits and that the test splits were not touched during development. If the authors can provide a clear construction protocol for D_val and verify disjointness from all nine test splits, I would view the paper as a solid contribution suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper to know about: MIRA is an 8B medical VLM that uses a small tool suite (zoom, ground, point, rotate, measure, search) and a two-stage pipeline—MCTS-distilled SFT trajectories plus GRPO RL with a reflection memory. The headline is a +7.4 average point gain over its Qwen3-VL-8B backbone across nine public benchmarks, with useful tool use up and harmful tool use down. The numbers are credible as reported.\n\nWhat is actually new: the validation-gated reflection memory (Section 3.3) is a genuine departure from the OpenThinkIMG/VTool-R1/DeepEyes/MedReasoner line. Failures are distilled into candidate memory edits, and an edit is accepted only if it improves rollout reward on a held-out validation set under the frozen policy. That is a real mechanism, not just a prompt tweak. The failure-driven correction SFT data (Section 2.3) is also new and well-motivated. The paper is unusually transparent about prompts, tool schemas, MCTS hyperparameters, and reward formulas.\n\nThe main soft spot is exactly what the stress-test flags: D_val, the validation set that gates memory acceptance, is never specified. It is not described in Section 3.3 or Appendix E beyond being a \"held-out validation set.\" The paper asserts test splits are never used for validation or checkpoint selection, but there is no dataset documentation to audit that claim. Because accepted memory rules are injected into the system prompt at inference, every acceptance decision is a form of model selection. If D_val overlaps any of the nine test splits, the +7.4 average is inflated. This is an addressable issue—the authors need to state the composition of D_val and its overlap with evaluation splits. I do not see a reason to believe there is leakage, but the burden is on them.\n\nTwo smaller concerns. First, the tool-use necessity analysis in Figure 4 and Table A2 labels the lesion benchmarks MedImgRU-QA and MedImgRU-MCQ, while Table 1 reports MedLesionVQA and MedLesionMCQ. That undermines the \"consistent across all nine benchmarks\" claim as written; it is presumably the same dataset under two names, but it needs to be fixed. Second, the headline reliability numbers (73.8% useful, 1.6% harmful) rest on a single GPT-4o judge with no validation of the judge against human labels. That is a minor weakness, not a fatal one. Also worth noting: MedLesion is co-authored by a MIRA author, so the two lesion benchmarks are not entirely independent of the method.\n\nWho this is for: anyone building medical VLMs with tool use or RL-improved reasoning. The recipe is reproducible at 8B scale and the release includes checkpoints, data, and the tool environment. A serious referee should engage with this paper. My recommendation: accept for review, with the D_val question as the primary condition.","headline":"MIRA is a genuinely new training recipe for medical VLMs—validation-gated reflection memory plus failure-driven SFT data—with an internally consistent +7.4-point gain, but the unspecified validation set for memory acceptance is the one thing a referee must pin down.","tokens_in":37880,"tokens_out":2335,"would_cite":true,"duration_ms":21486,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A two-stage training procedure that makes a medical vision-language model verify its tool use improves its average diagnostic score from 57.29 to 64.73 across nine benchmarks.","keywords":["medical visual question answering","agentic diagnosis","tool use verification","Monte Carlo Tree Search","reinforcement learning","reflection memory","evidence grounding","multimodal reasoning"],"falsifier":"Check whether any accepted reflection-memory rule, or the validation set used in the memory-selection rule, contains examples drawn from the test splits of SLAKE, PMC-VQA, VQA-RAD, PathVQA, MedLesionVQA, MedLesionMCQ, OmniMedVQA, MedXpertQA-MM, or MMMU-Medical. Concretely, recompute the headline average after re-running evaluation with memory rules that were rejected by the validation gate; if rejected memories produce larger test gains than accepted ones, the selection signal is not predictive. A simpler observation is to inspect the released memory-update logs for dataset names or test-set identifiers—if any appear, the separation claim fails.","tokens_in":36721,"feed_emoji":"🩺","tokens_out":5420,"duration_ms":51591,"temperature":0.7,"pith_summary":"MIRA claims that medical image diagnosis can be made more reliable by turning a static question-answering model into an agent that actively inspects images and checks whether each tool result supports its current diagnostic hypothesis. Its central quantitative assertion is that a two-stage pipeline—Monte Carlo Tree Search–generated tool trajectories for supervised fine-tuning, then reinforcement learning with a reflection memory—converts an 8B vision-language backbone into a stronger diagnostician, improving the average score from 57.29 to 64.73 across nine benchmarks. If that holds, selective and evidence-grounded tool use, not simply more tool calls, is what drives medical visual reasoning gains, and a compact open model can narrow the gap to much larger systems. The paper also reports useful tool use rising to 73.8% and harmful tool use falling to 1.6%, and its case studies show the agent re-examining evidence and correcting early conclusions.","feed_headline":"Tool-verifying medical agent lifts 8B VLM by 7.4 points","feed_subtitle":"Two-stage MCTS-plus-RL training raises useful tool use from 56.2% to 73.8% and cuts harmful calls on nine benchmarks.","key_machinery":"The load-bearing machinery is a two-stage training loop built on reflection. A tool-augmented Monte Carlo Tree Search data engine, offering six lightweight tools (zoom, grounding, point, rotate, measure, and web search), turns static VQA instances into verified evidence-seeking trajectories, with a unified verifier rendering spatial tool calls on the image and scoring each node for visual plausibility and semantic usefulness. Supervised fine-tuning on those trajectories plus failure-driven and intention-level reflection data provides a cold-start policy. GRPO reinforcement learning on fresh rollouts then applies a composite reward—format, answer correctness, and a consistency score paid only when the answer is correct—alongside a validation-gated reflection memory that distills repeated failures into candidate principles, injects them into the system prompt, and accepts them only when they improve mean rollout reward on a held-out validation set under a frozen policy. This memory is the distinctive mechanism: it makes failures reusable across cases while guarding against memorizing single outliers.","core_discovery":"The central discovery claimed is that a medical visual agent learns to decide when to use tools, verify spatial claims, and revise premature conclusions through a carefully staged curriculum. MIRA-SFT first builds a cold-start policy from MCTS-searched trajectories, where search nodes are jointly scored by visual grounding plausibility and semantic consistency with the ground-truth answer, then adds failure-driven correction reflections and intention-level thoughts. MIRA-RL then optimizes this policy online with GRPO on fresh multi-turn rollouts, using a composite reward that gates a consistency bonus behind answer correctness, and a validation-gated reflection memory that converts repeated failures into reusable diagnostic principles before later rollouts. The paper's central quantitative claim is the +7.44 average improvement over the backbone, with the largest gains on lesion-level tasks (MedLesionVQA +14.5, MedLesionMCQ +14.4). Fine-grained analysis locates the gains in basic perception and understanding/diagnosis/suggestion rather than in content recognition, which is consistent with the intended mechanism of evidence-grounded verification rather than memorization.","pith_inferences":["A direct test of the reflection-memory claim would be to ablate the validation gate while keeping the same candidate edits: if rejected edits sometimes improve test reward, the gate's conservatism may be hiding real gains.","The paper treats the validation set used in the memory-acceptance rule as unspecified; absent documentation that it is disjoint from the nine evaluation test sets, the headline improvement should be read with the possibility of validation-guided selection in mind.","The framework suggests a general principle for agentic systems beyond medicine: any tool-using model can benefit from a separate 'should I use this tool, and does the output confirm my hypothesis?' verification stage followed by corrective loops.","A testable extension is to measure tool-use necessity on unseen disease types or imaging modalities, since the paper's reflection memory is distilled from on-policy failures and may generalize less far under distribution shift."],"forward_implications":["If the central claim is correct, training medical vision-language models on evidence-verifying tool trajectories rather than static QA data becomes a practical recipe for improving medical visual reasoning on a fixed backbone.","Verification-gated tool use becomes a measurable objective: useful tool use rises, harmful tool use falls, and completed reasoning shortens, suggesting that reinforcement learning can make agents more selective rather than more verbose.","Because the gains concentrate in basic perception and understanding/diagnosis rather than content recognition, tool-augmented reasoning should be prioritized for interpretation and decision tasks rather than for factual recall.","The validation-gated reflection-memory mechanism implies that a small, editable, textual memory can guide policy during RL, potentially reducing the number of rollouts needed to learn from repeated failures.","A compact 8B medical agent may approach or surpass much larger open and medical-specific systems, narrowing the scale gap at a fixed inference budget."],"supporting_citations":[{"why":"Supplies the Qwen3-VL-8B backbone that MIRA fine-tunes and compares against.","marker":"[2]"},{"why":"Provides the bandit-based Monte Carlo planning algorithm used in the MCTS data engine.","marker":"[16]"},{"why":"Provides the GRPO reinforcement learning update used in the RL stage.","marker":"[38]"},{"why":"Training split source and evaluation benchmark for SLAKE.","marker":"[24]"},{"why":"Training split source and evaluation benchmark for PMC-VQA.","marker":"[25]"},{"why":"Training split source and evaluation benchmark for VQA-RAD.","marker":"[19]"},{"why":"Training split source and evaluation benchmark for PathVQA.","marker":"[12]"},{"why":"Supplies the MedLesionVQA and MedLesionMCQ benchmarks where MIRA reports its largest gains.","marker":"[48]"}],"fun_headline_variants":["Tool-verifying medical agent: +7.4 points, harmful calls down from 8.9% to 1.6%","MIRA: Agentic tools with reflective verification lift 8B VLM by 7.4","Medical agent verifies tools, gains 7.4 points on 9 benchmarks","MCTS-trained medical agent cuts harmful tool calls 82%, lifts accuracy 7.4"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison rests on the claim that the validation set used to accept reflection-memory rules never overlaps the nine test sets used for the final scores, and the paper supplies no dataset documentation to verify that separation.","fun_headline_variants_meta":{"raw":{"variants":["Tool-verifying medical agent: +7.4 points, harmful calls down from 8.9% to 1.6%","MIRA: Agentic tools with reflective verification lift 8B VLM by 7.4","Medical agent verifies tools, gains 7.4 points on 9 benchmarks","MCTS-trained medical agent cuts harmful tool calls 82%, lifts accuracy 7.4"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0006,"raw_usage":{"total_tokens":2856,"prompt_tokens":1052,"completion_tokens":1804,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":668,"completion_tokens_details":{"reasoning_tokens":1699}},"tokens_in":668,"tokens_out":1804,"duration_ms":13815,"temperature":1.0,"reasoning_tokens":1699,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:33:29.704462+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check whether any accepted reflection-memory rule, or the validation set used in the memory-selection rule, contains examples drawn from the test splits of SLAKE, PMC-VQA, VQA-RAD, PathVQA, MedLesionVQA, MedLesionMCQ, OmniMedVQA, MedXpertQA-MM, or MMMU-Medical. Concretely, recompute the headline average after re-running evaluation with memory rules that were rejected by the validation gate; if rejected memories produce larger test gains than accepted ones, the selection signal is not predictive. A simpler observation is to inspect the released memory-update logs for dataset names or test-set identifiers—if any appear, the separation claim fails.","supporting_citations":[{"cited_title":"Slake: A semantically-labeled knowledge- enhanced dataset for medical visual question answering","cited_arxiv_id":null,"evidence_quote":"Training split source and evaluation benchmark for SLAKE."},{"cited_title":"Pmc-vqa: Visual question answering for medical images","cited_arxiv_id":null,"evidence_quote":"Training split source and evaluation benchmark for PMC-VQA."},{"cited_title":"A dataset of clinically generated visual questions and answers about radiology images.Scientific data, 5(1):1–10, 2018","cited_arxiv_id":null,"evidence_quote":"Training split source and evaluation benchmark for VQA-RAD."},{"cited_title":"Medlesionvqa: A multimodal benchmark emulating clinical visual diagnosis for body surface health","cited_arxiv_id":null,"evidence_quote":"Supplies the MedLesionVQA and MedLesionMCQ benchmarks where MIRA reports its largest gains."}],"review_version":1}