{"id":"1aaae5ef-6408-4d82-af0c-eebbcfdb035d","arxiv_id":"2512.14157","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A three-stage-trained medical vision-language agent that calls zoom and segmentation tools during its chain of thought outperforms prior medical MLLMs on VQA and segmentation benchmarks.","lead":"Ophiuchus is a medical AI assistant that stops mid-reasoning to zoom into or segment suspicious image regions, then continues its diagnosis. The paper reports large gains on medical VQA and segmentation benchmarks using a three-stage training recipe and argues that tool-augmented 'thinking with images' beats both plain reasoning and static tool pipelines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Potential image-level leakage between training corpora (BiomedParseData/Malenia) and public VQA benchmarks is unreported and could inflate the headline SOTA gains.","rationale":"The reader's weakest assumption—teacher trajectory correctness—targets the in-domain test set and the model's learned tool-use policy, but it does not directly threaten the external validity of the public benchmarks, whose labels are independent of the authors' pipeline. A more load-bearing risk is image-level contamination: the training corpora, especially BiomedParseData with millions of images, are aggregated from public sources and may overlap with the evaluation benchmarks. The paper reports no deduplication, and the only leakage mitigation mentioned is for segmentation via a small in-house set. If overlap exists, the out-of-domain numbers in Table 1 and Table 9 are not a measure of generalization; they could reflect memorization of images and masks seen during training. This would undermine the abstract's 'consistently outperforms' claim even in the most optimistic scenario where all teacher trajectories are clinically correct. The proposed hash-based deduplication is an unambiguous, low-cost check that can settle the concern. I agree with the reader's CONDITIONAL verdict, but I would condition acceptance on passing this leakage test as well as addressing the trajectory-verification issue.","tokens_in":36502,"tokens_out":8178,"duration_ms":74888,"concrete_test":"Compute perceptual hashes (pHash) for all training images (BiomedParseData, Malenia) and all images in the test splits of PathVQA, SLAKE, VQA-RAD, OmniMedVQA, MMMU(H&M), and MedXpertQA. Flag any pair with a hamming distance below a threshold (e.g., <10). Re-run Ophiuchus and all baseline evaluations on the intersection-free subset (removing every flagged benchmark sample). If any benchmark accuracy changes by more than 2–3 points, the reported gains are at least partially attributable to image leakage rather than the proposed training recipe.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that the three-stage recipe yields SOTA generalization rests on public benchmark numbers that may be contaminated by training-set overlap. Ophiuchus is trained on BiomedParseData (3.4M image-mask-label triples spanning 9 modalities) and Malenia (1,514 triplets). Evaluation uses PathVQA, SLAKE, VQA-RAD, OmniMedVQA, MMMU(H&M), and MedXpertQA. The manuscript reports no deduplication between the training corpora and these evaluation images. BiomedParseData is a large aggregation of publicly available biomedical images, plausibly containing the same images or near-duplicates as those benchmarks. If so, the model has already seen the exact test images (with masks and class labels) during cold-start SFT and RL, so the out-of-domain scores reflect memorization or overfitting, not generalization. The paper only discusses leakage mitigation for segmentation via a held-out in-house set (Section 4.2), but that set is small and not used for the headline VQA results. Even if teacher trajectories are perfectly correct, this leakage would invalidate the claimed consistent outperformance. This is a concrete, falsifiable threat to the central claim, independent of the teacher-bias concern raised by the reader.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Ophiuchus, a 7B medical multimodal large language model that interleaves textual chain-of-thought reasoning with calls to external visual tools (zoom-in, SAM2, BiomedParse). The core contribution is a three-stage training recipe: cold-start supervised fine-tuning on 64k GPT-5/Gemini-generated reasoning traces, self-reflection fine-tuning on trajectories that flip from incorrect to correct across checkpoints, and GRPO-based agentic tool reinforcement learning with fine-grained rewards. The authors report consistent state-of-the-art results across eight medical VQA benchmarks and two segmentation test sets, and include ablations (Table 2), a same-tools/same-data comparison (Table 9), and robustness experiments across segmentation tools (Table 4).","tokens_in":36826,"tokens_out":9173,"duration_ms":81373,"significance":"If the empirical claims survive scrutiny, this is a genuinely useful contribution: it demonstrates a concrete training protocol for turning a general-purpose MLLM into a tool-orchestrating medical agent, with careful ablations isolating the contribution of each training stage and a held-out in-house test set. The paper also makes a serious effort to reduce circularity by evaluating on public benchmarks and by describing a multi-stage verification pipeline for teacher-generated data. The same-tools comparison in Table 9 and the tool-replacement robustness study in Table 4 are strong supporting pieces of evidence. However, the validation currently lacks two critical controls—training/test image-overlap analysis and statistical uncertainty quantification—and one comparison is under-specified. These are required before the headline claim of 'consistently outperforming SOTA' can be accepted.","major_comments":[{"comment":"Ophiuchus is trained on BiomedParseData (3.4M triples) and Malenia, then evaluated on PathVQA, SLAKE, VQA-RAD, OmniMedVQA, MMMU(H&M), and MedXpertQA. The paper reports no deduplication between these training corpora and the public evaluation images. Since BiomedParseData aggregates large numbers of public biomedical images, exact or near-duplicate overlap is plausible and would directly inflate the headline gains in Table 1. Please report an overlap analysis (e.g., perceptual hashing or exact-match at the image level) between every training image and each benchmark, and re-report all Table 1 numbers on the non-overlapping subset. If overlap is found, state whether the conclusions survive after excluding those samples.","section":"§3.3, §4.2, Table 1"},{"comment":"The in-domain Dtest-4k is described as a fixed-proportion split of the curated samples, but it is not stated that the split is performed by image identity. Because multiple QA/segmentation samples can be derived from the same image, the same image may appear in Dcold, Drl, and Dtest. That would bias the in-domain results (Dtest-VQA, Dtest-Seg). Please clarify the splitting unit; if samples were split without image-level grouping, re-split at the image level and report the affected numbers.","section":"Appendix A.1, Table 5"},{"comment":"All reported numbers are single-run point estimates with no standard deviations, confidence intervals, or significance tests. Several head-to-head differences are small (e.g., Table 4 VQA-RAD: 73.6 vs 73.3; Table 11: 59.4 vs 59.0), and the claim of 'consistently outperforming' relies on ranking without noise assessment. Please add variance estimates (at least 3 training seeds or bootstrap over test items) for the main comparisons, and give a significance test for the primary benchmark table.","section":"Tables 1–4, 9–14"},{"comment":"The same-tools/same-data comparison is central to the claim that the training framework, not the tool access or data, drives the gains. However, the description says only that open-source baselines were 'fine-tuned using the complete three-stage training data'; it does not state whether they were trained with the same cold-start SFT + self-reflection + ATRL algorithm and reward scheme, or simply SFT on the same data. If the latter, the comparison does not isolate the training method. Please clarify the exact training protocol for each baseline in Table 9, and if possible train one strong baseline with the identical three-stage recipe.","section":"Appendix C.4, Table 9"},{"comment":"The teacher-based data generation and verification pipeline is the sole source of supervised tool-use trajectories. The manuscript does not report how many samples were removed at each filtering step, nor any audit of residual errors in the retained data (e.g., inter-annotator agreement on the human cross-validation). Since the in-domain test set is generated by the same pipeline, systematic teacher bias could inflate the in-domain numbers. Please report filtering statistics and a human-expert audit of a random sample of the retained trajectories, with agreement rates.","section":"§3.3, Appendix A.4"}],"minor_comments":[{"comment":"Typos and formatting: 'LLaV A-Next-13B' unspaced, 'Boldand' in Table 1 caption, 'formate' in the reward description, and 'we also adjusted' in Appendix C.6. The notation 'Mcold−12' is ambiguous; consider 'Mcold-12'.","section":"Throughout"},{"comment":"The figure is very dense. Add a legend or annotations to clarify the three training stages and the flow of reward signals.","section":"Figure 2"},{"comment":"Report the number of QA pairs discarded by format/answer checks and by the five-dimension verification, to substantiate the 'high-quality' claim.","section":"§3.3"},{"comment":"Specify whether the GT region crop includes a margin and whether the same tool output images were used as observations for Ophiuchus.","section":"Table 3"},{"comment":"Add a column indicating whether each baseline was prompted with tools only or fine-tuned, and with which stages; this makes the comparison easier to interpret.","section":"Table 9"},{"comment":"Clarify whether Dreflect (3k instances) is disjoint from Dcold and whether the early/late checkpoints used for self-reflection sampling came from Dcold or Drl.","section":"Appendix A.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is technically interesting and the proposed training recipe is well-motivated, but the missing image-overlap analysis and lack of any variance estimation are load-bearing for the headline 'consistently outperforming SOTA' claim. I would encourage the authors to add the requested analyses rather than reject, as the core approach appears plausible and the appendix already contains much of the necessary experimental apparatus. Note also that the 'GPT-5' closed-source model is cited with a generic URL; if reviewers cannot access it, that may be a reproducibility concern."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, the training recipe is real: cold-start SFT, self-reflection SFT, and GRPO with fine-grained tool rewards form a sensible progression, and the ablations back it up. Second, the abstract's SOTA claims are not yet defensible, because the paper never reports deduplication between the training corpora (BiomedParseData, Malenia) and the public VQA benchmarks. BiomedParseData is a large aggregation of public medical images, so overlap with PathVQA, SLAKE, VQA-RAD, or OmniMedVQA is plausible. If those exact images or near-duplicates appeared in training, the out-of-domain scores reflect memorization, not generalization. The held-out in-house set is only used for segmentation, not for the headline VQA numbers. That is a concrete, falsifiable threat to the central claim.\n\nWhat the paper does well: the ablation design is the strongest part. Each training stage adds a clear increment; self-reflection SFT beats extra plain SFT epochs; the RL-only variant underuses tools; the conditional tool-use reward matters; and the same-tools/same-data comparison in Table 9 is the right kind of control. The training dynamics (tool-call count expand-then-compress) and the failure-case examples are genuinely informative. The verification pipeline for teacher trajectories is also more careful than most: GPT-o3 trace checking plus human cross-validation does not eliminate teacher bias, but it mitigates it.\n\nSoft spots, in proportion. The leakage issue is the load-bearing one and needs a direct answer: report exact overlap checks, image hashing, and benchmark provenance. Without that, the 20-point gains over closed-source models are not credible. Second, all tables are single-run point estimates; no error bars or significance tests. That is common in this subfield, but it matters more when the claim is 'consistent outperformance.' Third, the abstract says 'detection,' but there is no standard detection benchmark in the paper; the experiments are VQA and reasoning-based segmentation. That is an overclaim, though not central.\n\nIs the central recipe sound? I think yes, tentatively. The gains over the same-tools baselines suggest the training framework itself, not merely tool access, is doing work. But the paper has to close the leakage hole before the SOTA numbers can be taken at face value.\n\nFor a reader: if you work on medical MLLMs or agentic tool use, this is worth engaging. For a referee: yes, send it to review, but require a deduplication analysis and exact evaluation-split provenance before acceptance. If the overlap check is clean and artifacts are released, this is a solid systems contribution.","headline":"A genuinely useful three-stage tool-reasoning recipe for medical MLLMs, with strong ablations and an honest same-tools comparison, but the headline public-benchmark numbers are undermined by an unreported training/evaluation overlap risk.","tokens_in":37309,"tokens_out":3165,"would_cite":true,"duration_ms":33470,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a medical multimodal model trained in three stages—cold-start supervised fine-tuning, self-reflection fine-tuning, and agentic tool reinforcement learning—learns to invoke image tools (zoom and segmentation) during rea","keywords":["medical multimodal LLM","tool-augmented reasoning","agentic reinforcement learning","reasoning-based segmentation","medical visual question answering","self-reflection fine-tuning","fine-grained visual grounding"],"falsifier":"Take a held-out set of cases where an independent clinician panel's ground truth disagrees with the teacher labels used in training, and check whether tool invocations change answers in the direction of the clinician ground truth. If removing tool use—or replacing the teacher with a different generator—eliminates the gap, the claimed mechanism is not the one driving performance.","tokens_in":36399,"feed_emoji":"🔍","tokens_out":5138,"duration_ms":48346,"temperature":0.7,"pith_summary":"The authors are trying to establish that the bottleneck in medical multimodal reasoning is not model size but the inability to dynamically re-inspect fine-grained image regions. They propose Ophiuchus, a tool-augmented framework that interleaves textual chain-of-thought with calls to image tools, letting the model decide when to probe, where to look, and how to fold the returned visual evidence back into its reasoning. The key claim is that a three-stage training recipe—cold-start SFT, self-reflection fine-tuning, and agentic tool reinforcement learning—is what converts tool calling into genuine visual cognition, rather than a static pipeline. If correct, this would give medical AI agents a practical way to reduce hallucinations and false positives on tasks that require fine-grained spatial evidence. The paper reports consistent gains across eight benchmarks, including in-domain and out-of-domain VQA, detection, and reasoning-based segmentation.","feed_headline":"Tool-using medical AI beats text-only reasoning on 8 benchmarks","feed_subtitle":"A 7B model trained to zoom, segment, and reason interleaves visual evidence into its chain of thought.","key_machinery":"The load-bearing object is the interleaved reasoning trajectory, formally written as R = {(r_n, t_n, o_n)}_{n=1..N}, where each step contains natural-language thoughts, a tool invocation, and the resulting observation. Training on this object through the three stages teaches a policy for when to call a tool, where to probe, and whether to revise. The reinforcement learning stage uses fine-grained rewards: format adherence, exact-match (with IoU-thresholded piecewise rewards for segmentation), and a conditional tool-use bonus granted only when the model answers correctly and invokes at least one tool.","core_discovery":"Ophiuchus is a 7B-scale multimodal model trained to interleave textual thoughts, tool calls, and tool observations in a single reasoning loop. At each step the model may either answer directly or call one of three image tools—a text-prompted segmenter, a bounding-box-prompted segmenter, and a zoom/crop function—and the returned mask or crop is appended to the context as an observation that feeds the next thought. The three-stage training protocol is the central claim: cold-start supervised fine-tuning teaches basic tool selection; self-reflection fine-tuning on cases where the model corrected its own tool use strengthens revision behavior; and agentic tool reinforcement learning with rewards","pith_inferences":["If the recipe transfers, 'think with images' could extend to non-medical domains where global context hides small signals; a testable extension is applying the same three-stage recipe to satellite or microscopy imagery and measuring whether tool-augmented reasoning outperforms simply scaling up the base model.","The dependence on proprietary teacher models for data generation means open reproduction hinges on releasing that data; a stress test would regenerate trajectories with a smaller open teacher, or add adversarial perturbations to the verification pipeline, to see whether the gains are robust or partly inherited from teacher bias.","The conditional tool-use reward points to a broader principle: reinforcement learning should reward evidence that changes outcomes, not tool invocation itself. A natural variant would reward information gain—how much the tool observation improves the model's predicted answer—rather than conditional correctness.","The paper's own failure case shows that even a reflective agent can still misdiagnose when all tools fail; clinical deployment would likely need an explicit abstain behavior or mandatory human confirmation, which the current reward design does not incentivize."],"forward_implications":["Pre-segmenting regions and feeding crops to a model is not enough: the paper shows a pipeline with ground-truth crops underperforms Ophiuchus by large margins, so the gains come from weaving observations into reasoning, not merely from zooming in.","The policy is not brittle to a specific segmenter: swapping the default segmentation tool for alternatives changes scores by less than a point on average, implying the trained behavior is tool-agnostic orchestration.","Ophiuchus can surpass the external tools themselves, including an ensemble of both segmenters, so the model's reflection and composition add value beyond the tools' own ceilings.","The training dynamics show an expand-compress pattern—tool calls rise early during exploration and fall during exploitation—and the final policy rarely calls tools on questions the base model already answers correctly, indicating learned efficiency.","Across eight benchmarks, the model reports consistent gains over closed-source and open-source SOTA, with the largest relative improvements on fine-grained in-domain VQA and segmentation subsets."],"fun_headline_variants":["AI that zooms in on medical images beats text-only reasoning","Medical MLLM taught to segment, zoom, and reason with tools","7B model interleaves visual evidence into chain of thought","Tool-calling beats pure text on 8 medical benchmarks","Ophiuchus: training AI to think with images, not just text"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"Everything rests on the teacher-generated data being clinically sound: the QA pairs and tool-use trajectories are synthesized by proprietary frontier models guided by ground-truth masks and then filtered by a verification model plus human cross-checks (described in Section 3.3 and Appendix A.4), so if those teachers systematically mislabel or hallucinate in ways the filters miss, Ophiuchus inherits the bias and part of the reported gains is an artifact of data construction.","fun_headline_variants_meta":{"raw":{"variants":["AI that zooms in on medical images beats text-only reasoning","Medical MLLM taught to segment, zoom, and reason with tools","7B model interleaves visual evidence into chain of thought","Tool-calling beats pure text on 8 medical benchmarks","Ophiuchus: training AI to think with images, not just text"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00011,"raw_usage":{"total_tokens":902,"prompt_tokens":768,"completion_tokens":134,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":44}},"tokens_in":512,"tokens_out":134,"duration_ms":2462,"temperature":1.0,"reasoning_tokens":44,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T16:14:10.484436+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of cases where an independent clinician panel's ground truth disagrees with the teacher labels used in training, and check whether tool invocations change answers in the direction of the clinician ground truth. If removing tool use—or replacing the teacher with a different generator—eliminates the gap, the claimed mechanism is not the one driving performance.","supporting_citations":[],"review_version":1}