{"id":"424621a6-9607-4eba-b59c-ac3ad451d1a3","arxiv_id":"2505.11852","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"MedSG-Bench evaluates visual grounding across medical image sequences, shows that existing MLLMs score very low, and provides a large instruction-tuning set and a fine-tuned model, MedSeq-Grounder.","lead":"The paper introduces a benchmark, MedSG-Bench, that tests whether multimodal AI can point to matching or changing regions across sequences of medical images. It finds current models perform poorly and releases a training set plus a fine-tuned model for progress.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim for MedSeq-Grounder rests on an unspecified train/test split between MedSG-188K and MedSG-Bench; a contamination audit is needed.","rationale":"The reader's weakest_assumption is exactly the load-bearing condition: disjointness of MedSG-188K and MedSG-Bench. I agree. The benchmark itself is a real asset—76 datasets, eight tasks, 9,630 QA pairs—and the zero-shot baseline table is informative regardless of contamination, because those models are evaluated without training on the benchmark. The failure mode is specific and testable: Section 3.1.1 describes deduplication only within MedSG-Bench, and Section 4.1 does not describe a split between training and evaluation resources. This is an omitted-protocol issue that the authors can fix, rather than a fundamental flaw in the benchmark design. Consequently I do not move the reader's verdict: CONDITIONAL (accept pending the contamination audit or an explicit split description) remains appropriate. If the audit reveals overlap, the MedSeq-Grounder SOTA claim should be downgraded and the paper revised to either re-evaluate on a truly held-out set or remove the SOTA framing.","tokens_in":21357,"tokens_out":5464,"duration_ms":56163,"concrete_test":"Download the released MedSG-188K and MedSG-Bench from the provided Hugging Face repository. Compute exact image-level overlap by comparing image file hashes and any canonical image IDs; then, using available metadata, group images by source dataset and patient/case where possible. Re-run the released MedSeq-Grounder checkpoint on (a) the subset of MedSG-Bench with no matching image and no same-patient sequence appearing in MedSG-188K, and (b) the subset with overlaps. If average IoU on the disjoint subset is materially lower than the reported 72.55 (e.g., by more than 10 points) or is close to zero-shot baselines, the contamination concern lands. If the two subsets perform similarly and the split is documented, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"MedSG-Bench's headline result—that the fine-tuned MedSeq-Grounder reaches 72.55 average IoU and 79.71 Acc@0.5 in Table 3—is only interpretable if MedSG-188K training samples are disjoint from MedSG-Bench evaluation samples. The paper never states such a split. Section 4.1 says MedSG-188K 'is based on the eight tasks defined by MedSG-Bench,' and both resources are built from the same 76 public datasets with the same eight task templates; Section 5.3 then reports SOTA on MedSG-Bench without any held-out description. There is no validation split, no patient-level deduplication between the two resources, and no statement that benchmark sequences are excluded from the 188,163 training samples. Since MedSeq-Grounder is trained for roughly five epochs over 188K samples, memorization of evaluation images or near-duplicates (same patient/sequence with a different instruction template) would plausibly produce exactly the large gap from zero-shot models. If overlap exists, Table 3's MedSeq-Grounder row is invalid as evidence, and the 'state-of-the-art' contribution reduces to a distribution-fit result rather than a measurable generalization result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"MedSG-Bench introduces a benchmark for visual grounding in medical image sequences, built from 76 public datasets and organized around eight VQA-style tasks in two paradigms: Image Difference Grounding and Image Consistency Grounding. The authors evaluate a range of general-purpose and medical-specialized MLLMs in a zero-shot setting, reporting uniformly low performance (best zero-shot model, Migician, reaches 20.29 average IoU and 11.39 Acc@0.5). They also construct MedSG-188K, an instruction-tuning dataset, and fine-tune Qwen2.5-VL-7B to obtain MedSeq-Grounder, which reaches 72.55 average IoU and 79.71 Acc@0.5 on MedSG-Bench and is claimed as state of the art.","tokens_in":21628,"tokens_out":4285,"duration_ms":46615,"significance":"The benchmark fills a real gap: existing medical visual grounding benchmarks are single-image, while longitudinal, multi-view, and cross-modal comparisons are clinically important. The task taxonomy, the breadth of source datasets and modalities, and the public release of the benchmark and model are concrete strengths. The zero-shot evaluation of many MLLMs is also a useful falsifiable measurement. However, the state-of-the-art claim for MedSeq-Grounder is only interpretable if the MedSG-188K training data is disjoint from the MedSG-Bench test data; the paper currently does not establish this, so the headline result is unverifiable as written.","major_comments":[{"comment":"The paper never states that MedSG-188K training samples are disjoint from MedSG-Bench evaluation samples. Both resources are generated from the same 76 public datasets and the same eight task templates, and §4.1 explicitly says MedSG-188K is 'based on the eight tasks defined by MedSG-Bench.' There is no validation split, no patient-level or sequence-level deduplication between the two resources, and no statement that benchmark sequences were excluded from the 188,163 training samples. Without such a split, the 72.55 IoU / 79.71 Acc@0.5 row in Table 3 may reflect memorization or near-duplicate leakage rather than generalization, and the state-of-the-art claim in §5.3 is not a measurable result. The authors should specify the exact overlap-removal procedure, publish the split, and re-report MedSeq-Grounder on a verified held-out subset.","section":"§4.1, §5.3, Table 3"},{"comment":"The 'state-of-the-art' comparison is also apples-to-oranges: MedSeq-Grounder is fine-tuned on in-domain instruction data derived from the task definitions of MedSG-Bench, while all other models are evaluated zero-shot. Even with a clean train/test split, this does not establish state of the art over the baselines; it establishes that in-domain supervised tuning improves over zero-shot MLLMs. The authors should either fine-tune the strongest baselines on the same MedSG-188K training split and compare them on the held-out evaluation set, or explicitly reframe the MedSeq-Grounder result as a supervised upper bound rather than a state-of-the-art comparison.","section":"§5.3, Table 3"}],"minor_comments":[{"comment":"The text refers to 'MedSeqVG-188K', but the dataset is named MedSG-188K throughout the rest of the paper; the names should be unified.","section":"§5.3"},{"comment":"The column header does not label the paired IoU and Acc@0.5 values under each task; the reader must infer that each task cell contains two numbers. Adding explicit subheaders or formatting the rows as IoU/Acc@0.5 would improve readability.","section":"Table 3"},{"comment":"There is a duplicated word in 'spans 10 medical imaging modalities and and encompasses 114 distinct clinical tasks'; the extra 'and' should be removed.","section":"§3.3"},{"comment":"The evaluation protocol is described only as 'models were prompted to perform inference without access to in-context examples.' The paper should provide the exact prompt template and the parsing rule used to extract predicted bounding boxes, since both affect the reported IoU numbers and reproducibility.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"The benchmark itself is a solid contribution and the zero-shot baseline results are valuable independently of the MedSeq-Grounder claim. I would not reject the paper, but I would require the authors to demonstrate disjointness between MedSG-188K and MedSG-Bench, to publish the split, and to re-report the MedSeq-Grounder numbers on that split before the state-of-the-art claim can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: MedSG-Bench fills a real gap—no one has benchmarked fine-grained grounding across medical image sequences—and the zero-shot numbers are worth having. But the SOTA claim for MedSeq-Grounder is not interpretable as written because the paper never states that the MedSG-188K training set is disjoint from the MedSG-Bench test set. That is a load-bearing omission, not a nitpick.\n\nWhat's genuinely new: the two-paradigm/eight-task structure (difference vs. consistency grounding) is a sensible organizing scheme for clinical comparison tasks like pre/post-treatment tracking and cross-modal alignment. The construction protocol is described in detail, using 76 public datasets across 10 modalities, and the task generation steps—registered vs. non-registered difference, multi-view, object tracking, patch grounding, cross-modal, referring—cover a reasonable range of clinical scenarios. The baseline sweep is broad, and the observation that even top zero-shot models land around 20 IoU is a useful, credible signal that medical sequential grounding is hard.\n\nSoft spots, in order of seriousness. First, the split problem. Section 4.1 says MedSG-188K is built on the eight MedSG-Bench tasks; both resources draw from the same 76 datasets and the same templates. Section 5.3 reports SOTA with no mention of a held-out split, no validation split, no patient-level deduplication. The MedSeq-Grounder numbers in Table 3 are suspiciously far above the zero-shot baselines. Training on 188K samples for 15,000 steps could easily memorize evaluation images or near-duplicates if any overlap exists. The authors need to state explicitly how MedSG-Bench test sequences were excluded from MedSG-188K, and if they can't, they need to rebuild the training set with a clean split and retrain. Second, smaller: the evaluation protocol for MedSeq-Grounder is described differently from the zero-shot baselines; that's fine, but they should confirm prompt equivalence or at least acknowledge the difference. Third, minor wording: 'MedSeqVG-188K' appears in 5.3 (typo), and there are a couple of doubled words.\n\nBottom line: the benchmark itself is useful and the zero-shot results are informative. The SOTA claim is the weak point. With a clarifying statement or a clean re-split, this paper becomes a solid contribution. For peer review: yes, send it out—a good referee can push for the split fix. I would not desk-reject this.","headline":"MedSG-Bench fills a real gap—sequential medical grounding—but the SOTA claim for MedSeq-Grounder is uninterpretable until the authors disclose a train/test split between MedSG-188K and MedSG-Bench.","tokens_in":22146,"tokens_out":2674,"would_cite":false,"duration_ms":25529,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A benchmark for grounding across medical image sequences shows current MLLMs stay near 20% IoU zero-shot, while instruction tuning on the same task templates raises a 7B model to 72.55% IoU.","keywords":["medical image sequences","visual grounding","multimodal large language models","benchmark","difference grounding","consistency grounding","instruction tuning","IoU evaluation"],"falsifier":"To test the central claim, compute the maximum IoU between each MedSG-Bench evaluation sample's images and every MedSG-188K training sample's images, or compare their source-case identifiers; if any training-evaluation pair is the same or near-identical, the MedSeq-Grounder numbers in the main table are inflated. A cleaner falsifier is to retrain MedSeq-Grounder on MedSG-188K with any samples sharing a source case with the evaluation set removed, then check whether the 72.55% average IoU survives.","tokens_in":21193,"feed_emoji":"🩻","tokens_out":8154,"duration_ms":75187,"temperature":0.7,"pith_summary":"This paper attempts to establish that visual grounding across sequences of medical images is a distinct, underevaluated capability that current multimodal large language models largely lack. It introduces MedSG-Bench, a benchmark of 9,630 VQA-style question–answer pairs built from 76 public datasets and 10 imaging modalities, organized into two paradigms: difference grounding (locating what changed between two images) and consistency grounding (locating the same structure, tracking an object, matching a patch, or grounding a language reference across views, frames, or modalities). On this benchmark, the strongest zero-shot model averages only 20.29% IoU and 11.39% accuracy at IoU 0.5, which the paper takes as evidence that sequence-level grounding is unsolved. The paper then builds MedSG-188K, an instruction-tuning dataset from the same eight task templates, and reports that a 7B model fine-tuned on it, MedSeq-Grounder, reaches 72.55% average IoU and 79.71% Acc@0.5, establishing a state of the art for the benchmark.","feed_headline":"Best zero-shot vision model scores 20% on medical image sequences","feed_subtitle":"New 76-dataset benchmark: fine-tuning on its own tasks lifts a 7B model to 72.6% IoU.","key_machinery":"The load-bearing mechanism is the VQA-style grounding template that underlies all eight tasks. Each template takes a sequence of medical images (length two to six), a natural-language prompt drawn from one of two task families—Image Difference Grounding for localized changes and Image Consistency Grounding for invariant or shared semantics—and a ground-truth bounding box, so every sample is scored by the same IoU and Acc@0.5 metrics. The same templates are reused, with expanded phrasing and registered versus non-registered shifts, multi-view slices, tracked frames, extracted patches, cross-modal pairs, and referring expressions, to generate the 188,163 training samples of MedSG-188K, which is what lets MedSeq-Grounder learn the task distribution rather than only evaluate it.","core_discovery":"The central discovery is reported as a capability gap plus a recipe: no existing benchmark measures grounding on medical image sequences, and no evaluated multimodal large language model can do it well zero-shot, while a model tuned on a purpose-built instruction dataset can. The paper's claim is that a benchmark organized around two grounding paradigms—what changes across images and what stays the same across images—captures the clinically relevant forms of sequential reasoning, from pre/post-treatment comparison to cross-view and cross-modal alignment. The claimed result is that all zero-shot models, including a specialized multi-image grounding model, stay near 20% average IoU, whereas MedSeq-Grounder, a 7B multimodal model fine-tuned on MedSG-188K, reaches 72.55% average IoU.","pith_inferences":["We infer a possible overlap issue: Section 4.1 says MedSG-188K is based on the eight MedSG-Bench tasks and the same source datasets, while Section 5.3 reports state-of-the-art results without stating a train/eval split; if training samples overlap with evaluation samples, the reported 72.55% IoU is optimistic.","A testable extension would be to measure how much of MedSeq-Grounder's gain comes from the task templates rather than from dataset scale, by training on a smaller or template-restricted subset and checking whether IoU degrades.","We also infer that the two-paradigm taxonomy could transfer outside medicine: the same difference-versus-consistency grounding split could be used to benchmark temporal grounding in natural video or multi-view imagery."],"forward_implications":["If the benchmark's zero-shot results hold, medical sequence grounding is far from solved even for models that perform well on single-image medical VQA.","If the MedSeq-Grounder result holds, fine-tuning on grounding-supervised instruction data derived from the same task templates is a sufficient path to strong sequential grounding performance.","The paper's comparison suggests that medical-domain fine-tuning can hurt grounding, so future medical MLLMs should retain or include spatial grounding supervision.","The benchmark's task taxonomy—difference grounding versus consistency grounding—gives a common scoring ground for comparing future medical multi-image models."],"supporting_citations":[{"why":"Defines the single-image medical grounding benchmark that MedSG-Bench extends to sequences.","marker":"[16]"},{"why":"Another single-image grounding benchmark used as a comparison point for the missing sequence capability.","marker":"[52]"},{"why":"The first multi-image grounding model, used as the strongest zero-shot baseline in the evaluation.","marker":"[78]"},{"why":"The model family used as the base for MedSeq-Grounder and one of the general-purpose families evaluated.","marker":"[8]"},{"why":"The medical-specialized MLLM family evaluated, showing degraded grounding relative to general-purpose models.","marker":"[20]"},{"why":"A multi-image medical VQA benchmark that motivates the temporal and disease-progression aspect of sequence grounding.","marker":"[92]"},{"why":"The language model used to expand seed instruction templates into diverse phrasings for MedSG-188K.","marker":"[2]"}],"fun_headline_variants":["Zero-shot MLLMs score ~20% on medical sequences, tuned hits 72%","First benchmark for medical sequence grounding: 20% zero-shot vs 72% tuned","Medical sequence grounding: zero-shot ~20%, fine-tuned 72.5% IoU","New MedSG-Bench: 76 datasets, 8 tasks, exposes MLLM limits","From 20% zero-shot to 72% fine-tuned on medical sequences"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that MedSG-188K, the training set used to produce MedSeq-Grounder, contains no images that also appear in the MedSG-Bench evaluation set; if the two overlap, the reported state-of-the-art scores are not a fair measurement.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot MLLMs score ~20% on medical sequences, tuned hits 72%","First benchmark for medical sequence grounding: 20% zero-shot vs 72% tuned","Medical sequence grounding: zero-shot ~20%, fine-tuned 72.5% IoU","New MedSG-Bench: 76 datasets, 8 tasks, exposes MLLM limits","From 20% zero-shot to 72% fine-tuned on medical sequences"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001238,"raw_usage":{"total_tokens":5115,"prompt_tokens":1013,"completion_tokens":4102,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":629,"completion_tokens_details":{"reasoning_tokens":3987}},"tokens_in":629,"tokens_out":4102,"duration_ms":31309,"temperature":1.0,"reasoning_tokens":3987,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:46:04.172430+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"To test the central claim, compute the maximum IoU between each MedSG-Bench evaluation sample's images and every MedSG-188K training sample's images, or compare their source-case identifiers; if any training-evaluation pair is the same or near-identical, the MedSeq-Grounder numbers in the main table are inflated. A cleaner falsifier is to retrain MedSeq-Grounder on MedSG-188K with any samples sharing a source case with the evaluation set removed, then check whether the 72.55% average IoU survives.","supporting_citations":[{"cited_title":"MMXU: A Multi-Modal and Multi-X-ray Understanding Dataset for Disease Progression","cited_arxiv_id":"2502.11651","evidence_quote":"A multi-image medical VQA benchmark that motivates the temporal and disease-progression aspect of sequence grounding."}],"review_version":1}