REVIEW 3 major objections 5 minor 49 references
LocAnyMed: Vision-Language Grounding for Multimodal Medical Images
T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read A 200K-record unified corpus lifts medical grounding F1 from 10.6 to 85.6 through full fine-tuning of a general vision-language grounding model.
desk verdict A large, carefully built medical grounding dataset and a solid fine-tuning study, but the headline same-set F1 may be inflated by image-level splits on video-derived data; the cross-set result is the safer anchor. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is the LocAnyMed-200K corpus: a harmonized set of 209,910 image-query-answer records that serialize box coordinates, point coordinates, and an explicit <box>none</box> token into one generative output language. The second piece is full-parameter fine-tuning of the pretrained 3-billion-parameter grounding model on that corpus, with optional deterministic routing of visual features through shared and modality-specific expert branches. The corpus supplies the supervision; the fine-tuning transfers the general grounding skill to medical images.
What would settle it
Recompute Same-set Det. Val. after re-splitting all sources at the patient or case level (using metadata where available and perceptual hashes otherwise), and check whether F1@IoU 0.50 stays near 85.59. If it collapses toward the 10.64 baseline, the central claim is refuted. A second check: run the trained model on a never-before-seen medical imaging modality and see if any localization transfer remains.
Extended reading notes
Core claim
The central discovery is that a general grounding model, fine-tuned from head to toe on a unified corpus of medical grounding records, can become a strong multimodal medical localizer. The corpus converts masks, boxes, points, and negative queries into a single serialized format, and the model learns box, point, and no-target outputs from the same objective. On the held-out split, F1@IoU 0.50 rises from 10.64 to 85.59 and F1 Mean from 5.72 to 69.15, exceeding both closed-set detectors trained on the same records and zero-shot general VLMs. The paper further claims that routing visual projections through shared plus modality-specific experts gives a small cross-source robustness gain, and tha
Load-bearing premise
The held-out evaluation split is free of source leakage, so that related images from the same patient, case, or scan do not appear in both training and validation; if they do, the reported F1 gain could come from memorization rather than generalization.
Editorial extensions
If this is right
- Heterogeneous medical localization datasets can be merged into a single open-ended grounding task without task-specific heads or losses.
- A generative grounding model can jointly handle boxes, points, and absent-target rejection, with negative-query accuracy reaching 92.9% after adaptation.
- Visual-evidence rationales can be distilled into the model and improve cross-source robustness without degrading same-set accuracy.
- Closed-set detectors trained on the same records still excel at strict IoU in some modalities, so generative grounding does not fully replace them yet.
- Cross-set generalization remains much weaker than same-set performance, pointing to source shift as the next bottleneck.
Reading between the lines
- The headline 85.59 gain hinges on the held-out split being free of patient or case leakage; the paper uses deterministic image-level splits for some datasets without patient identifiers, so a stricter case-level split could lower the number. This is not a claim the paper makes, but a testable condition for its central result.
- The modality-routed expert design suggests that sharing a large visual backbone across imaging modalities benefits from a small deterministic amount of modality-specific capacity; the same recipe may transfer to other multi-domain medical tasks.
- The CoT rationales are generated by a general-purpose teacher and validated for format, not clinical truthfulness; a clinician rating study would be a natural next test.
- The dataset's unified format could serve as a common benchmark for medical visual grounding, but the paper does not itself propose it as such.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper constructs LocAnyMed-200K, a unified medical visual grounding dataset of 209,910 image-query-answer records spanning CT, ultrasound, X-ray, and optical imaging, together with a 20,000-record CoT rationale subset. It fine-tunes LocateAnything-3B, introduces modality-routed shared/expert visual FFNs, and reports F1@IoU 0.50 of 85.59 on same-set held-out validation versus 10.64 for the base model, and 33.63 on a cross-set test. The central claim is that large-scale unified medical grounding supervision, optionally combined with visual-evidence rationale supervision, equips a general grounding model with effective medical localization and improves cross-source generalization.
Significance. If the results hold, the dataset and training recipe are a useful contribution: they unify heterogeneous detection annotations into a common instruction format, support negative queries and point targets, and show that full-parameter domain adaptation of a general grounding model transfers to medical imaging. Strengths include careful metric definition (dataset-level matching, explicit absent-target handling), checkpoint curves, modality-wise and macro-F1 diagnostics, and the public release of code. The cross-set evaluation, built from disjoint source datasets and checked by source names, basenames, and perceptual hashes, is a particularly honest estimate of generalization. However, the headline same-set number is vulnerable to split leakage, and some model-selection choices are made on the test split, so the quantitative claims need reinforcement.
major comments (3)
- [S1.2, Table 1] S1.2 states that datasets without patient identifiers use a deterministic image-level split, and that Kvasir-SEG is split at image level. Kvasir-SEG is a colonoscopy video dataset; adjacent frames of the same polyp can be near-duplicates while having different file hashes. Because the abstract's headline F1 (10.64→85.59) and the checkpoint selection in Table S5 both use Same-set Det. Val., this split may measure memorization of specific polyps rather than generalization. I request a near-duplicate analysis between train and Same-set Det. Val. (perceptual hashes or frame distances), especially for Kvasir-SEG and any ultrasound/X-ray sources split at image level, plus F1 after removing leaked records. The Cross-set result (33.63) is not subject to this same-source leakage, but it cannot validate the 85.59 headline.
- [§4.3, Table 2] The 2:2 MoE allocation is chosen because it "performs best" on Cross-set Det. Test, and the rationale-augmented model is then evaluated on the same split. This is test-set selection: the reported cross-set gain from rationale supervision (+1.59 F1@0.50, +1.83 F1 Mean) is measured on the set used to pick the allocation/checkpoint. Differences among allocations are small (Cross F1 Mean 19.51 vs 18.39/18.39), and no variance estimates are given. Select the allocation on Same-set validation or a nested split, or report the full matrix of MoE allocations with rationale; otherwise the cross-source advantage of 2:2 and of rationale supervision is not established.
- [§4.2, Table 2, S4.4] No repeated runs, confidence intervals, or significance tests are reported. Several decisive differences are small: e.g., Same-set F1 Mean among MoE 3:1/2:2/7:1 is 69.17/68.87/69.02, and Cross F1 Mean is 19.51/18.39/18.39. The paper itself concedes in S4.4 that the results "should be treated as comparative trends rather than definitive rankings without repeated-run variance estimates." Since the abstract's central claim is a single number, the authors should provide at least 3 seeds for the main LocAnyMed and the rationale variant and report means and standard deviations. This is needed to support the ranking claims and the claimed improvements.
minor comments (5)
- [§3.3] Two references to "Figure 3" in this section describe the architecture and the modality-routed FFN, but Figure 3 in the main text is the dataset-composition figure; the architecture is Figure 2. Please correct the cross-references.
- [Tables 1, S4] Several numeric entries are missing spaces due to formatting, e.g., "27.499.2816.02", "33.631.5819.12", and "73.9253.47". Please fix the table formatting.
- [§3.2] The abstract says "approximately 200K" while the corpus contains exactly 209,910 records. State both numbers for precision.
- [S1.2] The split protocol for ultrasound and X-ray sources without patient identifiers is described only generically. For reproducibility, specify which sources have official/patient-level splits and which use the deterministic image-level split.
- [Table S4] Cross-set CT F1 Mean decreases from 6.60 (LocateAnything-3B) to 6.02 (LocAnyMed), even though F1@0.50 improves. This modality-specific caveat should be mentioned when summarizing cross-source generalization.
Circularity Check
No significant circularity: the central claims are empirical fine-tuning results measured against held-out data, not derivations that reduce to their inputs.
full rationale
The paper's main claims are empirical: fine-tuning LocateAnything-3B on LocAnyMed-200K improves F1@IoU 0.50 from 10.64 to 85.59 on Same-set Det. Val., and to 33.63 on Cross-set Det. Test. These are measurements on evaluation splits, not quantities derived by definition from the training data. No equation in the paper defines the reported F1 in terms of a fitted parameter or a self-citation. The CoT-20K subset is generated by GPT-5.5 as a teacher, but the original ground-truth coordinates are appended byte-for-byte and the grounding evaluation is independent of the teacher's rationales. The modality-routed experts are initialized from the original dense projection, so the architecture preserves the pretrained output at initialization; this is a construction fact, not a circular prediction. The paper's references to LocateAnything (Wang et al., 2026) are to a third-party base model, not to the authors' own prior work, so there is no load-bearing self-citation chain. One caveat, which is a generalization-validity concern rather than circularity, is that S4.2 selects the final checkpoint using Same-set Det. Val. F1 Mean, meaning the same-set numbers are somewhat optimistic as a held-out estimate; the cross-set evaluation provides a more independent anchor. This does not make the derivation circular, because the reported numbers still come from running the model on unseen records, not from re-stating the training objective or a fitted parameter. Overall, the derivation chain is self-contained and empirically grounded.
Assumptions & free parameters
free parameters (5)
- shared-to-expert channel allocation =
2:2 (ds=576, de=576)
- training checkpoint =
step 1500 for dense SFT and MoE
- target-present to absent-target ratio =
approximately 3:1
- CoT subset size =
20,000 records
- CoT rationale length =
20-45 words
assumptions (5)
- domain assumption Training and validation splits are independent and identically distributed; no same-patient leakage.
- domain assumption Source dataset annotations are complete enough that unannotated images can be treated as true negatives.
- domain assumption The imaging modality is known and reliable at inference.
- domain assumption Normalized coordinates in [0,1000] remain aligned because the model resizes without cropping.
- domain assumption GPT-5.5-generated rationales are a reliable supervision source.
Cite this review
Pith. "Pith review of LocAnyMed: Vision-Language Grounding for Multimodal Medical Images." pith.science (2026). https://pith.science/paper/Y6LVVNBE
@misc{pith2026260803322,
author = {Pith},
title = {Pith review of: LocAnyMed: Vision-Language Grounding for Multimodal Medical Images},
year = {2026},
howpublished = {\url{https://pith.science/paper/Y6LVVNBE}},
note = {Machine review of arXiv:2608.03322}
}
read the original abstract
Medical visual grounding connects free-form clinical queries to spatial evidence in medical images and is an important component of interpretable medical artificial intelligence. However, general-purpose grounding models are predominantly trained on natural images, while existing medical localization resources remain fragmented across imaging modalities, datasets, and task formulations. To address this gap, we construct LocAnyMed-200K, a multimodal medical visual grounding dataset containing approximately 200K image-query-answer examples across computed tomography, optical medical imaging, ultrasound, and X-ray. We harmonize heterogeneous detection and localization resources into a unified free-form instruction format that supports one or multiple bounding boxes, point coordinates, and no-target outputs for negative queries. Full-parameter fine-tuning of LocateAnything-3B on LocAnyMed-200K improves F1@IoU 0.50 from 10.64 to 85.59 on a held-out evaluation split, demonstrating that large-scale domain-specific supervision can equip a general grounding model with effective medical localization capabilities. Beyond spatial coordinates, a clinically interpretable grounding system should also communicate the evidence supporting its prediction. We therefore derive LocAnyMed-CoT-20K, a rationale-augmented subset that connects anatomical context, visual observations, and spatial conclusions through structured reasoning and further improves cross-source generalization through fine-tuning. Together, these resources provide a unified foundation for studying both localization accuracy and rationale quality across heterogeneous medical imaging modalities. The code is publicly available at https://github.com/MiliLab/LocAnyMed.
Reference graph
Works this paper leans on
-
[1]
International conference on medical image computing and computer-assisted intervention , pages=
nnDetection: a self-configuring method for medical object detection , author=. International conference on medical image computing and computer-assisted intervention , pages=. 2021 , organization=
work page 2021
-
[2]
International Conference on Medical Image Computing and Computer-Assisted Intervention , pages=
Medical phrase grounding with region-phrase context contrastive alignment , author=. International Conference on Medical Image Computing and Computer-Assisted Intervention , pages=. 2023 , organization=
work page 2023
-
[3]
IEEE Transactions on Pattern Analysis and Machine Intelligence , year=
Uncertainty-aware medical diagnostic phrase identification and grounding , author=. IEEE Transactions on Pattern Analysis and Machine Intelligence , year=
-
[4]
arXiv preprint arXiv:2605.27365 , year=
LocateAnything: Fast and high-quality vision-language grounding with parallel box decoding , author=. arXiv preprint arXiv:2605.27365 , year=
-
[5]
Nature communications , volume=
Segment anything in medical images , author=. Nature communications , volume=. 2024 , publisher=
work page 2024
-
[6]
A foundation model for joint segmentation, detection and recognition of biomedical objects across nine modalities , author=. Nature methods , volume=. 2025 , publisher=
work page 2025
-
[7]
international conference on medical image computing and computer-assisted intervention , pages=
M4oe: A foundation model for medical multimodal image segmentation with mixture of experts , author=. international conference on medical image computing and computer-assisted intervention , pages=. 2024 , organization=
work page 2024
-
[8]
arXiv preprint arXiv:2506.08356 , year=
Medmoe: modality-specialized mixture of experts for medical vision-language understanding , author=. arXiv preprint arXiv:2506.08356 , year=
Show all 49 references
-
[9]
arXiv preprint arXiv:2506.04034 , year=
Rex-thinker: Grounded object referring via chain-of-thought reasoning , author=. arXiv preprint arXiv:2506.04034 , year=
-
[10]
arXiv preprint arXiv:2510.04477 , year=
MedCLM: Learning to Localize and Reason via a CoT-Curriculum in Medical Vision-Language Models , author=. arXiv preprint arXiv:2510.04477 , year=
-
[11]
Journal of medical imaging , volume=
DeepLesion: automated mining of large-scale lesion annotations and universal lesion detection with deep learning , author=. Journal of medical imaging , volume=. 2018 , publisher=
2018
-
[12]
Proceedings of the IEEE/CVF international conference on computer vision , pages=
Mdetr-modulated detection for end-to-end multi-modal understanding , author=. Proceedings of the IEEE/CVF international conference on computer vision , pages=
-
[13]
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
Grounded language-image pre-training , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=
-
[14]
European conference on computer vision , pages=
Simple open-vocabulary object detection , author=. European conference on computer vision , pages=. 2022 , organization=
2022
-
[15]
European conference on computer vision , pages=
Grounding dino: Marrying dino with grounded pre-training for open-set object detection , author=. European conference on computer vision , pages=. 2024 , organization=
2024
-
[16]
International Conference on Learning Representations , volume=
Grounding multimodal large language models to the world , author=. International Conference on Learning Representations , volume=
-
[17]
arXiv preprint arXiv:2510.12798 , year=
Detect anything via next point prediction , author=. arXiv preprint arXiv:2510.12798 , year=
-
[18]
2026 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages=
MedROV: Towards Real-Time Open-Vocabulary Detection Across Diverse Medical Imaging Modalities , author=. 2026 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages=. 2026 , organization=
2026
-
[19]
arXiv preprint arXiv:2410.12694 , year=
Vividmed: Vision language model with versatile visual grounding for medicine , author=. arXiv preprint arXiv:2410.12694 , year=
-
[20]
Proceedings of the Computer Vision and Pattern Recognition Conference , pages=
MIMO: A medical vision language model with visual referring multimodal input and pixel grounding multimodal output , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=
-
[21]
Proceedings of the AAAI Conference on Artificial Intelligence , volume=
Towards a multimodal large language model with pixel-level insight for biomedicine , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
-
[22]
Nature Communications , year=
A universal foundation model for grounded biomedical image interpretation , author=. Nature Communications , year=
-
[23]
European conference on computer vision , pages=
Making the most of text semantics to improve biomedical vision--language processing , author=. European conference on computer vision , pages=. 2022 , organization=
2022
-
[24]
arXiv preprint arXiv:2601.06847 , year=
MedGround: Bridging the Evidence Gap in Medical Vision-Language Models with Verified Grounding Data , author=. arXiv preprint arXiv:2601.06847 , year=
-
[25]
Advances in Neural Information Processing Systems , volume=
Uni-med: a unified medical generalist foundation model for multi-task learning via connector-MoE , author=. Advances in Neural Information Processing Systems , volume=
-
[26]
arXiv preprint arXiv:2505.14231 , year=
Univg-r1: Reasoning guided universal visual grounding with reinforcement learning , author=. arXiv preprint arXiv:2505.14231 , year=
-
[27]
Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=
Gemex: A large-scale, groundable, and explainable medical vqa benchmark for chest x-ray diagnosis , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=
-
[28]
arXiv preprint arXiv:2603.06665 , year=
Better eyes, better thoughts: Why vision chain-of-thought fails in medicine , author=. arXiv preprint arXiv:2603.06665 , year=
-
[29]
Nature communications , volume=
The medical segmentation decathlon , author=. Nature communications , volume=. 2022 , publisher=
2022
-
[30]
Radiology: Artificial Intelligence , volume=
TotalSegmentator: robust segmentation of 104 anatomic structures in CT images , author=. Radiology: Artificial Intelligence , volume=. 2023 , publisher=
2023
-
[31]
Medical image analysis , volume=
Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challenge , author=. Medical image analysis , volume=. 2017 , publisher=
2017
-
[32]
International Conference on Medical Image Computing and Computer-Assisted Intervention , pages=
Beyond adapting SAM: Towards end-to-end ultrasound image segmentation via auto prompting , author=. International Conference on Medical Image Computing and Computer-Assisted Intervention , pages=. 2024 , organization=
2024
-
[33]
Medical physics , volume=
BUS-BRA: A breast ultrasound dataset for assessing computer-aided diagnosis systems , author=. Medical physics , volume=. 2024 , publisher=
2024
-
[34]
Scientific data , volume=
A pediatric wrist trauma X-ray dataset (GRAZPEDWRI-DX) for machine learning , author=. Scientific data , volume=. 2022 , publisher=
2022
-
[35]
Scientific data , volume=
Fracatlas: A dataset for fracture classification, localization and segmentation of musculoskeletal radiographs , author=. Scientific data , volume=. 2023 , publisher=
2023
-
[36]
arXiv preprint arXiv:2305.19112 , year=
Dentex: An abnormal tooth detection with dental enumeration and diagnosis benchmark for panoramic x-rays , author=. arXiv preprint arXiv:2305.19112 , year=
-
[37]
International conference on multimedia modeling , pages=
Kvasir-seg: A segmented polyp dataset , author=. International conference on multimedia modeling , pages=. 2019 , organization=
2019
-
[38]
saliency maps from physicians , author=
WM-DOVA maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians , author=. Computerized medical imaging and graphics , volume=. 2015 , publisher=
2015
-
[39]
Information Sciences , volume=
Diagnostic assessment of deep learning algorithms for diabetic retinopathy screening , author=. Information Sciences , volume=. 2019 , publisher=
2019
-
[40]
IEEE transactions on medical imaging , volume=
Ridge-based vessel segmentation in color images of the retina , author=. IEEE transactions on medical imaging , volume=. 2004 , publisher=
2004
-
[41]
IEEE Transactions on Medical imaging , volume=
Locating blood vessels in retinal images by piecewise threshold probing of a matched filter response , author=. IEEE Transactions on Medical imaging , volume=. 2000 , publisher=
2000
-
[42]
European conference on computer vision , pages=
End-to-end object detection with transformers , author=. European conference on computer vision , pages=. 2020 , organization=
2020
-
[43]
arXiv preprint arXiv:2606.03748 , year=
Ultralytics YOLO26: unified real-time end-to-end vision models , author=. arXiv preprint arXiv:2606.03748 , year=
-
[44]
arXiv preprint arXiv:2406.03459 , year=
Lw-detr: A transformer replacement to yolo for real-time detection , author=. arXiv preprint arXiv:2406.03459 , year=
-
[45]
arXiv preprint arXiv:2603.18739 , year=
EdgeCrafter: Compact ViTs for Edge Dense Prediction via Task-Specialized Distillation , author=. arXiv preprint arXiv:2603.18739 , year=
-
[46]
arXiv preprint arXiv:2412.10302 , year=
Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding , author=. arXiv preprint arXiv:2412.10302 , year=
-
[47]
arXiv preprint arXiv:2512.17436 , year=
Xiaomi MiMo-VL-Miloco Technical Report , author=. arXiv preprint arXiv:2512.17436 , year=
-
[48]
5 technical report , author=
Ovis2. 5 technical report , author=. arXiv preprint arXiv:2508.11737 , year=
-
[49]
arXiv preprint arXiv:2511.21631 , year=
Qwen3-vl technical report , author=. arXiv preprint arXiv:2511.21631 , year=
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.