Pith. sign in

REVIEW 2 major objections 4 minor 2 cited by

MedSG-Bench: A Benchmark for Medical Image Sequences Grounding

T0 review · 2 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A benchmark for grounding across medical image sequences shows current MLLMs stay near 20% IoU zero-shot, while instruction tuning on the same task templates raises a 7B model to 72.55% IoU.

desk verdict MedSG-Bench fills a real gap—sequential medical grounding—but the SOTA claim for MedSeq-Grounder is uninterpretable until the authors disclose a train/test split between MedSG-188K and MedSG-Bench. read the letter →

arxiv 2505.11852 v1 pith:WMPMRD5P submitted 2025-05-17 cs.CV

Jingkun Yue , Siqi Zhang , Zinan Jia , Huihuan Xu , Zongbo Han , Xiaohong Liu , Guangyu Wang This is my paper · ORCID
classification cs.CV
keywords medicalimagesequencesvisualgroundingmultimodallargelanguagemodelsbenchmarkdifferenceconsistencyinstructiontuningIoUevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper attempts to establish that visual grounding across sequences of medical images is a distinct, underevaluated capability that current multimodal large language models largely lack. It introduces MedSG-Bench, a benchmark of 9,630 VQA-style question–answer pairs built from 76 public datasets and 10 imaging modalities, organized into two paradigms: difference grounding (locating what changed between two images) and consistency grounding (locating the same structure, tracking an object, matching a patch, or grounding a language reference across views, frames, or modalities). On this benchmark, the strongest zero-shot model averages only 20.29% IoU and 11.39% accuracy at IoU 0.5, which the paper takes as evidence that sequence-level grounding is unsolved. The paper then builds MedSG-188K, an instruction-tuning dataset from the same eight task templates, and reports that a 7B model fine-tuned on it, MedSeq-Grounder, reaches 72.55% average IoU and 79.71% Acc@0.5, establishing a state of the art for the benchmark.

What carries the argument

The load-bearing mechanism is the VQA-style grounding template that underlies all eight tasks. Each template takes a sequence of medical images (length two to six), a natural-language prompt drawn from one of two task families—Image Difference Grounding for localized changes and Image Consistency Grounding for invariant or shared semantics—and a ground-truth bounding box, so every sample is scored by the same IoU and Acc@0.5 metrics. The same templates are reused, with expanded phrasing and registered versus non-registered shifts, multi-view slices, tracked frames, extracted patches, cross-modal pairs, and referring expressions, to generate the 188,163 training samples of MedSG-188K, which is what lets MedSeq-Grounder learn the task distribution rather than only evaluate it.

What would settle it

To test the central claim, compute the maximum IoU between each MedSG-Bench evaluation sample's images and every MedSG-188K training sample's images, or compare their source-case identifiers; if any training-evaluation pair is the same or near-identical, the MedSeq-Grounder numbers in the main table are inflated. A cleaner falsifier is to retrain MedSeq-Grounder on MedSG-188K with any samples sharing a source case with the evaluation set removed, then check whether the 72.55% average IoU survives.

Watch

Extended reading notes

Core claim

The central discovery is reported as a capability gap plus a recipe: no existing benchmark measures grounding on medical image sequences, and no evaluated multimodal large language model can do it well zero-shot, while a model tuned on a purpose-built instruction dataset can. The paper's claim is that a benchmark organized around two grounding paradigms—what changes across images and what stays the same across images—captures the clinically relevant forms of sequential reasoning, from pre/post-treatment comparison to cross-view and cross-modal alignment. The claimed result is that all zero-shot models, including a specialized multi-image grounding model, stay near 20% average IoU, whereas MedSeq-Grounder, a 7B multimodal model fine-tuned on MedSG-188K, reaches 72.55% average IoU.

Load-bearing premise

The load-bearing assumption is that MedSG-188K, the training set used to produce MedSeq-Grounder, contains no images that also appear in the MedSG-Bench evaluation set; if the two overlap, the reported state-of-the-art scores are not a fair measurement.

Editorial extensions

If this is right

  • If the benchmark's zero-shot results hold, medical sequence grounding is far from solved even for models that perform well on single-image medical VQA.
  • If the MedSeq-Grounder result holds, fine-tuning on grounding-supervised instruction data derived from the same task templates is a sufficient path to strong sequential grounding performance.
  • The paper's comparison suggests that medical-domain fine-tuning can hurt grounding, so future medical MLLMs should retain or include spatial grounding supervision.
  • The benchmark's task taxonomy—difference grounding versus consistency grounding—gives a common scoring ground for comparing future medical multi-image models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We infer a possible overlap issue: Section 4.1 says MedSG-188K is based on the eight MedSG-Bench tasks and the same source datasets, while Section 5.3 reports state-of-the-art results without stating a train/eval split; if training samples overlap with evaluation samples, the reported 72.55% IoU is optimistic.
  • A testable extension would be to measure how much of MedSeq-Grounder's gain comes from the task templates rather than from dataset scale, by training on a smaller or template-restricted subset and checking whether IoU degrades.
  • We also infer that the two-paradigm taxonomy could transfer outside medicine: the same difference-versus-consistency grounding split could be used to benchmark temporal grounding in natural video or multi-view imagery.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. MedSG-Bench introduces a benchmark for visual grounding in medical image sequences, built from 76 public datasets and organized around eight VQA-style tasks in two paradigms: Image Difference Grounding and Image Consistency Grounding. The authors evaluate a range of general-purpose and medical-specialized MLLMs in a zero-shot setting, reporting uniformly low performance (best zero-shot model, Migician, reaches 20.29 average IoU and 11.39 Acc@0.5). They also construct MedSG-188K, an instruction-tuning dataset, and fine-tune Qwen2.5-VL-7B to obtain MedSeq-Grounder, which reaches 72.55 average IoU and 79.71 Acc@0.5 on MedSG-Bench and is claimed as state of the art.

Significance. The benchmark fills a real gap: existing medical visual grounding benchmarks are single-image, while longitudinal, multi-view, and cross-modal comparisons are clinically important. The task taxonomy, the breadth of source datasets and modalities, and the public release of the benchmark and model are concrete strengths. The zero-shot evaluation of many MLLMs is also a useful falsifiable measurement. However, the state-of-the-art claim for MedSeq-Grounder is only interpretable if the MedSG-188K training data is disjoint from the MedSG-Bench test data; the paper currently does not establish this, so the headline result is unverifiable as written.

major comments (2)
  1. [§4.1, §5.3, Table 3] The paper never states that MedSG-188K training samples are disjoint from MedSG-Bench evaluation samples. Both resources are generated from the same 76 public datasets and the same eight task templates, and §4.1 explicitly says MedSG-188K is 'based on the eight tasks defined by MedSG-Bench.' There is no validation split, no patient-level or sequence-level deduplication between the two resources, and no statement that benchmark sequences were excluded from the 188,163 training samples. Without such a split, the 72.55 IoU / 79.71 Acc@0.5 row in Table 3 may reflect memorization or near-duplicate leakage rather than generalization, and the state-of-the-art claim in §5.3 is not a measurable result. The authors should specify the exact overlap-removal procedure, publish the split, and re-report MedSeq-Grounder on a verified held-out subset.
  2. [§5.3, Table 3] The 'state-of-the-art' comparison is also apples-to-oranges: MedSeq-Grounder is fine-tuned on in-domain instruction data derived from the task definitions of MedSG-Bench, while all other models are evaluated zero-shot. Even with a clean train/test split, this does not establish state of the art over the baselines; it establishes that in-domain supervised tuning improves over zero-shot MLLMs. The authors should either fine-tune the strongest baselines on the same MedSG-188K training split and compare them on the held-out evaluation set, or explicitly reframe the MedSeq-Grounder result as a supervised upper bound rather than a state-of-the-art comparison.
minor comments (4)
  1. [§5.3] The text refers to 'MedSeqVG-188K', but the dataset is named MedSG-188K throughout the rest of the paper; the names should be unified.
  2. [Table 3] The column header does not label the paired IoU and Acc@0.5 values under each task; the reader must infer that each task cell contains two numbers. Adding explicit subheaders or formatting the rows as IoU/Acc@0.5 would improve readability.
  3. [§3.3] There is a duplicated word in 'spans 10 medical imaging modalities and and encompasses 114 distinct clinical tasks'; the extra 'and' should be removed.
  4. [§5.1] The evaluation protocol is described only as 'models were prompted to perform inference without access to in-context examples.' The paper should provide the exact prompt template and the parsing rule used to extract predicted bounding boxes, since both affect the reported IoU numbers and reproducibility.

Circularity Check

1 steps flagged · score 5.0 of 10

MedSeq-Grounder's SOTA result rests on a train/test relationship that the paper never documents: MedSG-188K is built from the same eight MedSG-Bench tasks and the same 76 datasets, and MedSeq-Grounder is then evaluated on MedSG-Bench.

  1. fitted input called prediction [Section 4.1 (MedSG-188K construction), Section 5.3 and Table 3 (MedSeq-Grounder results)]
    "The construction of MedSG-188K is based on the eight tasks defined by MedSG-Bench. ... With instruction tuning on our MedSeqVG-188K dataset, the proposed MedSeq-Grounder achieves state-of-the-art performance across all tasks, demonstrating its effectiveness and robustness in sequential medical visual grounding."

    MedSG-188K is generated from the same eight task definitions and the same 76 public datasets that constitute MedSG-Bench (Sections 3.3 and 4.1), yet the paper never states that MedSG-Bench evaluation samples are excluded from the 188,163 training samples, nor does it document any patient-level or sequence-level split. MedSeq-Grounder is then fine-tuned on MedSG-188K and evaluated on MedSG-Bench, producing the headline 72.55 average IoU and 79.71 Acc@0.5 in Table 3. If the evaluation sequences, or near-duplicates such as the same patient/sequence with a different instruction template, appear in training, the SOTA row is a fitting or memorization artifact rather than a measurable generalization result.

full rationale

The benchmark-side findings are independent external measurements and are not circular: the zero-shot evaluations of Migician, Qwen2.5-VL, HuatuoGPT-Vision, and the other baselines are computed from released checkpoints against MedSG-Bench, so those numbers are genuine benchmark outcomes. The circularity burden is specifically on the MedSeq-Grounder contribution. MedSG-188K is explicitly 'based on the eight tasks defined by MedSG-Bench,' both resources draw on the same 76 public datasets, and the paper provides no train/test split, no exclusion statement, and no deduplication between the 188,163 training samples and the 9,630 evaluation pairs before Table 3 reports state-of-the-art performance after instruction tuning. Consequently, the fine-tuned model's headline number is interpretable only under an unstated assumption of disjointness; if such a split exists, the performance claim is legitimate, but the paper must exhibit it. The zero-shot benchmark comparisons remain valid, so the overall circularity is partial rather than total.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The benchmark rests mainly on domain assumptions about annotation reliability, synthetic task validity, and metric interpretation, plus a handful of hand-chosen construction constants. No new physical or mathematical entities are introduced. The main unmet need is evidence that the training and test sets are disjoint.

free parameters (5)
  • random_translation_range = 20 pixels
    Task 2 applies a random shift of up to 20 pixels along each axis; this hand-chosen range sets task difficulty.
  • image_resize_dimension = 336x336
    All images are resized to 336x336 (Section 3.1.2); resolution affects how small targets appear.
  • patch_sample_and_select = 15 sampled, 5 selected
    Task 6 samples 15 patches and manually selects up to 5; the selection threshold is a design choice.
  • instruction_variants_per_task = 10
    GPT-4 expanded each seed template into 10 variants for MedSG-188K (Section 4.1).
  • training_steps_and_batch = 15000 steps, batch 64
    MedSeq-Grounder training schedule (Section 4.2); not fitted to data, but it affects the reported SOTA.
assumptions (5)
  • domain assumption Bounding boxes derived from segmentation masks are accurate enough for IoU-based evaluation.
    Section 3.1.1 keeps datasets with local annotations; no re-annotation or clinician verification is reported.
  • domain assumption Synthetic perturbations and 20-pixel translations model real clinical differences.
    Section 3.2.1 generates Task 1/2 via CutPaste and generative models; clinical validity is assumed.
  • domain assumption IoU and Acc@0.5 capture grounding quality.
    Section 5.1 and Appendix C adopt these standard metrics without justification for sequence grounding.
  • domain assumption The 76 datasets are mutually exclusive and clean.
    Section 3.1.1 states cross-referencing and manual review, but no verification set is provided.
  • domain assumption GPT-4 generated instruction variants are diverse and unambiguous.
    Section 4.1 describes template expansion; the 188K samples are not manually inspected.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MedSG-Bench: A Benchmark for Medical Image Sequences Grounding." pith.science (2026). https://pith.science/paper/WMPMRD5P

@misc{pith2026250511852,
  author       = {Pith},
  title        = {Pith review of: MedSG-Bench: A Benchmark for Medical Image Sequences Grounding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WMPMRD5P}},
  note         = {Machine review of arXiv:2505.11852}
}
read the original abstract

Visual grounding is essential for precise perception and reasoning in multimodal large language models (MLLMs), especially in medical imaging domains. While existing medical visual grounding benchmarks primarily focus on single-image scenarios, real-world clinical applications often involve sequential images, where accurate lesion localization across different modalities and temporal tracking of disease progression (e.g., pre- vs. post-treatment comparison) require fine-grained cross-image semantic alignment and context-aware reasoning. To remedy the underrepresentation of image sequences in existing medical visual grounding benchmarks, we propose MedSG-Bench, the first benchmark tailored for Medical Image Sequences Grounding. It comprises eight VQA-style tasks, formulated into two paradigms of the grounding tasks, including 1) Image Difference Grounding, which focuses on detecting change regions across images, and 2) Image Consistency Grounding, which emphasizes detection of consistent or shared semantics across sequential images. MedSG-Bench covers 76 public datasets, 10 medical imaging modalities, and a wide spectrum of anatomical structures and diseases, totaling 9,630 question-answer pairs. We benchmark both general-purpose MLLMs (e.g., Qwen2.5-VL) and medical-domain specialized MLLMs (e.g., HuatuoGPT-vision), observing that even the advanced models exhibit substantial limitations in medical sequential grounding tasks. To advance this field, we construct MedSG-188K, a large-scale instruction-tuning dataset tailored for sequential visual grounding, and further develop MedSeq-Grounder, an MLLM designed to facilitate future research on fine-grained understanding across medical sequential images. The benchmark, dataset, and model are available at https://huggingface.co/MedSG-Bench

Figures

Figures reproduced from arXiv: 2505.11852 by the authors.

Figure 1
Figure 1. Examples of medical image sequences grounding [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. An illustration of medical image sequences grounding tasks included in MedSG-Bench. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Overview of the MedSG-Bench construction protocol. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Proportions of image sequence length (left), data distribution across tasks (middle), and target-to-image size ratios (right) in MedSG-Bench. Task 5: Visual Concept Grounding In clinical scenarios, lesions can exhibit high variability in locations (e.g., across anatomi…
Figure 6
Figure 6. Figure 6: Proportions of image sequence length (left), data distribution across tasks (right) in MedSG-188K. C Evaluation Metric We evaluate model performance using two standard metrics: Intersection over Union (IoU) and Accuracy at IoU threshold 0.5 (Acc@0.5). These metrics are…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Generalize LMMs to Versatile Visual Modalities via Fabricated Modality Synthesis

    cs.CV 2026-07 conditional novelty 6.5 of 10

    Fabricating appearance-varied images and modality contexts from RGB lets LMMs improve perception and understanding on real and synthetic non-RGB modalities without in-modality training.

  2. LoMeVQA: A Comprehensive Benchmark for Longitudinal Medical VQA

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A 206K multi-task longitudinal medical VQA benchmark shows current MLLMs fail at temporal reasoning, while fine-tuned MedLong-8B sets a strong baseline.

Reference graph

Works this paper leans on

138 extracted references · 38 canonical work pages · cited by 2 Pith papers

  1. [1]

    https://tianchi.aliyun.com/dataset/83075, 2020

    Chestimage. https://tianchi.aliyun.com/dataset/83075, 2020

    2020

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Advance warning methodologies for covid-19 using chest x-ray images

    Mete Ahishali, Aysen Degerli, Mehmet Yamac, Serkan Kiranyaz, Muhammad EH Chowdhury, Khalid Hameed, Tahir Hamid, Rashid Mazhar, and Moncef Gabbouj. Advance warning methodologies for covid-19 using chest x-ray images. Ieee Access, 9:41052–41065, 2021

    2021

  4. [4]

    Dataset of breast ultrasound images

    Walid Al-Dhabyani, Mohammed Gomaa, Hussien Khaled, and Aly Fahmy. Dataset of breast ultrasound images. Data in brief, 28:104863, 2020

    2020

  5. [5]

    2017 robotic instrument segmentation challenge

    Max Allan, Alex Shvets, Thomas Kurmann, Zichen Zhang, Rahul Duggal, Yun-Hsuan Su, Nicola Rieke, Iro Laina, Niveditha Kalavakonda, Sebastian Bodenstedt, et al. 2017 robotic instrument segmentation challenge. arXiv preprint arXiv:1902.06426, 2019

    arXiv 2017

  6. [6]

    The medical segmentation decathlon

    Michela Antonelli, Annika Reinke, Spyridon Bakas, Keyvan Farahani, Annette Kopp- Schneider, Bennett A Landman, Geert Litjens, Bjoern Menze, Olaf Ronneberger, Ronald M Summers, et al. The medical segmentation decathlon. Nature communications, 13(1):4128, 2022

    2022

  7. [7]

    Neural segmentation of seeding rois (srois) for pre-surgical brain tractography

    Itzik Avital, Ilya Nelkenbaum, Galia Tsarfaty, Eli Konen, Nahum Kiryati, and Arnaldo Mayer. Neural segmentation of seeding rois (srois) for pre-surgical brain tractography. IEEE transactions on medical imaging, 39(5):1655–1667, 2019

    2019

  8. [8]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

Show all 138 references
  1. [9]

    Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features

    Spyridon Bakas, Hamed Akbari, Aristeidis Sotiras, Michel Bilello, Martin Rozycki, Justin S Kirby, John B Freymann, Keyvan Farahani, and Christos Davatzikos. Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features. Scientif...

    2017

  2. [10]

    Identifying the best machine learning algorithms for brain tumor segmentation, pro- gression assessment, and overall survival prediction in the brats challenge

    Spyridon Bakas, Mauricio Reyes, Andras Jakab, Stefan Bauer, Markus Rempfler, Alessan- dro Crimi, Russell Takeshi Shinohara, Christoph Berger, Sung Min Ha, Martin Rozycki, et al. Identifying the best machine learning algorithms for brain tumor segmentation, pro- gression assess...

    2018 arXiv

  3. [11]

    Maira-2: Grounded radiology report generation

    Shruthi Bannur, Kenza Bouzid, Daniel C Castro, Anton Schwaighofer, Anja Thieme, Sam Bond-Taylor, Maximilian Ilse, Fernando Pérez-García, Valentina Salvatelli, Harshita Sharma, et al. Maira-2: Grounded radiology report generation. arXiv preprint arXiv:2406.04449, 2024

    2024 arXiv

  4. [12]

    Wm-dova maps for accurate polyp highlighting in colonoscopy: Validation vs

    Jorge Bernal, F Javier Sánchez, Gloria Fernández-Esparrach, Debora Gil, Cristina Rodríguez, and Fernando Vilariño. Wm-dova maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians. Computerized medical imaging and graphics, 43:99–111, 2015

    2015

  5. [13]

    Comparative validation of polyp detection methods in video colonoscopy: results from the miccai 2015 endoscopic vision challenge

    Jorge Bernal, Nima Tajkbaksh, Francisco Javier Sanchez, Bogdan J Matuszewski, Hao Chen, Lequan Yu, Quentin Angermann, Olivier Romain, Bjørn Rustad, Ilangko Balasingham, et al. Comparative validation of polyp detection methods in video colonoscopy: results from the miccai 2015 ...

    2015

  6. [14]

    Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: is the problem solved? IEEE transactions on medical imaging, 37(11):2514–2525, 2018

    Olivier Bernard, Alain Lalande, Clement Zotti, Frederick Cervenansky, Xin Yang, Pheng-Ann Heng, Irem Cetin, Karim Lekadir, Oscar Camara, Miguel Angel Gonzalez Ballester, et al. Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: is t...

    2018

  7. [15]

    N. Bloch. Nci-isbi. https://www.cancerimagingarchive.net/analysis-result/ isbi-mr-prostate-2013/ , 2015

    2013

  8. [16]

    Making the most of text semantics to improve biomedical vision–language processing

    Benedikt Boecking, Naoto Usuyama, Shruthi Bannur, Daniel C Castro, Anton Schwaighofer, Stephanie Hyland, Maria Wetscherek, Tristan Naumann, Aditya Nori, Javier Alvarez-Valle, et al. Making the most of text semantics to improve biomedical vision–language processing. In European...

    2022

  9. [17]

    Padchest: A large chest x-ray image dataset with multi-label annotated reports

    Aurelia Bustos, Antonio Pertusa, Jose-Maria Salinas, and Maria De La Iglesia-Vaya. Padchest: A large chest x-ray image dataset with multi-label annotated reports. Medical image analysis, 66:101797, 2020

    2020

  10. [18]

    Lung segmentation in chest radiographs using anatomical atlases with nonrigid registration

    Sema Candemir, Stefan Jaeger, Kannappan Palaniappan, Jonathan P Musco, Rahul K Singh, Zhiyun Xue, Alexandros Karargyris, Sameer Antani, George Thoma, and Clement J McDonald. Lung segmentation in chest radiographs using anatomical atlases with nonrigid registration. IEEE transa...

    2013

  11. [19]

    Longitudinal multiple sclerosis lesion segmentation: resource and challenge

    Aaron Carass, Snehashis Roy, Amod Jog, Jennifer L Cuzzocreo, Elizabeth Magrath, Adrian Gherman, Julia Button, James Nguyen, Ferran Prados, Carole H Sudre, et al. Longitudinal multiple sclerosis lesion segmentation: resource and challenge. NeuroImage, 148:77–102, 2017

    2017

  12. [20]

    Huatuogpt-vision, towards injecting medical visual knowledge into multimodal llms at scale

    Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Ruifei Zhang, Zhenyang Cai, Ke Ji, et al. Huatuogpt-vision, towards injecting medical visual knowledge into multimodal llms at scale. arXiv preprint arXiv:2406.19280, 2024

    2024 arXiv

  13. [21]

    Shikra: Unleashing multimodal llm’s referential dialogue magic

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023

    2023 arXiv

  14. [22]

    Towards generalizable tumor synthesis

    Qi Chen, Xiaoxi Chen, Haorui Song, Zhiwei Xiong, Alan Yuille, Chen Wei, and Zongwei Zhou. Towards generalizable tumor synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11147–11158, 2024

    2024

  15. [23]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024

    2024 arXiv

  16. [24]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024

    2024 arXiv

  17. [25]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vis...

    2024

  18. [26]

    Medical phrase grounding with region-phrase context contrastive alignment

    Zhihao Chen, Yang Zhou, Anh Tran, Junting Zhao, Liang Wan, Gideon Su Kai Ooi, Lionel Tim-Ee Cheng, Choon Hua Thng, Xinxing Xu, Yong Liu, et al. Medical phrase grounding with region-phrase context contrastive alignment. In International Conference on Medical Image Computing and...

    2023

  19. [27]

    Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic)

    Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging ...

    2018 arXiv

  20. [28]

    Covid-19 image data collection: Prospective predictions are the future

    Joseph Paul Cohen, Paul Morrison, Lan Dao, Karsten Roth, Tim Q Duong, and Marzyeh Ghassemi. Covid-19 image data collection: Prospective predictions are the future. arXiv preprint arXiv:2006.11988, 2020

    2006 arXiv

  21. [29]

    Ctooth+: A large-scale dental cone beam computed tomography dataset and benchmark for tooth volume segmentation

    Weiwei Cui, Yaqi Wang, Yilong Li, Dan Song, Xingyong Zuo, Jiaojiao Wang, Yifan Zhang, Huiyu Zhou, Bung san Chong, Liaoyuan Zeng, et al. Ctooth+: A large-scale dental cone beam computed tomography dataset and benchmark for tooth volume segmentation. In MICCAI Workshop on Data A...

    2022

  22. [30]

    Ctooth: a fully annotated 3d dataset and benchmark for tooth volume segmentation on cone beam computed tomography images

    Weiwei Cui, Yaqi Wang, Qianni Zhang, Huiyu Zhou, Dan Song, Xingyong Zuo, Gangyong Jia, and Liaoyuan Zeng. Ctooth: a fully annotated 3d dataset and benchmark for tooth volume segmentation on cone beam computed tomography images. In International Conference on Intelligent Roboti...

    2022

  23. [31]

    Reliable covid-19 detection using chest x-ray images

    Aysen Degerli, Mete Ahishali, Serkan Kiranyaz, Muhammad EH Chowdhury, and Moncef Gabbouj. Reliable covid-19 detection using chest x-ray images. In 2021 IEEE International Conference on Image Processing (ICIP), pages 185–189. IEEE, 2021

    2021

  24. [32]

    Covid-19 infection map generation and detection from chest x-ray images

    Aysen Degerli, Mete Ahishali, Mehmet Yamac, Serkan Kiranyaz, Muhammad EH Chowdhury, Khalid Hameed, Tahir Hamid, Rashid Mazhar, and Moncef Gabbouj. Covid-19 infection map generation and detection from chest x-ray images. Health information science and systems, 9(1):15, 2021

    2021

  25. [33]

    Osegnet: Operational segmentation network for covid-19 detection using chest x-ray images

    Aysen Degerli, Serkan Kiranyaz, Muhammad EH Chowdhury, and Moncef Gabbouj. Osegnet: Operational segmentation network for covid-19 detection using chest x-ray images. In 2022 IEEE International Conference on Image Processing (ICIP), pages 2306–2310. IEEE, 2022

    2022

  26. [34]

    Ctspine1k: A large-scale dataset for spinal vertebrae segmentation in computed tomography

    Yang Deng, Ce Wang, Yuan Hui, Qian Li, Jun Li, Shiwei Luo, Mengke Sun, Quan Quan, Shuxin Yang, You Hao, et al. Ctspine1k: A large-scale dataset for spinal vertebrae segmentation in computed tomography. arXiv preprint arXiv:2105.14711, 2021

    2021 arXiv

  27. [35]

    Automatic atlas-based segmentation of the prostate: A miccai 2009 prostate segmentation challenge entry

    Jason Dowling, Jurgen Fripp, Peter Greer, Sébastien Ourselin, and Olivier Salvado. Automatic atlas-based segmentation of the prostate: A miccai 2009 prostate segmentation challenge entry. Worskshop in Med Image Comput Comput Assist Interv, 24:17–24, 2009

    2009

  28. [36]

    Joint optic disc and cup segmentation based on multi-label deep network and polar transformation

    Huazhu Fu, Jun Cheng, Yanwu Xu, Damon Wing Kee Wong, Jiang Liu, and Xiaochun Cao. Joint optic disc and cup segmentation based on multi-label deep network and polar transformation. IEEE transactions on medical imaging, 37(7):1597–1605, 2018

    2018

  29. [37]

    Palm: Pathologic myopia challenge

    Huazhu Fu, Fei Li, José Ignacio Orlando, Hrvoje Bogunovic, Xu Sun, Jingan Liao, Yanwu Xu, Shaochong Zhang, and Xiulan Zhang. Palm: Pathologic myopia challenge. (No Title), 2019

    2019

  30. [38]

    Age challenge: angle closure glaucoma evaluation in anterior segment optical coherence tomography

    Huazhu Fu, Fei Li, Xu Sun, Xingxing Cao, Jingan Liao, Jose Ignacio Orlando, Xing Tao, Yuex- iang Li, Shihao Zhang, Mingkui Tan, et al. Age challenge: angle closure glaucoma evaluation in anterior segment optical coherence tomography. Medical Image Analysis, 66:101798, 2020

    2020

  31. [39]

    Rim-one: An open retinal image database for optic nerve evaluation

    Francisco Fumero, Silvia Alayón, José L Sanchez, Jose Sigut, and M Gonzalez-Hernandez. Rim-one: An open retinal image database for optic nerve evaluation. In2011 24th international symposium on computer-based medical systems (CBMS), pages 1–6. IEEE, 2011

    2011

  32. [40]

    Bayeseg: Bayesian modeling for medical image segmentation with interpretable generalizability

    Shangqi Gao, Hangqi Zhou, Yibo Gao, and Xiahai Zhuang. Bayeseg: Bayesian modeling for medical image segmentation with interpretable generalizability. Medical Image Analysis, 89:102889, 2023

    2023

  33. [41]

    Thyroid nodule segmentation and classification in ultrasound images

    HM Gireesha and S Nanda. Thyroid nodule segmentation and classification in ultrasound images. International Journal of Engineering Research and Technology, 2014

    2014

  34. [42]

    Bagls, a multihospital benchmark for automatic glottis segmentation

    Pablo Gómez, Andreas M Kist, Patrick Schlegel, David A Berry, Dinesh K Chhetri, Stephan Dürr, Matthias Echternach, Aaron M Johnson, Stefan Kniesburges, Melda Kunduk, et al. Bagls, a multihospital benchmark for automatic glottis segmentation. Scientific data, 7(1):186, 2020

    2020

  35. [43]

    Computer aided detection for pulmonary embolism challenge (cad-pe)

    Germán González, Daniel Jimenez-Carretero, Sara Rodríguez-López, Carlos Cano-Espinosa, Miguel Cazorla, Tanya Agarwal, Vinit Agarwal, Nima Tajbakhsh, Michael B Gotway, Jianming Liang, et al. Computer aided detection for pulmonary embolism challenge (cad-pe). arXiv preprint arXi...

    2003 arXiv

  36. [44]

    Segmentation of knee images: a grand challenge

    Tobias Heimann, Bryan J Morrison, Martin A Styner, Marc Niethammer, and Simon Warfield. Segmentation of knee images: a grand challenge. In Proc. MICCAI Workshop on Medical Image Analysis for the Clinic, volume 1. Beijing, China, 2010

    2010

  37. [45]

    Isles 2022: A multi-center magnetic resonance imaging stroke lesion segmentation dataset

    Moritz R Hernandez Petzsche, Ezequiel de la Rosa, Uta Hanning, Roland Wiest, Waldo Valenzuela, Mauricio Reyes, Maria Meyer, Sook-Lei Liew, Florian Kofler, Ivan Ezhov, et al. Isles 2022: A multi-center magnetic resonance imaging stroke lesion segmentation dataset. Scientific da...

    2022

  38. [46]

    The endotect 2020 challenge: evaluation and comparison of classification, segmentation and inference time for endoscopy

    Steven A Hicks, Debesh Jha, Vajira Thambawita, Pål Halvorsen, Hugo L Hammer, and Michael A Riegler. The endotect 2020 challenge: evaluation and comparison of classification, segmentation and inference time for endoscopy. In Pattern Recognition. ICPR International 12 Workshops ...

    2020

  39. [47]

    Computed tomography images for intracranial hemorrhage detection and segmentation

    Murtadha Hssayeni, M Croock, A Salman, H Al-khafaji, Z Yahya, and B Ghoraani. Computed tomography images for intracranial hemorrhage detection and segmentation. Intracranial hemorrhage segmentation using a deep convolutional model. Data, 5(1):14, 2020

    2020

  40. [48]

    Label-free liver tumor segmentation

    Qixin Hu, Yixiong Chen, Junfei Xiao, Shuwen Sun, Jieneng Chen, Alan L Yuille, and Zongwei Zhou. Label-free liver tumor segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7422–7432, 2023

    2023

  41. [49]

    Domain specific convolution and high frequency reconstruction based unsupervised domain adaptation for medical image segmentation

    Shishuai Hu, Zehui Liao, and Yong Xia. Domain specific convolution and high frequency reconstruction based unsupervised domain adaptation for medical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 650–659. S...

    2022

  42. [50]

    Medical-diff-vqa: a large-scale medical dataset for difference visual question answering on chest x-ray images

    Xinyue Hu, L Gu, Q An, M Zhang, L Liu, K Kobayashi, T Harada, R Summers, and Y Zhu. Medical-diff-vqa: a large-scale medical dataset for difference visual question answering on chest x-ray images. PhysioNet, 12:13, 2023

    2023

  43. [51]

    Om- nimedvqa: A new large-scale comprehensive evaluation benchmark for medical lvlm

    Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, and Ping Luo. Om- nimedvqa: A new large-scale comprehensive evaluation benchmark for medical lvlm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22170–22183, 2024

    2024

  44. [52]

    Towards a multimodal large language model with pixel-level insight for biomedicine

    Xiaoshuang Huang, Lingdong Shen, Jia Liu, Fangxin Shang, Hongxiang Li, Haifeng Huang, and Yehui Yang. Towards a multimodal large language model with pixel-level insight for biomedicine. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 3779–3787, 2025

    2025

  45. [53]

    Automatic tuberculosis screening using chest radiographs

    Stefan Jaeger, Alexandros Karargyris, Sema Candemir, Les Folio, Jenifer Siegelman, Fiona Callaghan, Zhiyun Xue, Kannappan Palaniappan, Rahul K Singh, Sameer Antani, et al. Automatic tuberculosis screening using chest radiographs. IEEE transactions on medical imaging, 33(2):233...

    2013

  46. [54]

    Hugging face

    Shashank Mohan Jain. Hugging face. In Introduction to transformers for NLP: With the hugging face library and models to solve problems, pages 51–67. Springer, 2022

    2022

  47. [55]

    Kvasir-instrument: Diagnostic and therapeutic tool segmentation dataset in gastrointestinal endoscopy

    Debesh Jha, Sharib Ali, Krister Emanuelsen, Steven A Hicks, Vajira Thambawita, Enrique Garcia-Ceja, Michael A Riegler, Thomas De Lange, Peter T Schmidt, Håvard D Johansen, et al. Kvasir-instrument: Diagnostic and therapeutic tool segmentation dataset in gastrointestinal endosc...

    2021

  48. [56]

    Amos: A large-scale abdominal multi-organ bench- mark for versatile medical image segmentation

    Yuanfeng Ji, Haotian Bai, Chongjian Ge, Jie Yang, Ye Zhu, Ruimao Zhang, Zhen Li, Lingyan Zhanng, Wanling Ma, Xiang Wan, et al. Amos: A large-scale abdominal multi-organ bench- mark for versatile medical image segmentation. Advances in neural information processing systems, 35:...

    2022

  49. [57]

    Mantis: Interleaved multi-image instruction tuning

    Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi-image instruction tuning. arXiv preprint arXiv:2405.01483, 2024

    2024 arXiv

  50. [58]

    Chatrex: Taming multimodal llm for joint perception and understanding

    Qing Jiang, Gen Luo, Yuqin Yang, Yuda Xiong, Yihao Chen, Zhaoyang Zeng, Tianhe Ren, and Lei Zhang. Chatrex: Taming multimodal llm for joint perception and understanding. arXiv preprint arXiv:2411.18363, 2024

    2024 arXiv

  51. [59]

    Zhouqiang Jiang. 4c2021. https://aistudio.baidu.com/datasetdetail/89548, 2021

    2021

  52. [60]

    Xr-msf-unet: Automatic segmentation model for covid-19 lung ct images

    XIE Juanying and ZHANG Kaiyun. Xr-msf-unet: Automatic segmentation model for covid-19 lung ct images. Journal of Frontiers of Computer Science & Technology, 16(8), 2022

    2022

  53. [61]

    Covid-19 ct lung and infection segmentation dataset

    Ma Jun, Ge Cheng, Wang Yixin, An Xingle, Gao Jiantao, Yu Ziqi, Zhang Minqing, Liu Xin, Deng Xueyuan, Cao Shucheng, et al. Covid-19 ct lung and infection segmentation dataset. (No Title), 2020

    2020

  54. [62]

    Algorithms for left atrial wall segmentation and thickness–evaluation on an open-source ct and mri image database

    Rashed Karim, Lauren-Emma Blake, Jiro Inoue, Qian Tao, Shuman Jia, R James Housden, Pranav Bhagirath, Jean-Luc Duval, Marta Varela, Jonathan M Behar, et al. Algorithms for left atrial wall segmentation and thickness–evaluation on an open-source ct and mri image database. Medic...

    2018

  55. [63]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015–4026, 2023

    2023

  56. [64]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer ...

    2017

  57. [65]

    Standardized assessment of automatic segmentation of white matter hyperintensities and results of the wmh segmentation challenge

    Hugo J Kuijf, J Matthijs Biesbroek, Jeroen De Bresser, Rutger Heinen, Simon Andermatt, Mariana Bento, Matt Berseth, Mikhail Belyaev, M Jorge Cardoso, Adria Casamitjana, et al. Standardized assessment of automatic segmentation of white matter hyperintensities and results of the...

    2019

  58. [66]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 9579–9589, 2024

    2024

  59. [67]

    Deep learning methods for automatic evaluation of delayed enhancement-mri

    Alain Lalande, Zhihao Chen, Thibaut Pommier, Thomas Decourselle, Abdul Qayyum, Michel Salomon, Dominique Ginhac, Youssef Skandarani, Arnaud Boucher, Khawla Brahim, et al. Deep learning methods for automatic evaluation of delayed enhancement-mri. the results of the emidec chall...

    2022

  60. [68]

    Segthor: Segmentation of thoracic organs at risk in ct images

    Zoé Lambert, Caroline Petitjean, Bernard Dubray, and Su Kuan. Segthor: Segmentation of thoracic organs at risk in ct images. In 2020 Tenth International Conference on Image Processing Theory, Tools and Applications (IPTA), pages 1–6. IEEE, 2020

    2020

  61. [69]

    A dataset of clinically generated visual questions and answers about radiology images

    Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1–10, 2018

    2018

  62. [70]

    Deep learning for segmentation using an open large-scale dataset in 2d echocardiography

    Sarah Leclerc, Erik Smistad, Joao Pedrosa, Andreas Østvik, Frederic Cervenansky, Florian Espinosa, Torvald Espeland, Erik Andreas Rye Berg, Pierre-Marc Jodoin, Thomas Grenier, et al. Deep learning for segmentation using an open large-scale dataset in 2d echocardiography. IEEE ...

    2019

  63. [71]

    Computer-aided detection and diagnosis for prostate cancer based on mono and multi-parametric mri: a review

    Guillaume Lemaître, Robert Martí, Jordi Freixenet, Joan C Vilanova, Paul M Walker, and Fabrice Meriaudeau. Computer-aided detection and diagnosis for prostate cancer based on mono and multi-parametric mri: a review. Computers in biology and medicine, 60:8–31, 2015

    2015

  64. [72]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  65. [73]

    Cutpaste: Self-supervised learning for anomaly detection and localization

    Chun-Liang Li, Kihyuk Sohn, Jinsung Yoon, and Tomas Pfister. Cutpaste: Self-supervised learning for anomaly detection and localization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9664–9674, 2021

    2021

  66. [74]

    Development and clinical deployment of a smartphone- based visual field deep learning system for glaucoma detection.NPJ digital medicine, 3(1):123, 2020

    Fei Li, Diping Song, Han Chen, Jian Xiong, Xingyi Li, Hua Zhong, Guangxian Tang, Sujie Fan, Dennis SC Lam, Weihua Pan, et al. Development and clinical deployment of a smartphone- based visual field deep learning system for glaucoma detection.NPJ digital medicine, 3(1):123, 2020

    2020

  67. [75]

    Qubiq: Uncertainty quantification for biomedical image segmentation challenge.arXiv preprint arXiv:2405.18435, 2024

    Hongwei Bran Li, Fernando Navarro, Ivan Ezhov, Amirhossein Bayat, Dhritiman Das, Florian Kofler, Suprosanna Shit, Diana Waldmannstetter, Johannes C Paetzold, Xiaobin Hu, et al. Qubiq: Uncertainty quantification for biomedical image segmentation challenge.arXiv preprint arXiv:2...

    2024 arXiv

  68. [76]

    The state-of-the-art 3d anisotropic intracranial hemorrhage segmentation on non-contrast head ct: The instance challenge

    Xiangyu Li, Gongning Luo, Kuanquan Wang, Hongyu Wang, Jun Liu, Xinjie Liang, Jie Jiang, Zhenghao Song, Chunyue Zheng, Haokai Chi, et al. The state-of-the-art 3d anisotropic intracranial hemorrhage segmentation on non-contrast head ct: The instance challenge. arXiv preprint arX...

    2023 arXiv

  69. [77]

    Hematoma expansion context guided intracranial hemorrhage segmentation and uncertainty estimation

    Xiangyu Li, Gongning Luo, Wei Wang, Kuanquan Wang, Yue Gao, and Shuo Li. Hematoma expansion context guided intracranial hemorrhage segmentation and uncertainty estimation. IEEE Journal of Biomedical and Health Informatics, 26(3):1140–1151, 2021. 14

    2021

  70. [78]

    Migician: Revealing the magic of free-form multi-image grounding in multimodal large language models

    You Li, Heyu Huang, Chi Chen, Kaiyu Huang, Chao Huang, Zonghao Guo, Zhiyuan Liu, Jinan Xu, Yuhua Li, Ruixuan Li, et al. Migician: Revealing the magic of free-form multi-image grounding in multimodal large language models. arXiv preprint arXiv:2501.05767, 2025

    2025 arXiv

  71. [79]

    Evaluation of prostate segmentation algorithms for mri: the promise12 challenge

    Geert Litjens, Robert Toth, Wendy Van De Ven, Caroline Hoeks, Sjoerd Kerkstra, Bram Van Ginneken, Graham Vincent, Gwenael Guillard, Neil Birbeck, Jindang Zhang, et al. Evaluation of prostate segmentation algorithms for mri: the promise12 challenge. Medical image analysis, 18(2...

    2014

  72. [80]

    Slake: A semantically- labeled knowledge-enhanced dataset for medical visual question answering

    Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. Slake: A semantically- labeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th international symposium on biomedical imaging (ISBI), pages 1650–1654. IEEE, 2021

    2021

  73. [81]

    Deep learning to segment pelvic bones: large-scale ct datasets and baseline models

    Pengbo Liu, Hu Han, Yuanqi Du, Heqin Zhu, Yinhao Li, Feng Gu, Honghu Xiao, Jun Li, Chunpeng Zhao, Li Xiao, et al. Deep learning to segment pelvic bones: large-scale ct datasets and baseline models. International Journal of Computer Assisted Radiology and Surgery , 16:749–756, 2021

    2021

  74. [82]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chun- yuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision, pages 38–55. Springer, 2024

    2024

  75. [83]

    A vertebral segmentation dataset with fracture grading

    Maximilian T Löffler, Anjany Sekuboyina, Alina Jacob, Anna-Lena Grau, Andreas Scharr, Malek El Husseini, Mareike Kallweit, Claus Zimmer, Thomas Baum, and Jan S Kirschke. A vertebral segmentation dataset with fracture grading. Radiology: Artificial Intelligence , 2(4):e190138, 2020

    2020

  76. [84]

    Efficient automatic segmentation for multi-level pulmonary arteries: The parse challenge

    Gongning Luo, Kuanquan Wang, Jun Liu, Shuo Li, Xinjie Liang, Xiangyu Li, Shaowei Gan, Wei Wang, Suyu Dong, Wenyi Wang, et al. Efficient automatic segmentation for multi-level pulmonary arteries: The parse challenge. arXiv preprint arXiv:2304.03708, 2023

    2023 arXiv

  77. [85]

    Segrap2023: A benchmark of organs-at-risk and gross tumor volume segmentation for radiotherapy planning of nasopharyn- geal carcinoma

    Xiangde Luo, Jia Fu, Yunxin Zhong, Shuolin Liu, Bing Han, Mehdi Astaraki, Simone Ben- dazzoli, Iuliana Toma-Dasu, Yiwen Ye, Ziyang Chen, et al. Segrap2023: A benchmark of organs-at-risk and gross tumor volume segmentation for radiotherapy planning of nasopharyn- geal carcinoma...

    2025

  78. [86]

    Word: A large scale dataset, benchmark and clinical applicable study for abdominal organ segmentation from ct image

    Xiangde Luo, Wenjun Liao, Jianghong Xiao, Jieneng Chen, Tao Song, Xiaofan Zhang, Kang Li, Dimitris N Metaxas, Guotai Wang, and Shaoting Zhang. Word: A large scale dataset, benchmark and clinical applicable study for abdominal organ segmentation from ct image. Medical Image Ana...

    2022

  79. [87]

    X-metric: An n-dimensional information-theoretic framework for groupwise registration and deep combined computing

    Xinzhe Luo and Xiahai Zhuang. X-metric: An n-dimensional information-theoretic framework for groupwise registration and deep combined computing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(7):9206–9224, 2022

    2022

  80. [88]

    Groma: Localized visual tokenization for grounding multimodal large language models

    Chuofan Ma, Yi Jiang, Jiannan Wu, Zehuan Yuan, and Xiaojuan Qi. Groma: Localized visual tokenization for grounding multimodal large language models. In European Conference on Computer Vision, pages 417–435. Springer, 2024

    2024

  81. [89]

    Segment anything in medical images

    Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature Communications, 15(1):654, 2024

    2024

  82. [90]

    Abdomenct-1k: Is abdominal organ segmentation a solved problem? IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):6695– 6714, 2021

    Jun Ma, Yao Zhang, Song Gu, Cheng Zhu, Cheng Ge, Yichi Zhang, Xingle An, Congcong Wang, Qiyuan Wang, Xin Liu, et al. Abdomenct-1k: Is abdominal organ segmentation a solved problem? IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):6695– 6714, 2021

    2021

  83. [91]

    The multimodal brain tumor image segmentation benchmark (brats)

    Bjoern H Menze, Andras Jakab, Stefan Bauer, Jayashree Kalpathy-Cramer, Keyvan Farahani, Justin Kirby, Yuliya Burren, Nicole Porz, Johannes Slotboom, Roland Wiest, et al. The multimodal brain tumor image segmentation benchmark (brats). IEEE transactions on medical imaging, 34(1...

    1993

  84. [92]

    Mmxu: A multi-modal and multi-x-ray understanding dataset for disease progression

    Linjie Mu, Zhongzhen Huang, Shengqian Qin, Yakun Zhu, Shaoting Zhang, and Xiaofan Zhang. Mmxu: A multi-modal and multi-x-ray understanding dataset for disease progression. arXiv preprint arXiv:2502.11651, 2025

    2025 arXiv

  85. [93]

    Automatic segmentation of white matter tracts using multiple brain mri sequences

    Ilya Nelkenbaum, Galia Tsarfaty, Nahum Kiryati, Eli Konen, and Arnaldo Mayer. Automatic segmentation of white matter tracts using multiple brain mri sequences. In 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), pages 368–371. IEEE, 2020. 15

    2020

  86. [94]

    Vindr-mammo: A large-scale benchmark dataset for computer-aided diagnosis in full-field digital mammography

    Hieu T Nguyen, Ha Q Nguyen, Hieu H Pham, Khanh Lam, Linh T Le, Minh Dao, and Van Vu. Vindr-mammo: A large-scale benchmark dataset for computer-aided diagnosis in full-field digital mammography. Scientific Data, 10(1):277, 2023

    2023

  87. [95]

    Brain tumor mri dataset

    Msoud Nickparvar. Brain tumor mri dataset. Kaggle, 2021

    2021

  88. [96]

    chatgpto3

    OpenAI. chatgpto3. https://openai.com/index/thinking-with-images/ , 2025

    2025

  89. [97]

    Refuge challenge: A unified framework for evaluating automated methods for glaucoma assessment from fundus photographs

    José Ignacio Orlando, Huazhu Fu, João Barbosa Breda, Karel Van Keer, Deepti R Bathula, Andrés Diaz-Pinto, Ruogu Fang, Pheng-Ann Heng, Jeyoung Kim, JoonHo Lee, et al. Refuge challenge: A unified framework for evaluating automated methods for glaucoma assessment from fundus phot...

    2020

  90. [98]

    Interactive whole-heart segmentation in congenital heart disease

    Danielle F Pace, Adrian V Dalca, Tal Geva, Andrew J Powell, Mehdi H Moghari, and Polina Golland. Interactive whole-heart segmentation in congenital heart disease. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, ...

    2015

  91. [99]

    Dgmsnet: Spine segmentation for mr image by a detection-guided mixed-supervised segmentation network

    Shumao Pang, Chunlan Pang, Zhihai Su, Liyan Lin, Lei Zhao, Yangfan Chen, Yujia Zhou, Hai Lu, and Qianjin Feng. Dgmsnet: Spine segmentation for mr image by a detection-guided mixed-supervised segmentation network. Medical image analysis, 75:102261, 2022

    2022

  92. [100]

    Spineparsenet: spine parsing for volumetric mr image by a two-stage segmentation framework with semantic image representation

    Shumao Pang, Chunlan Pang, Lei Zhao, Yangfan Chen, Zhihai Su, Yujia Zhou, Meiyan Huang, Wei Yang, Hai Lu, and Qianjin Feng. Spineparsenet: spine parsing for volumetric mr image by a two-stage segmentation framework with semantic image representation. IEEE Transactions on Medic...

    2020

  93. [101]

    Kosmos-2: Grounding multimodal large language models to the world

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824, 2023

    2023 arXiv

  94. [102]

    Han-seg: The head and neck organ-at-risk ct and mr segmentation dataset

    Gašper Podobnik, Primož Strojan, Primož Peterlin, Bulat Ibragimov, and Tomaž Vrtovec. Han-seg: The head and neck organ-at-risk ct and mr segmentation dataset. Medical physics, 50(3):1917–1927, 2023

    1917

  95. [103]

    Glamm: Pixel grounding large multimodal model

    Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Re...

    2024

  96. [104]

    Evaluation of segmentation methods on head and neck ct: auto-segmentation challenge 2015

    Patrik F Raudaschl, Paolo Zaffino, Gregory C Sharp, Maria Francesca Spadea, Antong Chen, Benoit M Dawant, Thomas Albrecht, Tobias Gass, Christoph Langguth, Marcel Lüthi, et al. Evaluation of segmentation methods on head and neck ct: auto-segmentation challenge 2015. Medical ph...

    2015

  97. [105]

    Pixellm: Pixel reasoning with large multimodal model

    Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26374–26383, 2024

    2024

  98. [106]

    Ct-org, a new dataset for multiple organ segmentation in computed tomography

    Blaine Rister, Darvin Yi, Kaushik Shivakumar, Tomomi Nobashi, and Daniel L Rubin. Ct-org, a new dataset for multiple organ segmentation in computed tomography. Scientific Data, 7(1):381, 2020

    2020

  99. [107]

    Rapid artificial intelligence solutions in a pandemic—the covid-19-20 lung ct lesion segmentation challenge

    Holger R Roth, Ziyue Xu, Carlos Tor-Díez, Ramon Sanchez Jacob, Jonathan Zember, Jose Molto, Wenqi Li, Sheng Xu, Baris Turkbey, Evrim Turkbey, et al. Rapid artificial intelligence solutions in a pandemic—the covid-19-20 lung ct lesion segmentation challenge. Medical image analy...

    2022

  100. [108]

    Compar- ing algorithms for automated vessel segmentation in computed tomography scans of the lung: the vessel12 study

    Rina D Rudyanto, Sjoerd Kerkstra, Eva M Van Rikxoort, Catalin Fetita, Pierre-Yves Brillet, Christophe Lefevre, Wenzhe Xue, Xiangjun Zhu, Jianming Liang, Ilkay Öksüz, et al. Compar- ing algorithms for automated vessel segmentation in computed tomography scans of the lung: the v...

    2014

  101. [109]

    Verse: a vertebrae labelling and segmentation benchmark for multi-detector ct images

    Anjany Sekuboyina, Malek E Husseini, Amirhossein Bayat, Maximilian Löffler, Hans Liebl, Hongwei Li, Giles Tetteh, Jan Kukaˇcka, Christian Payer, Darko Štern, et al. Verse: a vertebrae labelling and segmentation benchmark for multi-detector ct images. Medical image analysis, 73...

    2021

  102. [110]

    Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the luna16 challenge

    Arnaud Arindra Adiyoso Setio, Alberto Traverso, Thomas De Bel, Moira SN Berens, Cas Van Den Bogaard, Piergiorgio Cerello, Hao Chen, Qi Dou, Maria Evelina Fantacci, Bram Geurts, et al. Validation, comparison, and combination of algorithms for automatic detection of pulmonary no...

    2017

  103. [111]

    Junji Shiraishi, Shigehiko Katsuragawa, Junpei Ikezoe, Tsuneo Matsumoto, Takeshi Kobayashi, Ken-ichi Komatsu, Mitate Matsui, Hiroshi Fujita, Yoshie Kodera, and Kunio Doi. Development of a digital image database for chest radiographs with and without a lung nodule: receiver ope...

    2000

  104. [112]

    A large annotated medical image dataset for the development and evaluation of segmentation algorithms

    Amber L Simpson, Michela Antonelli, Spyridon Bakas, Michel Bilello, Keyvan Farahani, Bram Van Ginneken, Annette Kopp-Schneider, Bennett A Landman, Geert Litjens, Bjoern Menze, et al. A large annotated medical image dataset for the development and evaluation of segmentation alg...

    1902 arXiv

  105. [113]

    Drishti-gs: Retinal image dataset for optic nerve head (onh) segmentation

    Jayanthi Sivaswamy, SR Krishnadas, Gopal Datt Joshi, Madhulika Jain, and A Ujjwaft Syed Tabish. Drishti-gs: Retinal image dataset for optic nerve head (onh) segmentation. In 2014 IEEE 11th international symposium on biomedical imaging (ISBI), pages 53–56. IEEE, 2014

    2014

  106. [114]

    The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions

    Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1):1–9, 2018

    2018

  107. [115]

    3d segmentation in the clinic: A grand challenge

    Bram Van Ginneken, Tobias Heimann, and Martin Styner. 3d segmentation in the clinic: A grand challenge. In MICCAI workshop on 3D segmentation in the clinic: a grand challenge, volume 1, pages 7–15, 2007

    2007

  108. [116]

    Benchmark on automatic six-month-old infant brain segmentation algorithms: the iseg-2017 challenge

    Li Wang, Dong Nie, Guannan Li, Élodie Puybareau, Jose Dolz, Qian Zhang, Fan Wang, Jing Xia, Zhengwang Wu, Jia-Wei Chen, et al. Benchmark on automatic six-month-old infant brain segmentation algorithms: the iseg-2017 challenge. IEEE transactions on medical imaging, 38(9):2219–2...

    2017

  109. [117]

    The extreme cardiac mri analysis challenge under respiratory motion (cmrxmotion)

    Shuo Wang, Chen Qin, Chengyan Wang, Kang Wang, Haoran Wang, Chen Chen, Cheng Ouyang, Xutong Kuang, Chengliang Dai, Yuanhan Mo, et al. The extreme cardiac mri analysis challenge under respiratory motion (cmrxmotion). arXiv preprint arXiv:2210.06385, 2022

    arXiv

  110. [118]

    Autolaparo: A new dataset of integrated multi-tasks for image-guided surgical automation in laparoscopic hysterectomy

    Ziyi Wang, Bo Lu, Yonghao Long, Fangxun Zhong, Tak-Hong Cheung, Qi Dou, and Yunhui Liu. Autolaparo: A new dataset of integrated multi-tasks for image-guided surgical automation in laparoscopic hysterectomy. In International Conference on Medical Image Computing and Computer-As...

    2022

  111. [119]

    Minimizing estimated risks on unlabeled data: A new formulation for semi-supervised medical image segmentation

    Fuping Wu and Xiahai Zhuang. Minimizing estimated risks on unlabeled data: A new formulation for semi-supervised medical image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):6021–6036, 2022

    2022

  112. [120]

    Freetumor: Large-scale generative tumor synthesis in computed tomography images for improving tumor recognition

    Linshan Wu, Jiaxin Zhuang, Yanning Zhou, Sunan He, Jiabo Ma, Luyang Luo, Xi Wang, Xuefeng Ni, Xiaoling Zhong, Mingxiang Wu, et al. Freetumor: Large-scale generative tumor synthesis in computed tomography images for improving tumor recognition. arXiv preprint arXiv:2502.18519, 2025

    2025 arXiv

  113. [121]

    Gsva: Generalized segmentation via multimodal large language models

    Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang. Gsva: Generalized segmentation via multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3858–3869, 2024

    2024

  114. [122]

    Towards visual grounding: A survey

    Linhui Xiao, Xiaoshan Yang, Xiangyuan Lan, Yaowei Wang, and Changsheng Xu. Towards visual grounding: A survey. arXiv preprint arXiv:2412.20206, 2024

    2024 arXiv

  115. [123]

    A global benchmark of algorithms for segmenting the left atrium from late gadolinium-enhanced cardiac magnetic resonance imaging

    Zhaohan Xiong, Qing Xia, Zhiqiang Hu, Ning Huang, Cheng Bian, Yefeng Zheng, Sulaiman Vesal, Nishant Ravikumar, Andreas Maier, Xin Yang, et al. A global benchmark of algorithms for segmenting the left atrium from late gadolinium-enhanced cardiac magnetic resonance imaging. Medi...

    2021

  116. [124]

    Convolutional sparse support estimator-based covid-19 recognition from x-ray images

    Mehmet Yamac, Mete Ahishali, Aysen Degerli, Serkan Kiranyaz, Muhammad EH Chowdhury, and Moncef Gabbouj. Convolutional sparse support estimator-based covid-19 recognition from x-ray images. IEEE Transactions on Neural Networks and Learning Systems, 32(5):1810–1820, 2021. 17

    2021

  117. [125]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024

    2024 arXiv

  118. [126]

    minicpm-o.https://github.com/OpenBMB/MiniCPM-o, 2025

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. minicpm-o.https://github.com/OpenBMB/MiniCPM-o, 2025

    2025

  119. [127]

    Imagetbad: A 3d computed tomography angiography image dataset for automatic segmentation of type-b aortic dissection

    Zeyang Yao, Wen Xie, Jiawei Zhang, Yuhao Dong, Hailong Qiu, Haiyun Yuan, Qianjun Jia, Tianchen Wang, Yiyi Shi, Jian Zhuang, et al. Imagetbad: A 3d computed tomography angiography image dataset for automatic segmentation of type-b aortic dissection. Frontiers in Physiology, 12:...

    2021

  120. [128]

    mplug-owl3: Towards long image-sequence understanding in multi-modal large language models

    Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models. arXiv preprint arXiv:2408.04840, 2024

    2024 arXiv

  121. [129]

    Gmai-mmbench: A comprehensive multimodal evaluation benchmark towards general medical ai

    Jin Ye, Guoan Wang, Yanjun Li, Zhongying Deng, Wei Li, Tianbin Li, Haodong Duan, Ziyan Huang, Yanzhou Su, Benyou Wang, et al. Gmai-mmbench: A comprehensive multimodal evaluation benchmark towards general medical ai. Advances in Neural Information Processing Systems, 37:94327–9...

    2024

  122. [130]

    Siim-acr pneumothorax segmentation

    Anna Zawacki, Carol Wu, George Shih, Julia Elliott, Mikhail Fomitchev, Mohannad Hussain, Paras Lakhani, Phil Culliton, and Shunxing Bao. Siim-acr pneumothorax segmentation. Mohannad ParasLakhani Hussain, 2019

    2019

  123. [131]

    Next-chat: An lmm for chat, detection and segmentation

    Ao Zhang, Yuan Yao, Wei Ji, Zhiyuan Liu, and Tat-Seng Chua. Next-chat: An lmm for chat, detection and segmentation. arXiv preprint arXiv:2311.04498, 2023

    2023 arXiv

  124. [132]

    Multi-site, multi-domain airway tree modeling

    Minghui Zhang, Yangqian Wu, Hanxiao Zhang, Yulei Qin, Hao Zheng, Wen Tang, Corey Arnold, Chenhao Pei, Pengxin Yu, Yang Nan, et al. Multi-site, multi-domain airway tree modeling. Medical image analysis, 90:102957, 2023

    2023

  125. [133]

    Psalm: Pixelwise segmentation with large multi-modal model

    Zheng Zhang, Yeyao Ma, Enming Zhang, and Xiang Bai. Psalm: Pixelwise segmentation with large multi-modal model. In European Conference on Computer Vision, pages 74–91. Springer, 2024

    2024

  126. [134]

    Llamafactory: Unified efficient fine-tuning of 100+ language models

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. arXiv preprint arXiv:2403.13372, 2024

    2024 arXiv

  127. [135]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025

    2025 arXiv

  128. [136]

    Multivariate mixture model for myocardial segmentation combining multi- source images

    Xiahai Zhuang. Multivariate mixture model for myocardial segmentation combining multi- source images. IEEE transactions on pattern analysis and machine intelligence, 41(12):2933– 2946, 2018

    2018

  129. [137]

    Medrg: Medical report grounding with multi-modal large language model

    Ke Zou, Yang Bai, Zhihao Chen, Yang Zhou, Yidi Chen, Kai Ren, Meng Wang, Xuedong Yuan, Xiaojing Shen, and Huazhu Fu. Medrg: Medical report grounding with multi-modal large language model. arXiv preprint arXiv:2404.06798, 2024. 18 A Dataset Details In this section, we provide t...

    2024 arXiv

  130. [138]

    MRI 3D https://service.tib.eu/ldmservice/dataset/ brats2020 BUSI[4] US 2D https://scholar.cu.edu.eg/?q=afahmy/pages/dataset CAD-PE[43] CT 3D https://ieee-dataport.org/open-access/cad-pe CAMUS[70] US 2D https://www.creatis.insa-lyon.fr/Challenge/camus/ Cause07[115] MRI 3D https...

    2018

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.