Pith. sign in

REVIEW 5 major objections 7 minor 58 references

Are Vision Language Models Ready for Clinical Diagnosis? A 3D Medical Benchmark for Tumor-centric Visual Question Answering

T0 review · 5 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that current 3D medical vision-language models handle measurement tasks but fail at lesion recognition and clinical reasoning, and are therefore not ready for clinical diagnosis.

desk verdict A genuinely useful large-scale 3D medical VQA benchmark with credible negative results on VLMs, though the 'medical reasoning' labels are unvalidated arithmetic proxies and the headline conclusion overreaches slightly. read the letter →

arxiv 2505.18915 v1 pith:NVSHMPMN submitted 2025-05-25 cs.CV

Yixiong Chen , Wenjie Xiao , Pedro R. A. S. Bassi , Xinze Zhou , Sezgin Er , Ibrahim Ethem Hamamci , Zongwei Zhou , Alan Yuille This is my paper · ORCID
classification cs.CV
keywords 3DmedicalVQAvisualquestionansweringabdominalCTtumordiagnosisvision-languagemodelsclinicalreasoningbenchmarklesionrecognition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DeepTumorVQA is a 3D visual-question-answering benchmark for abdominal tumor diagnosis built from 9,262 CT volumes and 395K expert-annotated questions spanning recognition, measurement, visual reasoning, and medical reasoning. Benchmarking RadFM, M3D, Merlin, and CT-CHAT, the paper claims that current models can perform adequately on measurement tasks and coarse quantification, but consistently struggle with small-lesion recognition and with reasoning tasks that build on it, especially medical reasoning. The authors conclude that no existing 3D medical VLM is clinically reliable for tumor diagnosis, and they argue that the benchmark's compositional structure lets future models be scored by diagnostic sub-skills rather than by reading comprehension. A sympathetic reader would care because the benchmark turns the vague question of whether medical VLMs are useful into measurable abilities tied to the radiological workflow.

What carries the argument

The central object is the DeepTumorVQA question-generation pipeline, a CLEVR-style functional program that converts organ and lesion segmentation masks into structured metadata such as volume, HU values, adjacency, and lesion counts, then composes deterministic functions over that metadata to produce templated questions in four difficulty tiers: Measurement, Recognition, Visual Reasoning, and Medical Reasoning. This machinery matters because it gives the benchmark a dependency structure: reasoning questions are literal compositions of recognition and measurement outputs, so a model's reasoning score can be traced back to its perception score. The evaluation machinery consists of three metrics, accuracy for multiple-choice questions, exact match for categorical free-text answers, and mean relative accuracy for numerical answers, chosen to avoid the weaknesses of BLEU and ROUGE for short factual medical answers.

What would settle it

A concrete check would be to take a sample of DeepTumorVQA cases, have independent board-certified radiologists give their own clinical decisions on the medical-reasoning questions, and compare those decisions with the benchmark's rule-generated ground truth; a disagreement rate comparable to the VLM error rate would falsify the claim that the benchmark measures clinical reasoning. A sharper version would be to swap the rule thresholds, for example resectability at 2.0 cm³ instead of 3.0 cm³, and retest the models: if accuracy follows the threshold rather than the underlying pathology, the models are applying rules rather than diagnosing.

Watch

Extended reading notes

Core claim

The paper's central claim is that state-of-the-art 3D medical vision-language models are not yet clinically reliable for tumor diagnosis: they perform adequately on measurement tasks such as organ volume and HU value estimation, and can approach segmentation-based methods on some recognition tasks, but their accuracy collapses on small or low-contrast lesion recognition and on visual and medical reasoning. The benchmark also exposes two drivers of that performance: large-scale multimodal pretraining, as in RadFM, matters more than parameter count alone, and the design of the vision module plus image preprocessing matters substantially, with tokenized 3D encoders and organ-cropping preprocessing improving lesion recognition. The intended consequence is that DeepTumorVQA serves as a standard testbed for tracking whether future 3D medical VLMs actually gain diagnostic competence.

Load-bearing premise

The load-bearing assumption is that the rule-based labels generated from segmentation metadata, such as resectability decided by cyst volume above 3.0 cm³ and steatosis by a pancreas-to-spleen HU ratio below 0.7, faithfully capture genuine clinical reasoning; if they do not, the conclusion that VLMs are unready for clinical diagnosis may only show that they fail at threshold arithmetic on noisy measurements.

Editorial extensions

If this is right

  • None of the four benchmarked VLMs reaches clinically acceptable performance on the full benchmark, and medical reasoning is the weakest category, so current 3D medical VLMs should not be used for autonomous tumor diagnosis.
  • Measurement questions are the most tractable, meaning existing models can already support quantitative readouts such as organ volume and HU values.
  • Large-scale multimodal pretraining and full fine-tuning, as in RadFM, yield more consistent gains than larger LLM backbones or LoRA tuning alone.
  • Organ-cropping preprocessing can dramatically improve lesion recognition, with one model's kidney tumor sensitivity rising from 0% to 80.9%, suggesting that anatomical localization priors can substitute for full voxel-level supervision.
  • Because reasoning questions are composed from recognition and measurement outputs, improvements in perception should propagate to reasoning scores, allowing the benchmark to track whether VLM gains reflect real diagnostic progress.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: because the medical-reasoning labels are rule-based proxies, such as resectability decided by cyst volume exceeding 3.0 cm³, the benchmark may measure the ability to apply fixed clinical thresholds rather than full radiological judgment, and a model could pass those items without understanding anatomy or clinical context.
  • Editorial extension: a direct test of whether the reasoning bottleneck is visual or logical would be to feed the benchmark's metadata tables as text only to the language models; if they still fail those reasoning questions, the deficit is in reasoning rather than perception.
  • Editorial extension: the authors' ablation that trains RadFM without measurement and recognition tasks suggests the benchmark can also quantify how much explicit perceptual supervision contributes to reasoning, and that large pretraining already encodes much of that knowledge implicitly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper presents DeepTumorVQA, a large-scale 3D medical VQA benchmark for abdominal tumors in CT. The dataset comprises 9,262 volumes from 17 public datasets, 395K question-answer pairs generated from radiologist-verified organ/lesion segmentations and structured metadata, organized into measurement, recognition, visual reasoning, and medical reasoning categories. Four VLM families (five model variants) are fine-tuned and evaluated. The authors report that models perform reasonably on measurement, poorly on lesion recognition and reasoning, with RadFM best overall; they also propose an nnUNet-based cropping preprocessing that improves lesion recognition. The manuscript argues current VLMs are not ready for clinical diagnosis and releases benchmark code/data.

Significance. If validated, DeepTumorVQA would be a valuable community resource: it is large-scale, multi-center, publicly released, and organized along a clinically motivated task hierarchy with radiologist involvement. The benchmark's finding that modern medical VLMs collapse to majority-class responses on lesion recognition (Table 4) is an important, sobering result for the field. The paper also ships open-source code and data and includes a concrete preprocessing intervention (nnM3D) with measurable improvements, which is a strength. However, the central clinical-readiness conclusion rests on medical-reasoning labels that are currently unvalidated rule-based proxies, and the model rankings lack statistical grounding; both issues must be addressed before the benchmark can serve as a rigorous diagnostic yardstick.

major comments (5)
  1. [Section 3.1, Appendix B Table 5] The Medical Reasoning ground truth is generated by deterministic arithmetic rules over segmentation-derived metadata, e.g., fatty liver from liver/spleen HU ratio, pancreatic steatosis from pancreas/spleen HU ratio < 0.7, pancreatic cyst resectability from cyst volume > 3.0 cm^3, and kidney lesion type from an HU threshold. Section 3.1 explicitly states that reasoning questions are 'systematically constructed by composing functions over outputs from the recognition and measurement stages.' The paper provides no validation that these rules reproduce expert clinical judgment; real resectability, staging, and lesion typing involve vascular involvement, enhancement patterns, and patient factors that these thresholds do not capture. Because the abstract's central conclusion ('still not meeting clinical needs') is largely driven by the Medical Reasoning results, the authors must either add a radiologist-agreement study validating a sample of these labels, or reposition the reasoning category as metadata-composition arithmetic and adjust the clinical-readiness claim accordingly.
  2. [Table 3, Section 4.2] All benchmark results are reported as single-run point estimates with no error bars, confidence intervals, or significance tests. The differences used to support the ranking claims are small: for multi-choice total average, RadFM 0.662 vs M3D-P3 0.632 vs CT-CHAT 0.628, and several subtype differences are under 0.02. Without variance estimates or paired significance testing, the conclusions that RadFM 'stands out' and that full fine-tuning versus LoRA matters are not statistically supported. The authors should report multiple seeds or bootstrap confidence intervals on the test set.
  3. [Table 4, Section 4.5] Table 4 reveals that several models (M3D-LLaMA2, M3D-Phi3, CT-CHAT) collapse to majority-class predictions on recognition tasks, yielding degenerately high sensitivity or specificity. Table 3 reports accuracy for recognition subtypes without sensitivity/specificity, and several free-text accuracies are identical to multi-choice values (e.g., colon lesion existence 0.859), which is consistent with such collapse. Reporting accuracy alone therefore overstates or obscures recognition capability. The paper should report sensitivity, specificity, or balanced accuracy for all recognition subtypes, not only the three in Table 4.
  4. [Table 1, Section 4.1] The benchmark is constructed entirely from public datasets (Table 1) that are standard in medical imaging research and are plausibly included in the pretraining corpora of the evaluated VLMs, especially RadFM with 16M 2D+3D pairs. The manuscript does not analyze potential overlap between benchmark volumes and pretraining data. Without a contamination analysis or evaluation on a non-public held-out cohort, the observed RadFM advantage may partially reflect data leakage rather than model capability. Please provide an overlap analysis or qualify the ranking accordingly.
  5. [Section 4.3, Figure 4] The ablation in Section 4.3 trains only RadFM with and without measurement/recognition tasks and describes a 'relatively small performance gap' without reporting the numerical accuracies or any uncertainty. This is insufficient evidence for the claim that basic tasks have limited impact on reasoning. The authors should include the full result table with per-subtype numbers and standard errors, or remove this claim.
minor comments (7)
  1. [Section 3.1] The definition of MRA (mean relative accuracy) is cited to reference [49] but not defined in the manuscript; please provide the formula.
  2. [Section 3.2] The statement defining 'clinical diagnosis' appears in the Question Generation paragraph; consider moving it to Section 3.1 where the diagnostic task hierarchy is introduced.
  3. [Table 3] The free-text and multi-choice columns are not clearly separated visually; consider using column groups or color coding to improve readability.
  4. [Appendix B Table 5] Several thresholds (e.g., the kidney lesion HU threshold and the fatty liver HU ratio criteria) are not stated in the table; please specify the exact values used and cite the clinical sources.
  5. [Section 4.5] The terms nnVLM and nnM3D are used before the naming convention is fully introduced; clarify that 'nn' refers to nnUNet-based preprocessing.
  6. [Figure 1] The figure is extremely dense; increasing font size or splitting it into two panels would improve legibility.
  7. [Table 1] Some dataset URLs in Table 1 are likely to become broken over time; consider adding DOIs or stable dataset identifiers.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark is a rule-based evaluation harness, and the central claim is an empirical finding, not a prediction derived from fitted inputs.

full rationale

DeepTumorVQA is a benchmark-construction and evaluation paper, not a derivation that predicts a target from fitted parameters. The central claim that current VLMs struggle with recognition and reasoning is an empirical result obtained by benchmarking external models (RadFM, M3D, Merlin, CT-CHAT) on the released dataset. The medical-reasoning questions are generated by deterministic rules over segmentation-derived metadata (e.g., pancreas/spleen HU ratio < 0.7 for steatosis, cyst volume > 3.0 cm3 for resectability), and Section 3.1 openly states that 'reasoning questions are systematically constructed by composing functions over outputs from the recognition and measurement stages.' This is a transparent, CLEVR-style compositional design choice, not a hidden equivalence: the reasoning labels are not fitted to model outputs, and no parameter is renamed as a prediction. The paper even analyzes the dependency in Section 4.3 by ablating measurement/recognition training. The concern that the rule-based proxies may not fully capture expert clinical reasoning is a construct-validity limitation, which the paper itself acknowledges in its Limitations section, but it is not circularity under the definitions used here. The only self-citations (e.g., following RadGPT [8] for report-style descriptions) are methodological and not load-bearing for the paper's main empirical conclusions. No uniqueness theorem, fitted input, or ansatz-smuggling pattern is present. Accordingly, the circularity score is 0.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The ledger shows that DeepTumorVQA's labels, especially in Medical Reasoning, depend on hand-set thresholds and on the accuracy of external segmentation masks. The paper introduces no new physical or conceptual entities, but it does rely on domain assumptions about annotation quality, pretraining-data hygiene, and metric validity.

free parameters (6)
  • pancreatic cyst resectability threshold = volume > 3.0 cm³
    Ground-truth answer for resectability is a hand-set volume cutoff, not a clinical decision; this defines a Medical Reasoning subtype (Appendix B Table 5).
  • pancreatic steatosis HU ratio threshold = pancreas/spleen HU ratio < 0.7
    Steatosis ground truth is set by this literature-derived but hand-applied cutoff (Appendix B Table 5).
  • lesion outlier volume ratio = 3×
    The 'lesion outlier' visual reasoning subtype is defined as largest lesion volume >3× the second largest (Appendix B Table 5).
  • liver lesion clustering thresholds = >3 lesions within 3 adjacent segments
    Clustering ground truth is defined by this hand-set rule (Appendix B Table 5).
  • kidney lesion HU threshold for tumor vs cyst = not specified in text
    Lesion type classification uses an unspecified HU threshold (Appendix B Table 5), controlling the Medical Reasoning 'lesion type classification' labels.
  • fatty liver HU thresholds = not specified in text
    Fatty liver subtype uses liver/spleen HU ratio and liver HU criteria from reference [53], but exact threshold values are not reported (Appendix B Table 5).
assumptions (5)
  • domain assumption Radiologist annotations of 7,629 lesions are accurate and the consensus double-check removes labeling noise.
    All QA labels derive from these annotations; annotation error propagates into every question (Section 3.2).
  • domain assumption The organ and lesion segmentation masks from the 17 public datasets are reliable enough to generate correct metadata.
    Metadata such as volume, HU, and location is computed from these masks; the paper itself notes imperfect segmentation may inject noise (Section 5 Limitations, Appendix A.1).
  • ad hoc to paper Rule-based proxies such as resectability by volume and steatosis by HU ratio are valid instantiations of clinical reasoning.
    Medical Reasoning ground truth is generated by these deterministic rules, so the benchmark measures threshold application, not necessarily clinical expertise (Section 3.1, Appendix B Table 5).
  • ad hoc to paper The benchmark source datasets do not substantially overlap with VLM pretraining corpora.
    RadFM and M3D were pretrained on large 3D CT collections; the paper provides no overlap analysis, yet uses RadFM's advantage to argue pretraining scale matters (Section 4.1, Tables 1 and 2).
  • domain assumption Task-specific metrics (accuracy, exact match, MRA) capture clinical correctness.
    MRA is cited to reference [49] but not defined in the main text; exact match treats near-correct numeric answers as wrong (Section 3.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Are Vision Language Models Ready for Clinical Diagnosis? A 3D Medical Benchmark for Tumor-centric Visual Question Answering." pith.science (2026). https://pith.science/paper/NVSHMPMN

@misc{pith2026250518915,
  author       = {Pith},
  title        = {Pith review of: Are Vision Language Models Ready for Clinical Diagnosis? A 3D Medical Benchmark for Tumor-centric Visual Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NVSHMPMN}},
  note         = {Machine review of arXiv:2505.18915}
}
read the original abstract

Vision-Language Models (VLMs) have shown promise in various 2D visual tasks, yet their readiness for 3D clinical diagnosis remains unclear due to stringent demands for recognition precision, reasoning ability, and domain knowledge. To systematically evaluate these dimensions, we present DeepTumorVQA, a diagnostic visual question answering (VQA) benchmark targeting abdominal tumors in CT scans. It comprises 9,262 CT volumes (3.7M slices) from 17 public datasets, with 395K expert-level questions spanning four categories: Recognition, Measurement, Visual Reasoning, and Medical Reasoning. DeepTumorVQA introduces unique challenges, including small tumor detection and clinical reasoning across 3D anatomy. Benchmarking four advanced VLMs (RadFM, M3D, Merlin, CT-CHAT), we find current models perform adequately on measurement tasks but struggle with lesion recognition and reasoning, and are still not meeting clinical needs. Two key insights emerge: (1) large-scale multimodal pretraining plays a crucial role in DeepTumorVQA testing performance, making RadFM stand out among all VLMs. (2) Our dataset exposes critical differences in VLM components, where proper image preprocessing and design of vision modules significantly affect 3D perception. To facilitate medical multimodal research, we have released DeepTumorVQA as a rigorous benchmark: https://github.com/Schuture/DeepTumorVQA.

Figures

Figures reproduced from arXiv: 2505.18915 by the authors.

Figure 1
Figure 1. Overview of tasks in the DeepTumorVQA benchmark. The dataset covers four core clinical [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Statistics of DeepTumorVQA. Left: the distribution of QA pairs for tasks across four main [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Overview of question construction in the DeepTumorVQA dataset. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The RadFM accuracy of reasoning tasks with or without measurement/recognition tasks. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Lesion recognition sensitivity of RadFM under different lesion sizes (left) and HU contrast [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Comparison of Youden’s Index (sensitivity + specificity - 1) of VLMs and the oracle. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Accuracy by question type across sex. scanner variation. Future work should incorporate domain adaptation or uncertainty modeling to ensure reliability across subpopulations. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 18 canonical work pages

  1. [1]

    Nlm at imageclef 2018 visual question answering in the medical domain

    Asma Ben Abacha, Soumya Gayen, Jason J Lau, Sivaramakrishnan Rajaraman, and Dina Demner-Fushman. Nlm at imageclef 2018 visual question answering in the medical domain. In CLEF (working notes), pages 1–10, 2018

    work page 2018

  2. [2]

    Phi-3 technical report: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219, 2024

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219, 2024

    arXiv 2024

  3. [3]

    Differentiating renal neoplasms from simple cysts on contrast-enhanced ct on the basis of attenuation and homogeneity.American Journal of Roentgenology, 208(4):801–804, 2017

    Nnenaya Agochukwu, Steffen Huber, Michael Spektor, Alexander Goehler, and Gary M Israel. Differentiating renal neoplasms from simple cysts on contrast-enhanced ct on the basis of attenuation and homogeneity.American Journal of Roentgenology, 208(4):801–804, 2017

    2017

  4. [4]

    Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35: 23716–23736, 2022

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35: 23716–23736, 2022

    2022

  5. [5]

    2017 robotic instrument segmentation challenge.arXiv preprint arXiv:1902.06426, 2019

    Max Allan, Alex Shvets, Thomas Kurmann, Zichen Zhang, Rahul Duggal, Yun-Hsuan Su, Nicola Rieke, Iro Laina, Niveditha Kalavakonda, Sebastian Bodenstedt, et al. 2017 robotic instrument segmentation challenge.arXiv preprint arXiv:1902.06426, 2019

    arXiv 2017

  6. [6]

    The medical segmentation decathlon.arXiv preprint arXiv:2106.05735, 2021

    Michela Antonelli, Annika Reinke, Spyridon Bakas, Keyvan Farahani, Bennett A Landman, Geert Litjens, Bjoern Menze, Olaf Ronneberger, Ronald M Summers, Bram van Ginneken, et al. The medical segmentation decathlon.arXiv preprint arXiv:2106.05735, 2021

    arXiv 2021

  7. [7]

    M3d: Advancing 3d medical image analysis with multi-modal large language models.arXiv preprint arXiv:2404.00578, 2024

    Fan Bai, Yuxin Du, Tiejun Huang, Max Q-H Meng, and Bo Zhao. M3d: Advancing 3d medical image analysis with multi-modal large language models.arXiv preprint arXiv:2404.00578, 2024

    arXiv 2024

  8. [8]

    Radgpt: Constructing 3d image-text tumor datasets.arXiv preprint arXiv:2501.04678, 2025

    Pedro RAS Bassi, Mehmet Can Yavuz, Kang Wang, Xiaoxi Chen, Wenxuan Li, Sergio Decher- chi, Andrea Cavalli, Yang Yang, Alan Yuille, and Zongwei Zhou. Radgpt: Constructing 3d image-text tumor datasets.arXiv preprint arXiv:2501.04678, 2025

    arXiv 2025

Show all 58 references
  1. [9]

    Overview of the vqa-med task at imageclef 2021: Visual question answering and generation in the medical domain

    Asma Ben Abacha, Mourad Sarrouti, Dina Demner-Fushman, Sadid A Hasan, and Henning Müller. Overview of the vqa-med task at imageclef 2021: Visual question answering and generation in the medical domain. InProceedings of the CLEF 2021 Conference and Labs of the Evaluation Forum-...

    2021

  2. [10]

    The liver tumor segmentation benchmark (lits).arXiv preprint arXiv:1901.04056, 2019

    Patrick Bilic, Patrick Ferdinand Christ, Eugene V orontsov, Grzegorz Chlebus, Hao Chen, Qi Dou, Chi-Wing Fu, Xiao Han, Pheng-Ann Heng, Jürgen Hesser, et al. The liver tumor segmentation benchmark (lits).arXiv preprint arXiv:1901.04056, 2019

    1901 arXiv

  3. [11]

    Merlin: A vision language foundation model for 3d computed tomography

    Louis Blankemeier, Joseph Paul Cohen, Ashwin Kumar, Dave Van Veen, Syed Jamal Safdar Gardezi, Magdalini Paschali, Zhihong Chen, Jean-Benoit Delbrouck, Eduardo Reis, Cesar Truyts, et al. Merlin: A vision language foundation model for 3d computed tomography. Research Square, pag...

    2024

  4. [12]

    Coca-cxr: Contrastive captioners learn strong temporal structures for chest x-ray vision-language understanding.arXiv preprint arXiv:2502.20509, 2025

    Yixiong Chen, Shawn Xu, Andrew Sellergren, Yossi Matias, Avinatan Hassidim, Shravya Shetty, Daniel Golden, Alan Yuille, and Lin Yang. Coca-cxr: Contrastive captioners learn strong temporal structures for chest x-ray vision-language understanding.arXiv preprint arXiv:2502.20509, 2025

    2025 arXiv

  5. [13]

    Rsna 2023 abdominal trauma detection, 2023

    Errol Colak, Hui-Ming Lin, Robyn Ball, Melissa Davis, Adam Flanders, Sabeena Jalal, Kirti Magudia, Brett Marinelli, Savvas Nicolaou, Luciano Prevedello, Jeff Rudie, George Shih, Maryam Vazirabad, and John Mongan. Rsna 2023 abdominal trauma detection, 2023. URL https://kaggle.c...

    2023

  6. [14]

    Preparing a collection of radiology examinations for distribution and retrieval.Journal of the American Medical Informatics Association, 23(2):304–310, 2016

    Dina Demner-Fushman, Marc D Kohli, Marc B Rosenman, Sonya E Shooshan, Laritza Ro- driguez, Sameer Antani, George R Thoma, and Clement J McDonald. Preparing a collection of radiology examinations for distribution and retrieval.Journal of the American Medical Informatics Associa...

    2016

  7. [15]

    Computed tomography evaluation of pancreatic steatosis: correlation with covid-19 prognosis.Future Virology, 17(4): 231–237, 2022

    Serkan Guneyli, Hakan Dogan, Omer Tarik Esengur, and Hur Hassoy. Computed tomography evaluation of pancreatic steatosis: correlation with covid-19 prognosis.Future Virology, 17(4): 231–237, 2022

    2022

  8. [16]

    Developing generalist foundation models from a multimodal dataset for 3d computed tomography.arXiv preprint arXiv:2403.17834, 2024

    Ibrahim Ethem Hamamci, Sezgin Er, Furkan Almas, Ayse Gulnihan Simsek, Sevval Nil Esirgun, Irem Dogan, Muhammed Furkan Dasdelen, Omer Faruk Durugol, Bastian Wittmann, Tamaz Amiranashvili, et al. Developing generalist foundation models from a multimodal dataset for 3d computed t...

    2024

  9. [17]

    Ct2rep: Automated radiology report generation for 3d medical imaging

    Ibrahim Ethem Hamamci, Sezgin Er, and Bjoern Menze. Ct2rep: Automated radiology report generation for 3d medical imaging. InInternational Conference on Medical Image Computing and Computer-Assisted Intervention, pages 476–486. Springer, 2024

    2024

  10. [18]

    Generatect: Text-conditional generation of 3d chest ct volumes

    Ibrahim Ethem Hamamci, Sezgin Er, Anjany Sekuboyina, Enis Simsar, Alperen Tezcan, Ayse Gulnihan Simsek, Sevval Nil Esirgun, Furkan Almas, Irem Do˘gan, Muhammed Furkan Dasdelen, et al. Generatect: Text-conditional generation of 3d chest ct volumes. InEuropean Conference on Comp...

    2024

  11. [19]

    Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet? InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6546–6555, 2018

    2018

  12. [20]

    Vision-language models for medical report generation and visual question answering: A review.Frontiers in Artificial Intelligence, 7:1430984, 2024

    Iryna Hartsock and Ghulam Rasool. Vision-language models for medical report generation and visual question answering: A review.Frontiers in Artificial Intelligence, 7:1430984, 2024

    2024

  13. [21]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

    2016

  14. [22]

    Pathvqa: 30000+ questions for medical visual question answering.arXiv preprint arXiv:2003.10286, 2020

    Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering.arXiv preprint arXiv:2003.10286, 2020

    2003 arXiv

  15. [23]

    An international challenge to use artificial intelligence to define the state-of-the-art in kidney and kidney tumor segmentation in ct imaging., 2020

    Nicholas Heller, Sean McSweeney, Matthew Thomas Peterson, Sarah Peterson, Jack Rickman, Bethany Stai, Resha Tejpaul, Makinna Oestreich, Paul Blake, Joel Rosenberg, et al. An international challenge to use artificial intelligence to define the state-of-the-art in kidney and kid...

    2020

  16. [24]

    Omnimed- vqa: A new large-scale comprehensive evaluation benchmark for medical lvlm

    Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, and Ping Luo. Omnimed- vqa: A new large-scale comprehensive evaluation benchmark for medical lvlm. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22170–22183, 2024

    2024

  17. [25]

    nnu-net: a self-configuring method for deep learning-based biomedical image segmentation.Nature methods, 18(2):203–211, 2021

    Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier-Hein. nnu-net: a self-configuring method for deep learning-based biomedical image segmentation.Nature methods, 18(2):203–211, 2021

    2021

  18. [26]

    Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation.Advances in Neural Information Processing Systems, 35:36722–36732, 2022

    Yuanfeng Ji, Haotian Bai, Chongjian Ge, Jie Yang, Ye Zhu, Ruimao Zhang, Zhen Li, Lingyan Zhanng, Wanling Ma, Xiang Wan, et al. Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation.Advances in Neural Information Processing Systems, 35:367...

    2022

  19. [27]

    Leading in the treatment of pancreatic cysts,

    Johns Hopkins Medicine. Leading in the treatment of pancreatic cysts,

  20. [28]

    Clevr: A diagnostic dataset for compositional language and elementary visual reasoning

    Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. InProceedings of the IEEE conference on computer vision and pattern recognition...

    2017

  21. [29]

    Towards visual dialog for radiology

    Olga Kovaleva, Chaitanya Shivade, Satyananda Kashyap, Karina Kanjaria, Joy Wu, Deddeh Ballah, Adam Coy, Alexandros Karargyris, Yufan Guo, David Beymer Beymer, et al. Towards visual dialog for radiology. InProceedings of the 19th SIGBioMed workshop on biomedical language proces...

    2020

  22. [30]

    Miccai multi-atlas labeling beyond the cranial vault–workshop and challenge

    Bennett Landman, Zhoubing Xu, J Igelsias, Martin Styner, T Langerak, and Arno Klein. Miccai multi-atlas labeling beyond the cranial vault–workshop and challenge. InProc. MICCAI Multi-Atlas Labeling Beyond Cranial Vault—Workshop Challenge, volume 5, page 12, 2015

    2015

  23. [31]

    A dataset of clinically generated visual questions and answers about radiology images.Scientific data, 5(1): 1–10, 2018

    Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images.Scientific data, 5(1): 1–10, 2018

    2018

  24. [32]

    Autorg-brain: Grounded report generation for brain mri.arXiv preprint arXiv:2407.16684, 2024

    Jiayu Lei, Xiaoman Zhang, Chaoyi Wu, Lisong Dai, Ya Zhang, Yanyong Zhang, Yanfeng Wang, Weidi Xie, and Yuehua Li. Autorg-brain: Grounded report generation for brain mri.arXiv preprint arXiv:2407.16684, 2024

    2024 arXiv

  25. [33]

    Llava-med: Training a large language-and-vision assistant for biomedicine in one day.Advances in Neural Information Processing Systems, 36: 28541–28564, 2023

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day.Advances in Neural Information Processing Systems, 36: 28541–28...

    2023

  26. [34]

    Medical visual question answering: A survey.Artificial Intelligence in Medicine, 143:102611, 2023

    Zhihong Lin, Donghao Zhang, Qingyi Tao, Danli Shi, Gholamreza Haffari, Qi Wu, Mingguang He, and Zongyuan Ge. Medical visual question answering: A survey.Artificial Intelligence in Medicine, 143:102611, 2023

    2023

  27. [35]

    Slake: A semantically- labeled knowledge-enhanced dataset for medical visual question answering

    Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. Slake: A semantically- labeled knowledge-enhanced dataset for medical visual question answering. In2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 1650–1654. IEEE, 2021

    2021

  28. [36]

    Word: Revisiting organs segmentation in the whole abdominal region

    Xiangde Luo, Wenjun Liao, Jianghong Xiao, Tao Song, Xiaofan Zhang, Kang Li, Guotai Wang, and Shaoting Zhang. Word: Revisiting organs segmentation in the whole abdominal region. arXiv preprint arXiv:2111.02403, 2021

    2021 arXiv

  29. [37]

    Abdomenct-1k: Is abdominal organ segmentation a solved problem.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021

    Jun Ma, Yao Zhang, Song Gu, Cheng Zhu, Cheng Ge, Yichi Zhang, Xingle An, Congcong Wang, Qiyuan Wang, Xin Liu, et al. Abdomenct-1k: Is abdominal organ segmentation a solved problem.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021

    2021

  30. [38]

    Fast and low-gpu-memory abdomen ct organ segmentation: the flare challenge.Medical Image Analysis, 82:102616, 2022

    Jun Ma, Yao Zhang, Song Gu, Xingle An, Zhihe Wang, Cheng Ge, Congcong Wang, Fan Zhang, Yu Wang, Yinan Xu, et al. Fast and low-gpu-memory abdomen ct organ segmentation: the flare challenge.Medical Image Analysis, 82:102616, 2022

    2022

  31. [39]

    Umass at imageclef medical visual question answering (med-vqa) 2018 task

    Yalei Peng, Feifan Liu, and Max P Rosen. Umass at imageclef medical visual question answering (med-vqa) 2018 task. InCLEF (working notes), pages 1–9, 2018

    2018

  32. [40]

    Cgmvqa: A new classification and generative model for medical visual question answering.IEEE Access, 8:50626–50636, 2020

    Fuji Ren and Yangyang Zhou. Cgmvqa: A new classification and generative model for medical visual question answering.IEEE Access, 8:50626–50636, 2020

    2020

  33. [41]

    Ct-org, a new dataset for multiple organ segmentation in computed tomography.Scientific Data, 7(1): 1–9, 2020

    Blaine Rister, Darvin Yi, Kaushik Shivakumar, Tomomi Nobashi, and Daniel L Rubin. Ct-org, a new dataset for multiple organ segmentation in computed tomography.Scientific Data, 7(1): 1–9, 2020

    2020

  34. [42]

    Deeporgan: Multi-level deep convolutional networks for automated pancreas segmentation

    Holger R Roth, Le Lu, Amal Farag, Hoo-Chang Shin, Jiamin Liu, Evrim B Turkbey, and Ronald M Summers. Deeporgan: Multi-level deep convolutional networks for automated pancreas segmentation. InInternational conference on medical image computing and computer- assisted interventio...

    2015

  35. [43]

    Medfusenet: An attention-based multimodal deep learning model for visual question answering in the medical domain.Scientific Reports, 11(1):19826, 2021

    Dhruv Sharma, Sanjay Purushotham, and Chandan K Reddy. Medfusenet: An attention-based multimodal deep learning model for visual question answering in the medical domain.Scientific Reports, 11(1):19826, 2021

    2021

  36. [44]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 12

    2023 arXiv

  37. [45]

    Towards generalist biomedical ai.Nejm Ai, 1(3):AIoa2300138, 2024

    Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Charles Lau, Ryutaro Tanno, Ira Ktena, et al. Towards generalist biomedical ai.Nejm Ai, 1(3):AIoa2300138, 2024

    2024

  38. [46]

    Multi-modal learning from unpaired images: Application to multi-organ segmentation in ct and mri

    Vanya V Valindria, Nick Pawlowski, Martin Rajchl, Ioannis Lavdas, Eric O Aboagye, Andrea G Rockall, Daniel Rueckert, and Ben Glocker. Multi-modal learning from unpaired images: Application to multi-organ segmentation in ct and mri. In2018 IEEE winter conference on applications...

    2018

  39. [47]

    Medclip: Contrastive learning from unpaired medical images and text

    Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. Medclip: Contrastive learning from unpaired medical images and text. InProceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing, v...

    2022

  40. [48]

    Towards generalist foundation model for radiology by leveraging web-scale 2d&3d medical data.arXiv preprint arXiv:2308.02463, 2023

    Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Towards generalist foundation model for radiology by leveraging web-scale 2d&3d medical data.arXiv preprint arXiv:2308.02463, 2023

    2023 arXiv

  41. [49]

    Thinking in space: How multimodal large language models see, remember, and recall spaces.arXiv preprint arXiv:2412.14171, 2024

    Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces.arXiv preprint arXiv:2412.14171, 2024

    2024 arXiv

  42. [50]

    Advancing multimodal medical capabilities of gemini.arXiv preprint arXiv:2405.03162, 2024

    Lin Yang, Shawn Xu, Andrew Sellergren, Timo Kohlberger, Yuchen Zhou, Ira Ktena, Atilla Kiraly, Faruk Ahmed, Farhad Hormozdiari, Tiam Jaroensri, et al. Advancing multimodal medical capabilities of gemini.arXiv preprint arXiv:2405.03162, 2024

    2024 arXiv

  43. [51]

    How big can it get? a comparative analysis of llms in architecture and scaling

    Ramez Yousri and Soha Safwat. How big can it get? a comparative analysis of llms in architecture and scaling. In2023 International Conference on Computer and Applications (ICCA), pages 1–5. IEEE, 2023

    2023

  44. [52]

    Coca: Contrastive captioners are image-text foundation models.arXiv preprint arXiv:2205.01917, 2022

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models.arXiv preprint arXiv:2205.01917, 2022

    2022 arXiv

  45. [53]

    Computed tomography scans in the evaluation of fatty liver disease in a population based study: the multi-ethnic study of atherosclerosis.Academic radiology, 19(7):811–818, 2012

    Irfan Zeb, Dong Li, Khurram Nasir, Ronit Katz, Vahid N Larijani, and Matthew J Budoff. Computed tomography scans in the evaluation of fatty liver disease in a population based study: the multi-ethnic study of atherosclerosis.Academic radiology, 19(7):811–818, 2012

    2012

  46. [54]

    Vision-language models for vision tasks: A survey.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  47. [55]

    Large-scale domain-specific pretraining for biomedical vision-language processing.arXiv preprint arXiv:2303.00915, 2(3):6, 2023

    Sheng Zhang, Yanbo Xu, Naoto Usuyama, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, Cliff Wong, et al. Large-scale domain-specific pretraining for biomedical vision-language processing.arXiv preprint arXiv:2303.00915, 2(3):6, 2023

    2023 arXiv

  48. [56]

    Pmc-vqa: Visual instruction tuning for medical visual question answering.arXiv preprint arXiv:2305.10415, 2023

    Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc-vqa: Visual instruction tuning for medical visual question answering.arXiv preprint arXiv:2305.10415, 2023

    2023 arXiv

  49. [57]

    Multiple (25) hypoattenuating liver masses. Largest one (segment 2) measures 3.2 x 1.7 cm. Total volume of all liver masses: 19.4 cm3

    Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Jiayu Lei, Ya Zhang, Yanfeng Wang, and Weidi Xie. Radgenome-chest ct: A grounded vision-language dataset for chest ct analysis.arXiv preprint arXiv:2404.16754, 2024. 13 A Metadata and Structured Description Generation To support systemati...

    2024 arXiv

  50. [2022]

    Accessed: 2025-05-05

    URL https://www.hopkinsmedicine.org/news/articles/2022/04/ leading-in-the-treatment-of-pancreatic-cysts. Accessed: 2025-05-05

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.