Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

MedVista3D: Vision-Language Modeling for Reducing Diagnostic Errors in 3D CT Disease Detection, Understanding and Reporting

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MedVista3D claims that aligning CT volumes and reports at whole-scan and organ scales yields one model that beats prior 3D models at disease detection, retrieval, and VQA, and transfers to segmentation and prognosis.

desk verdict Solid multi-scale 3D CT VLM with consistent empirical gains, but the headline claims about theory and mask-free clinical utility are over-egged. read the letter →

arxiv 2509.03800 v1 pith:CO5W4AFO submitted 2025-09-04 cs.CV

Yuheng Li , Yenho Chen , Yuxiang Lai , Jike Zhong , Vanessa Wildman , Xiaofeng Yang This is my paper
classification cs.CV
keywords 3Dcomputedtomographyvision-languagepretrainingmulti-scalealignmentzero-shotdiseaseclassificationreportretrievalmedicalvisualquestionansweringmutualinformationradiologysemantics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Radiologists interpreting 3D CT must simultaneously spot small local abnormalities and understand the whole volume; models trained to do only one of these tend to fail at the other, mirroring under-reading and inattentional blindness. MedVista3D is a vision-language pretraining scheme that trains one model on both scales—full CT volume with its report, and individual organs with their region descriptions. The paper shows that this two-scale contrastive objective is a valid lower bound on the mutual information between all visual and textual views, so it captures at least as much shared signal as either single-scale objective. It also cleans the language side by rewriting reports into explicit presence/absence findings and anchoring text embeddings with a similarity bank. On CT-RATE and Rad-ChestCT, the model reports the best zero-shot disease classification, report retrieval, and medical VQA numbers among the compared 3D VLMs, and it transfers to organ segmentation and prognosis prediction.

What carries the argument

The load-bearing object is the multi-scale alignment objective LMedVista3D = 1/2(LGlobal + LLocal) + LGlobal Semantic + LLocal Semantic. LGlobal is InfoNCE between whole-volume and whole-report embeddings; LLocal is InfoNCE between organ-region embeddings selected by mask pooling and their region-text phrases; the semantic terms repeat the alignment using RSMB-retrieved nearest-neighbor embeddings of LLM-rewritten disease sentences. The paper's theoretical step shows the averaged global/local loss is a lower bound on the unified mutual information I(XG, XL; YL, YG), which is at least max(I(XG; YG), I(XL; YL)). This is what lets one encoder serve both detection and reasoning; the RSMB queue (

What would settle it

Run local disease zero-shot classification on CT-RATE with the segmentation masks eroded, shifted, or replaced by no mask at all: if AUC holds near 0.780 without accurate masks, the mask dependency is not load-bearing; if it drops toward the ~0.66 global-only level, the paper's local-detection claim collapses.

Watch

Extended reading notes

Core claim

The paper's central claim is that a single 3D CT vision-language model can jointly support local detection, global understanding, and consistent reporting, and that the reason prior models fail is that each optimizes only one scale. MedVista3D pairs a global pathway (whole volume vs. full report) with a local pathway (mask-pooled organ region vs. region sentence) inside one encoder, trained with a multi-scale contrastive loss. The proof component shows that the averaged global-plus-local objective bounds a unified mutual information I(XG, XL; YL, YG), which dominates either single-scale mutual information. On the text side, reports are rewritten by an LLM as explicit presence/absence finding

Load-bearing premise

The local-detection gains require an accurate organ mask for the region being queried, at both training and test time; if that mask is wrong or unavailable, the local features are built from the wrong region and the reported reduction of under-reading errors does not follow.

Editorial extensions

If this is right

  • The same pretrained weights can be finetuned for pixel-level organ segmentation and patient-level prognosis, suggesting a single 3D CT backbone can replace separate task-specific encoders.
  • Zero-shot disease classification and report retrieval improve together, so a clinician could query a model for a finding without any task-specific training data.
  • Medical VQA and report generation improve on CT-RATE, with reported multiple-choice accuracy at 91.5%, indicating more faithful communication of findings.
  • The multi-scale mutual-information argument implies the objective should transfer to any setting with paired image-text data where region masks exist, not just chest CT.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because local alignment needs segmentation masks at inference, deploying MedVista3D in a real radiology workflow depends on an automatic segmenter being reliable for the organ in question; without it, the model likely behaves like a global-only model on local queries.
  • Editorial inference: the RSMB's top-1 nearest-neighbor retrieval may over-anchor paraphrases to a single template; testing top-k aggregation or hard-negative mining would reveal whether the gain comes from semantic matching or from simple denoising.
  • Editorial inference: LLM-rewritten supervision could encode hallucinated findings into the model; auditing rewrite faithfulness against the original report would bound how much of the reported gain is trustworthy.
  • Editorial inference: the unified mutual-information argument suggests the multi-scale objective should help any 3D modality with complementary local and global structure, such as MRI or PET, once paired text and organ masks are available.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. MedVista3D proposes a 3D CT vision-language pretraining framework that combines global volume-report alignment with local organ-level alignment conditioned on segmentation masks, augmented by LLM-rewritten radiology reports and a queue-based Radiology Semantic Matching Bank (RSMB). The authors claim theoretical support for a multi-scale mutual-information objective, and report state-of-the-art results on zero-shot disease classification, report retrieval, and medical VQA on CT-RATE and Rad-ChestCT, with additional transfer results for organ segmentation and COVID-19 prognosis prediction.

Significance. If the empirical results hold, the paper makes a useful contribution to 3D medical VLMs: it directly targets a known limitation of prior global-only (CT-CLIP) and local-only (fVLM) models, provides a unified multi-scale pretraining recipe, and evaluates on both internal and external datasets. The release of code and checkpoints is a concrete reproducibility benefit. However, the significance is conditional: the central clinical-interpretation claims (reducing under-reading errors) depend on a mask-available evaluation protocol, several headline margins are small and unaccompanied by uncertainty estimates, and the theoretical result is weaker than stated.

major comments (3)
  1. [§3.1, Eq. (5)] The mathematical claim that the multi-scale loss 'captures more mutual information' than single-scale alignment is not supported by the displayed inequality. Eq. (5) is a valid lower bound, but an average of two InfoNCE lower bounds is no tighter than the maximum of the two individual bounds; it is generally weaker. The chain-rule argument shows IUnified ≥ I(XG;YG) and IUnified ≥ I(XL;YL), not that LMulti-scale dominates either single-scale objective. The abstract and introduction state a theoretical demonstration of superior MI capture, but the paper only establishes a valid bound. This section should be rewritten to state precisely what is proved (a combined lower bound, not guaranteed tighter MI), or a genuinely tighter bound must be supplied.
  2. [§4.1, Tables 1-2; Appendix C] The claim of reduced under-reading errors is not established for realistic deployment because the local evaluation protocol supplies segmentation masks as input. The task description says 'Given text prompts and segmentation masks,' and §3.1 selects active tokens by thresholding the downsampled mask at 0.5. Appendix C concedes that local alignment 'requires anatomical priors (i.e. segmentation masks), which may not always be feasible in clinical settings.' On CT-RATE the local AUC margin over fVLM is only 0.002 (0.780 vs 0.778); on Rad-ChestCT the masks are produced by TotalSegmentator with no analysis of mask quality or of failures relative to lesion location. If the segmenter misses or distorts the organ containing the lesion, the local features are computed from the wrong tokens and the claimed benefit could disappear or reverse. Please add experiments with imperfect, missing, or pred
  3. [§4, Tables 1-3, and NeurIPS Checklist item 7] The paper reports no error bars, confidence intervals, or significance tests for any of the main quantitative results, and the checklist justifies 'NA' by training-set size and computational cost. Several key comparisons are very close (Table 1 local AUC 0.780 vs 0.778; Table 3 short-answer CIDER 1.817 vs 1.821), making it impossible to tell whether the reported advantages are reproducible or within run-to-run noise. For state-of-the-art claims on diagnostic tasks, at least bootstrap confidence intervals over the test set or multiple-seed means with standard deviations should be reported for the main tables. This is load-bearing for the abstract's SOTA claims.
minor comments (5)
  1. [§3.2, Eqs. (7)-(9)] Notation is inconsistent and has typesetting artifacts: 'LGN N' and 'LLN N' appear for the NN losses, and Eq. (9) contains an unbalanced parenthesis in the similarity term. Please clean up the equations.
  2. [Appendix C] The local text embedding is denoted tL_i in Appendix C but tr_i in §3.1 and the main equations. Please unify notation.
  3. [§4.3, Table 4] The text says mask pooling only 'slightly' compromises global zero-shot performance, but AUC drops from 0.664 to 0.609 in the table. Also, the full objective (0.778 global AUC) is worse than the '+ Global Semantic Alignment' intermediate (0.807); the paper should explicitly discuss this trade-off rather than presenting the final configuration as uniformly better.
  4. [§4.1, Tables 1-2] For fVLM, both 'our implementation' (†) and 'official checkpoint' (‡) are reported, and their results differ substantially (e.g., global AUC 0.644 vs 0.591 on CT-RATE). The reasons for this discrepancy should be explained, since it affects the fairness of the comparison.
  5. [General] The abbreviation 'LLaVA' is rendered inconsistently as 'LLaV A' throughout the tables and text. Please standardize.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are supported by held-out/external evaluation and the proposed losses are standard contrastive objectives, not self-referential by construction.

full rationale

The derivation chain is self-contained. The multi-scale objective LMulti-scale = 1/2[LGlobal + LLocal] is a direct combination of standard InfoNCE global and local alignment losses; the theoretical lower bound in Eq. (5) follows from the chain rule for mutual information and the cited InfoNCE lower bound [28], not by defining the result into the loss. The Radiology Semantic Matching Bank is a MoCo-style queue of text embeddings; using the model's own text encoder to retrieve a neighbor is a bootstrapping/self-training technique and does not make the held-out zero-shot evaluations equivalent to the training objective. The main empirical claims (disease zero-shot classification, report retrieval, VQA, segmentation, prognosis) are measured on the CT-RATE test split and on the external Rad-ChestCT dataset against published baselines, so they have independent grounding. The only self-citation ([25], LLaVA-1.5, cited in Appendix B for the projector architecture and two-stage training) is not load-bearing: it is used for a standard public component. Appendix C explicitly notes that local alignment requires anatomical priors (segmentation masks), which may be infeasible clinically; this is a real deployment limitation and a correctness/robustness concern, but it is not circularity because the local zero-shot protocol honestly provides masks to all models and does not fit the reported AUCs to the test labels. No equation reduces to its own input, and no fitted parameter is renamed as a prediction. Therefore no circular step is identified.

Assumptions & free parameters 4 free parameters · 6 assumptions · 1 invented entities

The central empirical claim rests on standard MI theory plus several domain assumptions: reliable organ masks, faithful LLM rewrites, and semantic equivalence of retrieved neighbors. The MI proof is mathematically valid but contributes little beyond known InfoNCE bounds; the domain assumptions are the real load-bearing commitments.

free parameters (4)
  • Contrastive temperature tau = learned, not reported
    Learnable temperature in InfoNCE losses, Eqs. 13 and 15; standard in CLIP-style training but affects alignment sharpness.
  • Mask pooling threshold = 0.5
    Threshold on the downsampled segmentation mask to select active tokens, Section 3.1; hand-chosen and not ablated.
  • Mask pooling transformer layer = 12th or last block
    Selected by comparing 1st, 2nd, and 12th blocks in Table 5; a design choice tuned on downstream local detection performance.
  • RSMB queue size = 64,000
    FIFO queue length for text embeddings, Section 3.2; hand-chosen and not ablated.
assumptions (6)
  • standard math InfoNCE loss is a lower bound on mutual information
    Used in Section 3.1, Eqs. (1)-(2), from Poole et al. [28].
  • standard math Chain rule: I(XG,XL; YL,YG) >= max(I(XL;YL), I(XG;YG))
    Used to justify the multi-scale objective in Section 3.1.
  • domain assumption LLM rewrites preserve true disease presence and absence
    Section 3.2 relies on GPT-4o and Qwen2.5 rewrites as supervision; no human verification is reported.
  • domain assumption Top-1 cosine neighbor in RSMB is semantically equivalent to the query text
    Section 3.2 uses the nearest neighbor as a training target; retrieval errors would inject wrong text supervision.
  • domain assumption Segmentation masks from Radgenome and TotalSegmentator are accurate enough for local feature extraction
    The local pathway selects tokens by mask pooling, Section 3.1 and Appendix C.
  • domain assumption Chest-CT pretraining transfers to external datasets and tasks
    External validation and transfer experiments on Rad-ChestCT, TotalSegmentator, and STOIC assume distributional and anatomical transfer.
invented entities (1)
  • Radiology Semantic Matching Bank (RSMB)
    purpose: A 64k first-in-first-out queue of text embeddings that supplies top-1 nearest-neighbor texts as additional contrastive targets for global and local image-text alignment.
    This is a new technical construct of the paper; its correctness is not validated externally, and it assumes that nearby embeddings in the model's own text space have equivalent clinical meaning.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MedVista3D: Vision-Language Modeling for Reducing Diagnostic Errors in 3D CT Disease Detection, Understanding and Reporting." pith.science (2026). https://pith.science/paper/CO5W4AFO

@misc{pith2026250903800,
  author       = {Pith},
  title        = {Pith review of: MedVista3D: Vision-Language Modeling for Reducing Diagnostic Errors in 3D CT Disease Detection, Understanding and Reporting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CO5W4AFO}},
  note         = {Machine review of arXiv:2509.03800}
}
read the original abstract

Radiologic diagnostic errors-under-reading errors, inattentional blindness, and communication failures-remain prevalent in clinical practice. These issues often stem from missed localized abnormalities, limited global context, and variability in report language. These challenges are amplified in 3D imaging, where clinicians must examine hundreds of slices per scan. Addressing them requires systems with precise localized detection, global volume-level reasoning, and semantically consistent natural language reporting. However, existing 3D vision-language models are unable to meet all three needs jointly, lacking local-global understanding for spatial reasoning and struggling with the variability and noise of uncurated radiology reports. We present MedVista3D, a multi-scale semantic-enriched vision-language pretraining framework for 3D CT analysis. To enable joint disease detection and holistic interpretation, MedVista3D performs local and global image-text alignment for fine-grained representation learning within full-volume context. To address report variability, we apply language model rewrites and introduce a Radiology Semantic Matching Bank for semantics-aware alignment. MedVista3D achieves state-of-the-art performance on zero-shot disease classification, report retrieval, and medical visual question answering, while transferring well to organ segmentation and prognosis prediction. Code and datasets will be released.

Figures

Figures reproduced from arXiv: 2509.03800 by the authors.

Figure 1
Figure 1. We visualize gradient activation maps for both global and local queries on global model [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Left: Global zero-shot performance of MedVista3D-ViT on CT-RATE. AUC scores are reported per disease, reflecting the model’s generalization across diverse pathologies. Right: LLM￾based refinement of radiology reports. To address ambiguity and inconsistency in uncurated CT-RATE reports, we apply large language models (e.g., GPT-4o, Qwen2.5) to rewrite them with improved clarity and clinical coherence. • We identify t… view at source ↗
Figure 3
Figure 3. a). MedVista3D encodes 3D CT volumes at both global and local scales. For local [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Impact of region masking on attention for CLIP, fVLM and MedVista3D (on CT-RATE). [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 2
Figure 2. Figure 2: Guidelines: • The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components. • Please refer to our LLM policy (https://neurips.cc/Conferences/2025/LLM) for what should or should not be…
Figure 5
Figure 5. Figure 5: Prompts for report-level rewrites to emphasize disease presences. [PITH_FULL_IMAGE:figures/full_fig_p022_5.png]
Figure 6
Figure 6. Figure 6: Prompts for region-level rewrite with few-shot prompting. [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Self-Supervised Dynamical System Representations for Physiological Time-Series

    cs.LG 2025-11 conditional novelty 6.0 of 10

    PULSE pretrains physiological time-series encoders by reconstructing random crops from inferred system parameters, improving label efficiency and transfer across four sensor domains.

Reference graph

Works this paper leans on

60 extracted references · 41 canonical work pages · cited by 1 Pith paper

  1. [1]

    Merlin: A vision language foundation model for 3d computed tomography

    Louis Blankemeier, Joseph Paul Cohen, Ashwin Kumar, Dave Van Veen, Syed Jamal Safdar Gardezi, Magdalini Paschali, Zhihong Chen, Jean-Benoit Delbrouck, Eduardo Reis, Cesar Truyts, et al. Merlin: A vision language foundation model for 3d computed tomography. Research Square, pages rs–3, 2024

    2024

  2. [2]

    A vision–language foundation model for the generation of realistic chest x-ray images

    Christian Bluethgen, Pierre Chambon, Jean-Benoit Delbrouck, Rogier van der Sluijs, Małgorzata Połacin, Juan Manuel Zambrano Chaves, Tanishq Mathew Abraham, Shivanshu Purohit, Curtis P Langlotz, and Akshay S Chaudhari. A vision–language foundation model for the generation of realistic chest x-ray images. Nature Biomedical Engineering, pages 1–13, 2024

    work page 2024

  3. [3]

    Making the most of text semantics to improve biomedical vision–language processing

    Benedikt Boecking, Naoto Usuyama, Shruthi Bannur, Daniel C Castro, Anton Schwaighofer, Stephanie Hyland, Maria Wetscherek, Tristan Naumann, Aditya Nori, Javier Alvarez-Valle, et al. Making the most of text semantics to improve biomedical vision–language processing. In European conference on computer vision, pages 1–21. Springer, 2022

    2022

  4. [4]

    Understanding and confronting our mis- takes: the epidemiology of error in radiology and strategies for error reduction

    Michael A Bruno, Eric A Walker, and Hani H Abujudeh. Understanding and confronting our mis- takes: the epidemiology of error in radiology and strategies for error reduction. Radiographics, 35(6):1668–1676, 2015

    work page 2015

  5. [5]

    Joint modeling of chest radiographs and radiology reports for pulmonary edema assessment

    Geeticka Chauhan, Ruizhi Liao, William Wells, Jacob Andreas, Xin Wang, Seth Berkowitz, Steven Horng, Peter Szolovits, and Polina Golland. Joint modeling of chest radiographs and radiology reports for pulmonary edema assessment. InMedical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8,...

    work page 2020

  6. [6]

    Contrastive localized language-image pre-training

    Hong-You Chen, Zhengfeng Lai, Haotian Zhang, Xinze Wang, Marcin Eichner, Keen You, Meng Cao, Bowen Zhang, Yinfei Yang, and Zhe Gan. Contrastive localized language-image pre-training. arXiv preprint arXiv:2410.02746, 2024

    arXiv 2024

  7. [7]

    A review of medical image data augmentation techniques for deep learning applications

    Phillip Chlap, Hang Min, Nym Vandenberg, Jason Dowling, Lois Holloway, and Annette Ha- worth. A review of medical image data augmentation techniques for deep learning applications. Journal of Medical Imaging and Radiation Oncology, 65(5):545–563, 2021

    work page 2021

  8. [8]

    Machine-learning-based multiple abnormality prediction with large-scale chest computed tomography volumes

    Rachel Lea Draelos, David Dov, Maciej A Mazurowski, Joseph Y Lo, Ricardo Henao, Geof- frey D Rubin, and Lawrence Carin. Machine-learning-based multiple abnormality prediction with large-scale chest computed tomography volumes. Medical image analysis, 67:101857, 2021

    work page 2021

Show all 60 references
  1. [9]

    Multi-view and multi-scale alignment for contrastive language-image pre-training in mammography

    Yuexi Du, John Onofrey, and Nicha C Dvornek. Multi-view and multi-scale alignment for contrastive language-image pre-training in mammography. arXiv preprint arXiv:2409.18119, 2024. 10

    2024 arXiv

  2. [10]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  3. [11]

    Devel- oping generalist foundation models from a multimodal dataset for 3d computed tomography

    Ibrahim Ethem Hamamci, Sezgin Er, Chenyu Wang, Furkan Almas, Ayse Gulnihan Simsek, Sevval Nil Esirgun, Irem Doga, Omer Faruk Durugol, Weicheng Dai, Murong Xu, et al. Devel- oping generalist foundation models from a multimodal dataset for 3d computed tomography. arXiv preprint ...

    2024

  4. [12]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  5. [13]

    Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition

    Shih-Cheng Huang, Liyue Shen, Matthew P Lungren, and Serena Yeung. Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3942–3951, 2021

    2021

  6. [14]

    Enhancing representation in medical vision-language foun- dation models via multi-scale information extraction techniques

    Weijian Huang, Cheng Li, Hong-Yu Zhou, Jiarun Liu, Hao Yang, Yong Liang, Guangming Shi, Hairong Zheng, and Shanshan Wang. Enhancing representation in medical vision-language foun- dation models via multi-scale information extraction techniques. In 2024 IEEE International Sympo...

    2024

  7. [15]

    Stu-net: Scalable and transferable medical image segmentation models empowered by large-scale supervised pre-training

    Ziyan Huang, Haoyu Wang, Zhongying Deng, Jin Ye, Yanzhou Su, Hui Sun, Junjun He, Yun Gu, Lixu Gu, Shaoting Zhang, et al. Stu-net: Scalable and transferable medical image segmentation models empowered by large-scale supervised pre-training. arXiv preprint arXiv:2304.06716, 2023

    2023 arXiv

  8. [16]

    nnu-net: a self-configuring method for deep learning-based biomedical image segmentation

    Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier-Hein. nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2):203–211, 2021

    2021

  9. [17]

    Fool me twice: delayed diagnoses in radiology with emphasis on perpetuated errors

    Young W Kim and Liem T Mansfield. Fool me twice: delayed diagnoses in radiology with emphasis on perpetuated errors. American journal of roentgenology, 202(3):465–470, 2014

    2014

  10. [18]

    Generating synthetic data for medical imaging

    Lennart R Koetzier, Jie Wu, Domenico Mastrodicasa, Aline Lutz, Matthew Chung, W Adam Koszek, Jayanth Pratap, Akshay S Chaudhari, Pranav Rajpurkar, Matthew P Lungren, et al. Generating synthetic data for medical imaging. Radiology, 312(3):e232471, 2024

    2024

  11. [19]

    Cxr-llava: a multimodal large language model for interpreting chest x-ray images

    Seowoo Lee, Jiwon Youn, Hyungjin Kim, Mansu Kim, and Soon Ho Yoon. Cxr-llava: a multimodal large language model for interpreting chest x-ray images. European Radiology, pages 1–13, 2025

    2025

  12. [20]

    Llava-med: Training a large language-and-vision assistant for biomedicine in one day

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36, 2024

    2024

  13. [21]

    Artificial general intelligence for medical imaging analysis

    Xiang Li, Lin Zhao, Lu Zhang, Zihao Wu, Zhengliang Liu, Hanqi Jiang, Chao Cao, Shaochen Xu, Yiwei Li, Haixing Dai, et al. Artificial general intelligence for medical imaging analysis. IEEE Reviews in Biomedical Engineering, 2024

    2024

  14. [22]

    Ct-glip: 3d grounded language-image pretraining with ct scans and radiology reports for full-body scenarios

    Jingyang Lin, Yingda Xia, Jianpeng Zhang, Ke Yan, Le Lu, Jiebo Luo, and Ling Zhang. Ct-glip: 3d grounded language-image pretraining with ct scans and radiology reports for full-body scenarios. arXiv preprint arXiv:2404.15272, 2024

    2024

  15. [23]

    Pmc-clip: Contrastive language-image pre-training using biomedical documents

    Weixiong Lin, Ziheng Zhao, Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc-clip: Contrastive language-image pre-training using biomedical documents. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 525–536. S...

    2023

  16. [24]

    Can medical vision-language pre-training succeed with purely synthetic data? arXiv preprint arXiv:2410.13523, 2024

    Che Liu, Zhongwei Wan, Haozhe Wang, Yinda Chen, Talha Qaiser, Chen Jin, Fariba Yousefi, Nikolay Burlutskiy, and Rossella Arcucci. Can medical vision-language pre-training succeed with purely synthetic data? arXiv preprint arXiv:2410.13523, 2024. 11

    2024 arXiv

  17. [25]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024

    2024

  18. [26]

    Representation learning with contrastive predictive coding

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018

    2018 arXiv

  19. [27]

    Unsupervised medical image translation with adversarial diffusion models

    Muzaffer Özbey, Onat Dalmaz, Salman UH Dar, Hasan A Bedel, ¸ Saban Özturk, Alper Güngör, and Tolga Çukur. Unsupervised medical image translation with adversarial diffusion models. IEEE Transactions on Medical Imaging, 2023

    2023

  20. [28]

    On variational bounds of mutual information

    Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning , pages 5171–5180. PMLR, 2019

    2019

  21. [29]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  22. [30]

    Study of thoracic ct in covid-19: the stoic project

    Marie-Pierre Revel, Samia Boussouar, Constance de Margerie-Mellon, Inès Saab, Thibaut Lapotre, Dominique Mompoint, Guillaume Chassagnon, Audrey Milon, Mathieu Lederlin, Souhail Bennani, et al. Study of thoracic ct in covid-19: the stoic project. Radiology, 301(1):E361–E370, 2021

    2021

  23. [31]

    Deep learning in medical image analysis

    Dinggang Shen, Guorong Wu, and Heung-Il Suk. Deep learning in medical image analysis. Annual review of biomedical engineering, 19(1):221–248, 2017

    2017

  24. [32]

    Large-scale and fine-grained vision- language pre-training for enhanced ct image understanding

    Zhongyi Shui, Jianpeng Zhang, Weiwei Cao, Sinuo Wang, Ruizhe Guo, Le Lu, Lin Yang, Xianghua Ye, Tingbo Liang, Qi Zhang, and Ling Zhang. Large-scale and fine-grained vision- language pre-training for enhanced ct image understanding. In The Thirteenth International Conference on...

    2025

  25. [33]

    Bioclip: A vision foundation model for the tree of life

    Samuel Stevens, Jiaman Wu, Matthew J Thompson, Elizabeth G Campolongo, Chan Hee Song, David Edward Carlyn, Li Dong, Wasila M Dahdul, Charles Stewart, Tanya Berger-Wolf, et al. Bioclip: A vision foundation model for the tree of life. In Proceedings of the IEEE/CVF Conference on...

    2024

  26. [34]

    Xraygpt: Chest radiographs summarization using medical vision-language models

    Omkar Thawkar, Abdelrahman Shaker, Sahal Shaji Mullappilly, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, Jorma Laaksonen, and Fahad Shahbaz Khan. Xraygpt: Chest radiographs summarization using medical vision-language models. arXiv preprint arXiv:2306.07971, 2023

    2023 arXiv

  27. [35]

    Communication errors in radiology–pitfalls and how to avoid them

    Stephen Waite, Jinel Moore Scott, Ian Drexler, Jennifer Martino, Alan Legasto, Brian Gale, and Srinivas Kolla. Communication errors in radiology–pitfalls and how to avoid them. Clinical imaging, 51:266–272, 2018

    2018

  28. [36]

    Multi- granularity cross-modal alignment for generalized medical visual representation learning

    Fuying Wang, Yuyin Zhou, Shujun Wang, Varut Vardhanabhuti, and Lequan Yu. Multi- granularity cross-modal alignment for generalized medical visual representation learning. Ad- vances in Neural Information Processing Systems, 35:33536–33549, 2022

    2022

  29. [37]

    Totalseg- mentator: robust segmentation of 104 anatomic structures in ct images

    Jakob Wasserthal, Hanns-Christian Breit, Manfred T Meyer, Maurice Pradella, Daniel Hinck, Alexander W Sauter, Tobias Heye, Daniel T Boll, Joshy Cyriac, Shan Yang, et al. Totalseg- mentator: robust segmentation of 104 anatomic structures in ct images. Radiology: Artificial Inte...

    2023

  30. [38]

    Medklip: Medical knowledge enhanced language-image pre-training for x-ray diagnosis

    Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Medklip: Medical knowledge enhanced language-image pre-training for x-ray diagnosis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21372–21383, 2023

    2023

  31. [39]

    Unimiss: Universal medical self-supervised learning via breaking dimensionality barrier

    Yutong Xie, Jianpeng Zhang, Yong Xia, and Qi Wu. Unimiss: Universal medical self-supervised learning via breaking dimensionality barrier. In European Conference on Computer Vision, pages 558–575. Springer, 2022. 12

    2022

  32. [40]

    Demystifying clip data

    Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang- Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. arXiv preprint arXiv:2309.16671, 2023

    2023 arXiv

  33. [41]

    Glipv2: unifying local- ization and vl understanding

    Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Harold Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: unifying local- ization and vl understanding. In Proceedings of the 36th International Conference on Neural Information Pr...

    2022

  34. [42]

    Biomedgpt: A unified and generalist biomedical generative pre-trained transformer for vision, language, and multimodal tasks

    Kai Zhang, Jun Yu, Eashan Adhikarla, Rong Zhou, Zhiling Yan, Yixin Liu, Zhengliang Liu, Lifang He, Brian Davison, Xiang Li, et al. Biomedgpt: A unified and generalist biomedical generative pre-trained transformer for vision, language, and multimodal tasks. arXiv e-prints, page...

    2023

  35. [43]

    Radgenome-chest ct: A grounded vision-language dataset for chest ct analysis

    Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Jiayu Lei, Ya Zhang, Yanfeng Wang, and Weidi Xie. Radgenome-chest ct: A grounded vision-language dataset for chest ct analysis. arXiv preprint arXiv:2404.16754, 2024

    2024 arXiv

  36. [44]

    Development of a large-scale medical visual question-answering dataset

    Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Development of a large-scale medical visual question-answering dataset. Communications Medicine, 4(1):277, 2024. 13 NeurIPS Paper Checklist

    2024

  37. [45]

    Each of these claims is supported by theoretical analysis, ablation studies, and experimental results

    Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: The claims made in the abstract and introduction are aligned with the actual contributions and findings presented in t...

  38. [46]

    Limitations

    Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: See conclusion section. Guidelines: • The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, ...

  39. [47]

    Guidelines: • The answer NA means that the paper does not include theoretical results

    Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? 14 Answer: [Yes] Justification: See section 3.1 and 3.2. Guidelines: • The answer NA means that the paper does not includ...

  40. [48]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and da...

  41. [49]

    Guidelines: • The answer NA means that paper does not include experiments requiring code

    Open access to data and code 15 Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We will publicly release...

  42. [50]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: Hyperparameter settings are r...

  43. [51]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [NA] Justification: We do not perform statistical significance testing ...

  44. [52]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: See Appendix B. Guideli...

  45. [53]

    All data used in this study are from publicly available, de- identified medical datasets, and no personally identifiable information (PII) was accessed or used

    Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: We have reviewed the NeurIPS Code of Ethics and confirm that our research confo...

  46. [54]

    Guidelines: 17 • The answer NA means that there is no societal impact of the work performed

    Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: See Appendix A. Guidelines: 17 • The answer NA means that there is no societal impact of the work perform...

  47. [55]

    Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: We use only pu...

  48. [56]

    Guidelines: • The answer NA means that the paper does not use existing assets

    Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We use publicly ...

  49. [57]

    These assets will be released with accompanying documentation upon paper acceptance

    New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We introduce several new assets as part of this work, including pretrained MedVista3D model checkpoints, LLM-rewritt...

  50. [58]

    All data used are from publicly available, de-identified medical datasets with appropriate licenses and do not involve any direct interaction with individuals

    Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...

  51. [59]

    Therefore, IRB approval was not required

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

  52. [60]

    Answer: [Yes] Justification: Large language models such as GPT-4o and Qwen2.5, were used to rewrite radiology reports for improving semantic clarity during pretraining

    Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the ...

    2025

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.