Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

Unifying Biomedical Vision-Language Expertise: Towards a Generalist Foundation Model via Multi-CLIP Knowledge Distillation

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that distilling nine biomedical CLIP models into a single student produces a model that beats every teacher across 58 datasets, nine imaging modalities, and six task types.

desk verdict A useful engineering-scale distillation paper whose benchmark suite will be reused, but whose 'outperforms all teachers' claim is undercut by missing survival comparisons and shaky HR numbers. read the letter →

arxiv 2506.22567 v1 pith:6WNBVWKS submitted 2025-06-27 cs.CV cs.AI

Shansong Wang , Zhecheng Jin , Mingzhe Hu , Mojtaba Safari , Feng Zhao , Chih-Wei Chang , Richard LJ Qiu , Justin Roper
show 2 more authors
David S. Yu Xiaofeng Yang
This is my paper · ORCID
classification cs.CVcs.AI
keywords biomedicalvision-languagemodelCLIPknowledgedistillationmulti-teachermedicalimage-textpretrainingzero-shotclassificationvisualquestionansweringsurvivalprediction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a single biomedical vision-language model can be built by distilling nine existing medical CLIP teachers into one student, without needing billion-scale raw image-text data. The student, MMKD-CLIP, is first pretrained on 2.9 million filtered PMC-OA image-text pairs and then refined on 19.2 million teacher feature pairs, and the paper reports it outperforms every one of its nine teachers across 58 datasets, nine imaging modalities, and six task types. If that is right, it matters because it turns fragmented specialty models into a practical route to a generalist medical AI: assemble the best available CLIPs and distill them once, rather than collecting enormous new corpora or training many separate models. The evidence spans zero-shot classification, linear probing, cross-modal retrieval, visual question answering, survival prediction, and cancer diagnosis.

What carries the argument

The machinery is offline multi-teacher feature distillation with two complementary objectives. Feature distillation (FD) minimizes the mean squared error between student and teacher image and text embeddings, and interactive contrastive learning (ICL) runs a contrastive loss that uses the student's image embedding as query against teacher text embeddings, and vice versa, increasing mutual information between student and teacher representations; both losses are combined with a small CLIP contrastive loss. Before distillation, each image-text pair is passed through nine teachers in a five-way zero-shot test, and a teacher is kept only if its softmax score for the correct caption exceeds 0.90, which yields trusted teacher features and defines the 19.2M quadruplets. A dual-stream autoencoder with per-CLIP projection encoders and decoders maps the teachers' native feature dimensions (768 for some, 512 for others) into a shared 512-dimensional space without contrastive disruption.

What would settle it

Collect the image-text pairs where no teacher reaches the 0.90 trustworthiness threshold, run MMKD-CLIP and all nine teachers on those excluded pairs for zero-shot classification or retrieval, and check whether the student still outperforms every teacher; if it does not, the claimed generalization holds only for samples at least one teacher already handled well.

Watch

Extended reading notes

Core claim

The paper's central discovery is that multi-teacher knowledge distillation can merge the strengths of radiology, pathology, ophthalmology, and generalist biomedical CLIP models into one student that beats each teacher individually. Concretely, MMKD-CLIP first runs CLIP-style contrastive pretraining on 2,911,190 image-text pairs covering 26 imaging modalities, then uses nine pretrained biomedical CLIP models as offline teachers, extracting 19,229,852 image-text-feature quadruplets and training the student with a feature-distillation loss plus an interactive contrastive loss against teacher embeddings. On the 58-dataset evaluation, the paper reports modality-level first place in 7 of 9 zero-shot classification modalities and consistent statistically significant gains over the best teacher on retrieval, VQA, survival prediction, and cancer diagnosis. The authors read this as evidence that multi-teacher distillation is a scalable paradigm for building a generalist biomedical foundation model under real-world data constraints.

Load-bearing premise

The load-bearing assumption is that a teacher is trustworthy for an image-text pair when it gives the correct caption better than 90 percent confidence in a five-way choice, which silently excludes the hardest or most unusual pairs from distillation and from the story the benchmark numbers tell.

Editorial extensions

If this is right

  • A single distilled model can replace a suite of specialty CLIP models for zero-shot classification, retrieval, and VQA, reducing the need to train and serve many separate backbones.
  • Low-label clinical settings gain the most: at 1% training data, linear probing on MRI improved by 5.46 AUC points over the second-best model, suggesting cheaper annotation for new tasks.
  • The same student supports prognostic use: fusing image and report features yields a higher concordance index (0.6873 vs 0.6675 for PLIP) across 12 TCGA cancer types.
  • The pipeline remains open to new teachers, so future biomedical CLIPs can be folded into the student without retraining from scratch.
  • The distilled student is at least competitive in supervised cancer diagnosis, reaching AUCs of 87.42% to 97.28% after fine-tuning on four histopathology benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves an untested population behind: image-text pairs where no teacher passed the 0.90 trustworthiness filter were simply absent from distillation, so the reported generalization may mostly cover samples at least one teacher already understood.
  • Because all nine teachers were trained on overlapping open-access biomedical literature, the student cannot create knowledge beyond the union of teacher competencies; a genuinely new modality would require a teacher that already knows it.
  • The same multi-teacher distillation recipe could transfer to other domains with several existing CLIP-style models, provided the trustworthiness threshold is re-calibrated for each domain's label noise.
  • The retrieval wins, while statistically significant, are modest in absolute terms (e.g., Recall@50 near 15% on a noisy clinical corpus), so a practitioner should judge whether the improvement matters for their operating point.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The manuscript proposes MMKD-CLIP, a biomedical CLIP-style student trained in two stages: contrastive pretraining on 2.9M PMC-OA image-text pairs, followed by feature-level distillation from nine biomedical CLIP teachers using 19.2M teacher feature pairs. Teachers are selected per sample by requiring a softmax confidence above 0.90 in a five-way zero-shot caption-matching task, and their heterogeneous feature dimensions are aligned with CLIP-specific projections and a shared autoencoder. The student is evaluated on 58 datasets spanning zero-shot classification, linear probing, cross-modal retrieval, VQA, survival prediction, and cancer diagnosis. The paper's central claim is that the student consistently outperforms every teacher model. The evidence supports superiority in many settings but not in the unqualified form used in the abstract: zero-shot results place the model top-3 in every modality but best/second-best on only 24 of 38 datasets, and survival prediction is compared only against PLIP rather than all nine teachers.

Significance. The paper contributes a large, reproducible distillation resource (19.2M feature quadruplets), a generally applicable two-stage multi-teacher distillation pipeline, and unusually broad benchmarking across 58 datasets and six task types, with code and data links provided. If the claims are confirmed, the work shows a practical route to a generalist medical CLIP without collecting new raw image-text corpora at billion-image scale. The main empirical result is meaningful but less sweeping than stated: because the student is trained to imitate teacher features, outperforming teachers is expected to some degree, although the external downstream evaluation prevents the result from being definitionally circular. The reported gains are often small (e.g., 0.24% AUC in fundus and OCT), so the statistical and filtering choices carry real weight.

major comments (5)
  1. [Section 2.2.5 and Fig. 5] Survival prediction is evaluated only against PLIP, not against the other eight teacher models, even though 12 of the 58 datasets come from this task. The abstract and contributions state unqualified superiority over all teachers, but the survival results provide no evidence for that comparison. Please benchmark all nine teachers on the TCGA cohorts or explicitly restrict the headline claim to the tasks where all teachers were compared.
  2. [Fig. 5a and Section 2.2.5] Several reported hazard ratios and 95% confidence intervals are internally inconsistent, for example HR=15.64 with CI 0.51-4.99, HR=8.22 with CI 0.65-3.57, and HR=8.05 with CI -0.01-4.18. A 95% CI that does not contain the point estimate, or that has a negative lower bound, indicates an error in calculation or reporting. These values must be recomputed and corrected before the risk-stratification evidence can be used to support the broad claim.
  3. [Section 4.2 and Fig. 7a] The trustworthy-teacher filter admits a teacher only when its softmax score on the correct text exceeds 0.90 in a five-way zero-shot choice. This likely removes hard, rare, or modality-shifted samples on which teachers disagree, yet the paper does not report the fraction of excluded samples per modality or the student's performance on the excluded set. Without that analysis, the claimed generalization across all nine modalities could largely reflect easy samples that individual teachers already handled.
  4. [Section 4.4 and Fig. 6b] The distillation objective contains tuned hyperparameters (alpha1=0.1, alpha2=50, alpha3=1, and the 0.90 trustworthiness threshold) but no sensitivity analysis over these choices. Since several headline improvements over the best teacher are small (e.g., 0.24% AUC in fundus and OCT), a sensitivity study is needed to show that the reported ordering is not an artifact of a single configuration.
  5. [Section 5 and figure captions] The statistical testing procedure is underspecified: it is not stated whether the Mann-Whitney U tests use bootstrap replicates, fold-wise outcomes, or dataset-level scores, and whether paired structure is accounted for. If bootstrap replicates are treated as independent samples, a p-value below 0.001 for a difference as small as 0.24% AUC is expected by construction. Please specify the test units and assumptions, and report effect sizes alongside p-values.
minor comments (5)
  1. [Section 4.9] The cancer type CESC is spelled as 'CESE' in the text; please correct the typo.
  2. [Section 4.13 and Results] The methods section lists C-Indextd, IBS, and INBLL as evaluation metrics, but none of these metrics is reported anywhere in the results; either report them or remove them from the metric list.
  3. [Throughout] The name 'PMC-CLIP' is written inconsistently as 'PMCCLIP' in several places; please unify the notation.
  4. [Fig. 2 caption] The caption contains a grammatical error: 'which estimated using the bootstrap method' should read 'which were estimated using the bootstrap method.'
  5. [Section 4.3] The text says the dimension-alignment autoencoder is 'as illustrated in Fig. 1 b,' but Fig. 1b depicts pretraining, not the dual-stream autoencoder; the intended reference appears to be Fig. 7b.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the student's outperformance over teachers is an empirical result, not an entailment of the distillation objective.

full rationale

The paper's central claim is empirical: a student trained with CLIP-style pretraining plus feature distillation and interactive contrastive loss from nine medical CLIP teachers outperforms those teachers on 58 external benchmarks. None of the loss terms in Eqs. (3)-(10) define or force this outcome; L_FD minimizes MSE to teacher features and L_ICL contrasts student features against teacher features, but neither equation contains the downstream benchmark metric (AUC, Recall, Accuracy, or C-Index). Evaluation datasets are external (MedMNIST, TCGA, SLAKE, VQA-RAD, etc.) and are not used to fit the student's parameters, so the headline result is measured rather than constructed. The 'trustworthy teacher' filter (Section 4.2) selects training pairs where a teacher exceeds 0.90 softmax confidence; this affects the training distribution but does not define the evaluation outcome. The only support gap is the survival task (Section 2.2.5), where only PLIP is compared despite the abstract claiming superiority over all nine teachers; this is an evidence gap and a correctness risk (the reported hazard-ratio confidence intervals in Fig. 5a are internally inconsistent), but it is not circularity. Self-citations (Refs. 9, 17, 18) are peripheral and not load-bearing, and the distillation theory citation (CLIP-KD, Ref. 27) is external. No derivation step reduces to its own inputs by construction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No free parameters beyond common training hyperparameters; two hand-set design choices (loss weights, trust threshold) directly shape the distilled model. No new entities are postulated.

free parameters (2)
  • Distillation loss weights alpha1, alpha2, alpha3 = 0.1, 50, 1
    Hand-set in Eq. (10); no sensitivity analysis or search procedure reported; they control the balance of CLIP, feature-distillation, and interactive-contrastive losses.
  • Trustworthy teacher threshold = 0.90
    Used in Section 4.2 to decide whether a teacher contributes features for a given pair; no ablation on threshold; changes size and composition of the 19.2M distillation set.
assumptions (4)
  • domain assumption Filtered PMC-OA subset is representative enough for a generalist biomedical representation
    The paper filters BIOMEDICA Concept-Filtering to 2.9M pairs with labels Microscopy/Clinical Imaging; if these labels or captions are noisy or skewed, the student inherits the bias.
  • ad hoc to paper Teacher features remain informative after autoencoder dimension alignment
    Section 4.3 trains dual-stream autoencoders with reconstruction losses; no downstream validation that the aligned space retains clinically relevant information.
  • ad hoc to paper Five-way zero-shot classification with 0.90 softmax threshold is a valid trustworthiness filter
    Section 4.2/Fig. 7a; a teacher may score high on an easy forced-choice pair and still be unreliable on hard clinical images.
  • standard math InfoNCE and MSE objectives are appropriate for aligning student and teacher representations
    Section 4.4; standard approach, supported by CLIP-KD citation; no new proof.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unifying Biomedical Vision-Language Expertise: Towards a Generalist Foundation Model via Multi-CLIP Knowledge Distillation." pith.science (2026). https://pith.science/paper/6WNBVWKS

@misc{pith2026250622567,
  author       = {Pith},
  title        = {Pith review of: Unifying Biomedical Vision-Language Expertise: Towards a Generalist Foundation Model via Multi-CLIP Knowledge Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6WNBVWKS}},
  note         = {Machine review of arXiv:2506.22567}
}
read the original abstract

CLIP models pretrained on natural images with billion-scale image-text pairs have demonstrated impressive capabilities in zero-shot classification, cross-modal retrieval, and open-ended visual answering. However, transferring this success to biomedicine is hindered by the scarcity of large-scale biomedical image-text corpora, the heterogeneity of image modalities, and fragmented data standards across institutions. These limitations hinder the development of a unified and generalizable biomedical foundation model trained from scratch. To overcome this, we introduce MMKD-CLIP, a generalist biomedical foundation model developed via Multiple Medical CLIP Knowledge Distillation. Rather than relying on billion-scale raw data, MMKD-CLIP distills knowledge from nine state-of-the-art domain-specific or generalist biomedical CLIP models, each pretrained on millions of biomedical image-text pairs. Our two-stage training pipeline first performs CLIP-style pretraining on over 2.9 million biomedical image-text pairs from 26 image modalities, followed by feature-level distillation using over 19.2 million feature pairs extracted from teacher models. We evaluate MMKD-CLIP on 58 diverse biomedical datasets, encompassing over 10.8 million biomedical images across nine image modalities. The evaluation spans six core task types: zero-shot classification, linear probing, cross-modal retrieval, visual question answering, survival prediction, and cancer diagnosis. MMKD-CLIP consistently outperforms all teacher models while demonstrating remarkable robustness and generalization across image domains and task settings. These results underscore that multi-teacher knowledge distillation is a scalable and effective paradigm for building high-performing biomedical foundation models under the practical constraints of real-world data availability.

Figures

Figures reproduced from arXiv: 2506.22567 by the authors.

Figure 1
Figure 1. Overview of MMKD-CLIP dataset curation, model training, and evaluation. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Zero-shot classification performance of MMKD-CLIP across nine medical imaging modalities. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Linear probing image classification and cross-modal retrieval performance of MMKD-CLIP. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Visual question answering performance and qualitative examples. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Survival prediction with MMKD-CLIP and PLIP. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Supervised cancer diagnosis performance and ablation study of MMKD-CLIP. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Trustworthy teacher selection, multi-teacher feature alignment, and teacher feature distribution. [PITH_FULL_IMAGE:figures/full_fig_p026_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Capabilities of GPT-5 on Multimodal Medical Reasoning

    cs.CL 2025-08 reject novelty 4.0 of 10

    A benchmark study reports GPT-5 outperforming GPT-4o and pre-licensed human experts on most medical QA tasks, but not consistently on VQA-RAD.

Reference graph

Works this paper leans on

80 extracted references · 53 canonical work pages · cited by 1 Pith paper

  1. [1]

    Clip and complementary methods

    Markus Hafner, Maria Katsantoni, Tino Köster, James Marks, Joyita Mukherjee, Dorothee Staiger, Jernej Ule, and Mihaela Zavolan. Clip and complementary methods. Nature Reviews Methods Primers, 1(1):20, 2021

    work page 2021

  2. [2]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PmLR, 2021

    2021

  3. [3]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26296–26306, June 2024

    work page 2024

  4. [4]

    Foundation models for generalist medical artificial intelligence

    Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M Krumholz, Jure Leskovec, Eric J Topol, and Pranav Rajpurkar. Foundation models for generalist medical artificial intelligence. Nature, 616(7956):259–265, 2023

    2023

  5. [5]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888–12900. PMLR, 2022

    2022

  6. [6]

    Cxr-clip: Toward large scale chest x-ray language-image pre-training

    Kihyun You, Jawook Gu, Jiyeon Ham, Beomhee Park, Jiho Kim, Eun K Hong, Woonhyuk Baek, and Byungseok Roh. Cxr-clip: Toward large scale chest x-ray language-image pre-training. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 101–111. Springer, 2023

    work page 2023

  7. [7]

    Merlin: A vision language foundation model for 3d computed tomography

    Louis Blankemeier, Joseph Paul Cohen, Ashwin Kumar, Dave Van Veen, Syed Jamal Safdar Gardezi, Magdalini Paschali, Zhihong Chen, Jean-Benoit Delbrouck, Eduardo Reis, Cesar Truyts, et al. Merlin: A vision language foundation model for 3d computed tomography. Research Square, pages rs–3, 2024

    2024

  8. [8]

    Artificial intelligence for multimodal data integration in oncology

    Jana Lipkova, Richard J Chen, Bowen Chen, Ming Y Lu, Matteo Barbieri, Daniel Shao, Anurag J Vaidya, Chengkuan Chen, Luoting Zhuang, Drew FK Williamson, et al. Artificial intelligence for multimodal data integration in oncology. Cancer cell, 40(10):1095–1110, 2022

    work page 2022

Show all 80 references
  1. [9]

    Triad: Vision foundation model for 3d magnetic resonance imaging

    Shansong Wang, Mojtaba Safari, Qiang Li, Chih-Wei Chang, Richard LJ Qiu, Justin Roper, David S Yu, and Xiaofeng Yang. Triad: Vision foundation model for 3d magnetic resonance imaging. arXiv preprint arXiv:2502.14064, 2025

    2025 arXiv

  2. [10]

    A whole-slide foundation model for digital pathology from real-world data

    Hanwen Xu, Naoto Usuyama, Jaspreet Bagga, Sheng Zhang, Rajesh Rao, Tristan Naumann, Cliff Wong, Zelalem Gero, Javier González, Yu Gu, et al. A whole-slide foundation model for digital pathology from real-world data. Nature, 630(8015):181–188, 2024

    2024

  3. [11]

    A visual–language foundation model for pathology image analysis using medical twitter

    Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas J Montine, and James Zou. A visual–language foundation model for pathology image analysis using medical twitter. Nature medicine, 29(9):2307–2316, 2023. 21

    2023

  4. [12]

    A foundation model for generalizable disease detection from retinal images

    Yukun Zhou, Mark A Chia, Siegfried K Wagner, Murat S Ayhan, Dominic J Williamson, Robbert R Struyven, Timing Liu, Moucheng Xu, Mateo G Lozano, Peter Woodward-Court, et al. A foundation model for generalizable disease detection from retinal images. Nature, 622(7981):156–163, 2023

    2023

  5. [13]

    Ophclip: Hierarchical retrieval-augmented learning for ophthalmic surgical video-language pretraining

    Ming Hu, Kun Yuan, Yaling Shen, Feilong Tang, Xiaohao Xu, Lin Zhou, Wei Li, Ying Chen, Zhongxing Xu, Zelin Peng, et al. Ophclip: Hierarchical retrieval-augmented learning for ophthalmic surgical video-language pretraining. arXiv preprint arXiv:2411.15421, 2024

    2024 arXiv

  6. [14]

    Transparent medical image ai via an image–text foundation model grounded in medical literature

    Chanwoo Kim, Soham U Gadgil, Alex J DeGrave, Jesutofunmi A Omiye, Zhuo Ran Cai, Roxana Daneshjou, and Su-In Lee. Transparent medical image ai via an image–text foundation model grounded in medical literature. Nature Medicine, 30(4):1154–1165, 2024

    2024

  7. [15]

    A generalist vision–language foundation model for diverse biomedical tasks

    Kai Zhang, Rong Zhou, Eashan Adhikarla, Zhiling Yan, Yixin Liu, Jun Yu, Zhengliang Liu, Xun Chen, Brian D Davison, Hui Ren, et al. A generalist vision–language foundation model for diverse biomedical tasks. Nature Medicine, pages 1–13, 2024

    2024

  8. [16]

    Quantifying the reasoning abilities of llms on real-world clinical cases

    Pengcheng Qiu, Chaoyi Wu, Shuyu Liu, Weike Zhao, Zhuoxia Chen, Hongfei Gu, Chuanjin Peng, Ya Zhang, Yanfeng Wang, and Weidi Xie. Quantifying the reasoning abilities of llms on real-world clinical cases. arXiv preprint arXiv:2503.04691, 2025

    2025 arXiv

  9. [17]

    Unsupervised mri motion artifact disentanglement: introducing maudgan

    Mojtaba Safari, Xiaofeng Yang, Chih-Wei Chang, Richard L J Qiu, Ali Fatemi, and Louis Archambault. Unsupervised mri motion artifact disentanglement: introducing maudgan. Physics in Medicine & Biology, 69(11):115057, may 2024

    2024

  10. [18]

    Mri super-resolution reconstruction using efficient diffusion probabilistic model with residual shifting

    Mojtaba Safari, Shansong Wang, Zach Eidex, Qiang Li, Richard L J Qiu, Erik H Middlebrooks, David S Yu, and Xiaofeng Yang. Mri super-resolution reconstruction using efficient diffusion probabilistic model with residual shifting. Physics in Medicine & Biology, 2025

    2025

  11. [19]

    Multi-modal explainable medical ai assistant for trustworthy human-ai collaboration

    Honglong Yang, Shanshan Song, Yi Qin, Lehan Wang, Haonan Wang, Xinpeng Ding, Qixiang Zhang, Bodong Du, and Xiaomeng Li. Multi-modal explainable medical ai assistant for trustworthy human-ai collaboration. arXiv preprint arXiv:2505.06898, 2025

    2025 arXiv

  12. [20]

    Deep learning based multimodal biomedical data fusion: An overview and comparative review

    Junwei Duan, Jiaqi Xiong, Yinghui Li, and Weiping Ding. Deep learning based multimodal biomedical data fusion: An overview and comparative review. Information Fusion, page 102536, 2024

    2024

  13. [21]

    The future of multimodal artificial intelligence models for integrating imaging and clinical metadata: a narrative review

    Benjamin D Simon, Kutsev Bengisu Ozyoruk, David G Gelikman, Stephanie A Harmon, and Barı¸ s Türkbey. The future of multimodal artificial intelligence models for integrating imaging and clinical metadata: a narrative review. Diagnostic and interventional radiology (Ankara, Turk...

    2024

  14. [22]

    Pmc open access subset

    National Library of Medicine. Pmc open access subset. 2003. Accessed: 2025-05-25

    2003

  15. [23]

    Pmc-clip: Contrastive language-image pre-training using biomedical documents

    Weixiong Lin, Ziheng Zhao, Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc-clip: Contrastive language-image pre-training using biomedical documents. InInternational Conference on Medical Image Computing and Computer-Assisted Intervention, pages 525–536. Sp...

    2023

  16. [24]

    Biomedica: An open biomedical image-caption archive, dataset, and vision-language models derived from scientific literature

    Alejandro Lozano, Min Woo Sun, James Burgess, Liangyu Chen, Jeffrey J Nirschl, Jeffrey Gu, Ivan Lopez, Josiah Aklilu, Austin Wolfgang Katzer, Collin Chiu, et al. Biomedica: An open biomedical image-caption archive, dataset, and vision-language models derived from scientific li...

    2025 arXiv

  17. [25]

    Conceptclip: Towards trustworthy medical ai via concept-enhanced contrastive langauge-image pre-training

    Yuxiang Nie, Sunan He, Yequan Bie, Yihui Wang, Zhixuan Chen, Shu Yang, and Hao Chen. Conceptclip: Towards trustworthy medical ai via concept-enhanced contrastive langauge-image pre-training. arXiv preprint arXiv:2501.15579, 2025

    2025 arXiv

  18. [26]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural informa...

    2022

  19. [27]

    Clip-kd: An empirical study of clip model distillation

    Chuanguang Yang, Zhulin An, Libo Huang, Junyu Bi, Xinqiang Yu, Han Yang, Boyu Diao, and Yongjun Xu. Clip-kd: An empirical study of clip model distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15952–15962, 2024

    2024

  20. [28]

    Enabling multimodal generation on clip via vision-language knowledge distillation

    Wenliang Dai, Lu Hou, Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung. Enabling multimodal generation on clip via vision-language knowledge distillation. arXiv preprint arXiv:2203.06386, 2022

    2022 arXiv

  21. [29]

    Medicalnarratives: Connecting medical vision and language with localized narratives

    Wisdom O Ikezogwo, Kevin Zhang, Mehmet Saygin Seyfioglu, Fatemeh Ghezloo, Linda Shapiro, and Ranjay Krishna. Medicalnarratives: Connecting medical vision and language with localized narratives. arXiv preprint arXiv:2501.04184, 2025. 22

    2025

  22. [30]

    Medclip: Contrastive learning from unpaired medical images and text

    Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. Medclip: Contrastive learning from unpaired medical images and text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing, ...

    2022

  23. [31]

    A multimodal biomedical foundation model trained from fifteen million image–text pairs

    Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. A multimodal biomedical foundation model trained from fifteen million image–text pairs. NEJM AI, 2(1):AIoa2400640, 2025

    2025

  24. [32]

    Unimed-clip: Towards a unified image-text pretraining paradigm for diverse medical imaging modalities

    Muhammad Uzair Khattak, Shahina Kunhimon, Muzammal Naseer, Salman Khan, and Fahad Shahbaz Khan. Unimed-clip: Towards a unified image-text pretraining paradigm for diverse medical imaging modalities. arXiv preprint arXiv:2412.10372, 2024

    2024 arXiv

  25. [33]

    Sedigheh Eslami, Christoph Meinel, and Gerard De Melo. Pubmedclip: How much does clip benefit visual question answering in the medical domain? In Findings of the Association for Computational Linguistics: EACL 2023, pages 1181–1193, 2023

    2023

  26. [34]

    Quilt-1m: One million image-text pairs for histopathology

    Wisdom Ikezogwo, Saygin Seyfioglu, Fatemeh Ghezloo, Dylan Geva, Fatwir Sheikh Mohammed, Pavan Kumar Anand, Ranjay Krishna, and Linda Shapiro. Quilt-1m: One million image-text pairs for histopathology. Advances in neural information processing systems, 36:37995–38017, 2023

    2023

  27. [35]

    Medtrinity-25m: A large-scale multimodal dataset with multigranular annotations for medicine

    Yunfei Xie, Ce Zhou, Lang Gao, Juncheng Wu, Xianhang Li, Hong-Yu Zhou, Sheng Liu, Lei Xing, James Zou, Cihang Xie, et al. Medtrinity-25m: A large-scale multimodal dataset with multigranular annotations for medicine. arXiv preprint arXiv:2408.02900, 2024

    2024 arXiv

  28. [36]

    Multiple instance captioning: Learning representations from histopathol- ogy textbooks and articles

    Jevgenij Gamper and Nasir Rajpoot. Multiple instance captioning: Learning representations from histopathol- ogy textbooks and articles. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16549–16559, 2021

    2021

  29. [37]

    Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering

    Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th international symposium on biomedical imaging (ISBI), pages 1650–1654. IEEE, 2021

    2021

  30. [38]

    A dataset of clinically generated visual questions and answers about radiology images

    Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1–10, 2018

    2018

  31. [39]

    Komura, A

    D. Komura, A. Kawabe, K. Fukuta, K. Sano, T. Umezaki, H. Koda, R. Suzuki, K. Tominaga, M. Ochi, H. Konishi, F. Masakado, N. Saito, Y . Sato, T. Onoyama, S. Nishida, G. Furuya, H. Katoh, H. Yamashita, K. Kakimi, Y . Seto, T. Ushiku, M. Fukayama, and S. Ishikawa. Universal encod...

    2022

  32. [40]

    Bracs: A dataset for breast carcinoma subtyping in h&e histology images

    Nadia Brancati, Anna Maria Anniciello, Pushpak Pati, Daniel Riccio, Giosuè Scognamiglio, Guillaume Jaume, Giuseppe De Pietro, Maurizio Di Bonito, Antonio Foncubierta, Gerardo Botti, et al. Bracs: A dataset for breast carcinoma subtyping in h&e histology images. Database, 2022:...

    2022

  33. [41]

    The cancer genome atlas pan-cancer analysis project

    John N Weinstein, Eric A Collisson, Gordon B Mills, Kenna R Shaw, Brad A Ozenberger, Kyle Ellrott, Ilya Shmulevich, Chris Sander, and Joshua M Stuart. The cancer genome atlas pan-cancer analysis project. Nature genetics, 45(10):1113–1120, 2013

    2013

  34. [42]

    Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer

    Babak Ehteshami Bejnordi, Mitko Veta, Paul Johannes Van Diest, Bram Van Ginneken, Nico Karssemeijer, Geert Litjens, Jeroen AWM Van Der Laak, Meyke Hermsen, Quirine F Manson, Maschenka Balkenhol, et al. Diagnostic assessment of deep learning algorithms for detection of lymph no...

    2017

  35. [43]

    Demystifying clip data

    Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. arXiv preprint arXiv:2309.16671, 2023

    2023 arXiv

  36. [44]

    Biomedbert: A pre-trained biomedical language model for qa and ir

    Souradip Chakraborty, Ekaba Bisong, Shweta Bhatt, Thomas Wagner, Riley Elliott, and Francesco Mosconi. Biomedbert: A pre-trained biomedical language model for qa and ir. In Proceedings of the 28th international conference on computational linguistics, pages 669–679, 2020

    2020

  37. [45]

    Representation learning with contrastive predictive coding

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018

    2018 arXiv

  38. [46]

    Attention-based deep multiple instance learning

    Maximilian Ilse, Jakub Tomczak, and Max Welling. Attention-based deep multiple instance learning. In International conference on machine learning, pages 2127–2136. PMLR, 2018

    2018

  39. [47]

    A vision–language foundation model for precision oncology

    Jinxi Xiang, Xiyue Wang, Xiaoming Zhang, Yinghua Xi, Feyisope Eweje, Yijiang Chen, Yuchen Li, Colin Bergstrom, Matthew Gopaulchan, Ted Kim, et al. A vision–language foundation model for precision oncology. Nature, pages 1–10, 2025. 23

    2025

  40. [48]

    Tcga-reports: A machine-readable pathology report resource for benchmarking text-based ai models

    Jenna Kefeli and Nicholas Tatonetti. Tcga-reports: A machine-readable pathology report resource for benchmarking text-based ai models. Patterns, 5(3), 2024

    2024

  41. [49]

    Covid-net: A tailored deep convolutional neural network design for detection of covid-19 cases from chest x-ray images

    Linda Wang, Zhong Qiu Lin, and Alexander Wong. Covid-net: A tailored deep convolutional neural network design for detection of covid-19 cases from chest x-ray images. Scientific reports, 10(1):19549, 2020

    2020

  42. [50]

    Philip Kegelmeyer

    Michael Heath, Kevin Bowyer, Daniel Kopans, Richard Moore, and W. Philip Kegelmeyer. The digital database for screening mammography. In M. J. Yaffe, editor,Proceedings of the Fifth International Workshop on Digital Mammography, pages 212–218. Medical Physics Publishing, 2001

    2001

  43. [51]

    Two public chest x-ray datasets for computer-aided screening of pulmonary diseases

    Stefan Jaeger, Sema Candemir, Sameer Antani, Yì-Xiáng J Wáng, Pu-Xuan Lu, and George Thoma. Two public chest x-ray datasets for computer-aided screening of pulmonary diseases. Quantitative imaging in medicine and surgery, 4(6):475, 2014

    2014

  44. [52]

    Siim-acr pneumothorax segmentation

    Anna Zawacki, Carol Wu, George Shih, Julia Elliott, Mikhail Fomitchev, Mohannad Hussain, Paras Lakhani, Phil Culliton, and Shunxing Bao. Siim-acr pneumothorax segmentation. https://www.kaggle.com/compe titions/siim-acr-pneumothorax-segmentation , 2019. Kaggle competition. Acce...

    2019

  45. [53]

    Augmenting the national institutes of health chest radiograph dataset with expert annotations of possible pneumonia

    George Shih, Carol C Wu, Safwan S Halabi, Marc D Kohli, Luciano M Prevedello, Tessa S Cook, Arjun Sharma, Judith K Amorosa, Veronica Arteaga, Maya Galperin-Aizenberg, et al. Augmenting the national institutes of health chest radiograph dataset with expert annotations of possib...

    2019

  46. [54]

    Brain Tumor Multimodal Image (CT & MRI)

    Murtoza Likhon. Brain Tumor Multimodal Image (CT & MRI). https://www.kaggle.com/datasets/mu rtozalikhon/brain-tumor-multimodal-image-ct-and-mri , 2023. Accessed: 2025-06-02

    2023

  47. [55]

    A comprehensive and easy-to-use multi-domain multi-task medical imaging meta-dataset (medimeta)

    Stefano Woerner, Arthur Jaques, and Christian F Baumgartner. A comprehensive and easy-to-use multi-domain multi-task medical imaging meta-dataset (medimeta). arXiv preprint arXiv:2404.16000, 2024

    2024

  48. [56]

    Covid-net ct-2: Enhanced deep neural networks for detection of covid-19 from chest ct images through bigger, more diverse learning

    Hayden Gunraj, Ali Sabri, David Koff, and Alexander Wong. Covid-net ct-2: Enhanced deep neural networks for detection of covid-19 from chest ct images through bigger, more diverse learning. Frontiers in Medicine, 8:729287, 2022

    2022

  49. [57]

    Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification

    Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data, 10(1):41, 2023

    2023

  50. [58]

    Brain tumor mri dataset, 2021

    Msoud Nickparvar. Brain tumor mri dataset, 2021

    2021

  51. [59]

    Deep-learning-assisted diagnosis for knee magnetic resonance imaging: development and retrospective validation of mrnet

    Nicholas Bien, Pranav Rajpurkar, Robyn L Ball, Jeremy Irvin, Allison Park, Erik Jones, Michael Bereket, Bhavik N Patel, Kristen W Yeom, Katie Shpanskaya, et al. Deep-learning-assisted diagnosis for knee magnetic resonance imaging: development and retrospective validation of mr...

    2018

  52. [60]

    Role of inter-and extra-lesion tissue, transfer learning, and fine-tuning in the robust classification of breast lesions

    Iulia-Nela Anghelache Nastase, Simona Moldovanu, Keka C Biswas, and Luminita Moraru. Role of inter-and extra-lesion tissue, transfer learning, and fine-tuning in the robust classification of breast lesions. Scientific Reports, 14(1):22754, 2024

    2024

  53. [61]

    Dataset of breast ultrasound images

    Walid Al-Dhabyani, Mohammed Gomaa, Hussien Khaled, and Aly Fahmy. Dataset of breast ultrasound images. Data in brief, 28:104863, 2020

    2020

  54. [62]

    Diabetic retinopathy detection

    Emma Dugas, Jared, Jorge, and Will Cukierski. Diabetic retinopathy detection. https://kaggle.com/com petitions/diabetic-retinopathy-detection , 2015. Kaggle competition. Accessed: 2025-06-02

    2015

  55. [63]

    Automatic detection of 39 fundus diseases and conditions in retinal photographs using deep neural networks

    Ling-Ping Cen, Jie Ji, Jian-Wei Lin, Si-Tong Ju, Hong-Jie Lin, Tai-Ping Li, Yun Wang, Jian-Feng Yang, Yu-Fen Liu, Shaoying Tan, et al. Automatic detection of 39 fundus diseases and conditions in retinal photographs using deep neural networks. Nature communications, 12(1):4828, 2021

    2021

  56. [64]

    Fives: A fundus image dataset for artificial intelligence based vessel segmentation

    Kai Jin, Xingru Huang, Jingxing Zhou, Yunxiang Li, Yan Yan, Yibao Sun, Qianni Zhang, Yaqi Wang, and Juan Ye. Fives: A fundus image dataset for artificial intelligence based vessel segmentation. Scientific data, 9(1):475, 2022

    2022

  57. [65]

    Retinal oct image classification - c8 [data set]

    Obuli Sai Naren. Retinal oct image classification - c8 [data set]. https://doi.org/10.34740/KAGGLE/DS V/2736749, 2021. Kaggle. Accessed: 2025-06-02

    2021

  58. [66]

    Bach: Grand challenge on breast cancer histology images

    Guilherme Aresta, Teresa Araújo, Scotty Kwok, Sai Saketh Chennamsetty, Mohammed Safwan, Varghese Alex, Bahram Marami, Marcel Prastawa, Monica Chan, Michael Donovan, et al. Bach: Grand challenge on breast cancer histology images. Medical image analysis, 56:122–139, 2019

    2019

  59. [67]

    Lung and colon cancer histopathological image dataset (lc25000)

    Andrew A Borkowski, Marilyn M Bui, L Brannon Thomas, Catherine P Wilson, Lauren A DeLand, and Stephen M Mastorides. Lung and colon cancer histopathological image dataset (lc25000). arXiv preprint arXiv:1912.12142, 2019. 24

    1912 arXiv

  60. [68]

    100,000 histological images of human colorectal cancer and healthy tissue

    Jakob Nikolas Kather, Niels Halama, and Alexander Marx. 100,000 histological images of human colorectal cancer and healthy tissue. (No Title), 2018

    2018

  61. [69]

    Viable and necrotic tumor assessment from whole slide images of osteosarcoma using machine-learning and deep-learning models

    Harish Babu Arunachalam, Rashika Mishra, Ovidiu Daescu, Kevin Cederberg, Dinesh Rakheja, Anita Sengupta, David Leonard, Rami Hallac, and Patrick Leavey. Viable and necrotic tumor assessment from whole slide images of osteosarcoma using machine-learning and deep-learning models...

    2019

  62. [70]

    Large-scale pretraining on pathological images for fine-tuning of small pathological benchmarks

    Masakata Kawai, Noriaki Ota, and Shinsuke Yamaoka. Large-scale pretraining on pathological images for fine-tuning of small pathological benchmarks. In Workshop on Medical Image Learning with Limited and Noisy Data, pages 257–267. Springer, 2023

    2023

  63. [71]

    Multi-class texture analysis in colorectal cancer histology

    Jakob Nikolas Kather, Cleo-Aron Weis, Francesco Bianconi, Susanne M Melchers, Lothar R Schad, Timo Gaiser, Alexander Marx, and Frank Gerrit Zöllner. Multi-class texture analysis in colorectal cancer histology. Scientific reports, 6(1):1–11, 2016

    2016

  64. [72]

    Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study

    Jakob Nikolas Kather, Johannes Krisam, Pornpimol Charoentong, Tom Luedde, Esther Herpel, Cleo-Aron Weis, Timo Gaiser, Alexander Marx, Nektarios A Valous, Dyke Ferber, et al. Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multic...

    2019

  65. [73]

    Deep learning for the detection of anatomical tissue structures and neoplasms of the skin on scanned histopathological tissue sections

    Katharina Kriegsmann, Frithjof Lobers, Christiane Zgorzelski, Joerg Kriegsmann, Charlotte Janssen, Rolf Rüdinger Meliß, Thomas Muley, Ulrich Sack, Georg Steinbuss, and Mark Kriegsmann. Deep learning for the detection of anatomical tissue structures and neoplasms of the skin on...

    2022

  66. [74]

    Interpretable classification of alzheimer’s disease pathologies with a convolutional neural network pipeline

    Ziqi Tang, Kangway V Chuang, Charles DeCarli, Lee-Way Jin, Laurel Beckett, Michael J Keiser, and Brittany N Dugger. Interpretable classification of alzheimer’s disease pathologies with a convolutional neural network pipeline. Nature communications, 10(1):2173, 2019

    2019

  67. [75]

    Deep learning from multiple experts improves identification of amyloid neuropathologies

    Daniel R Wong, Ziqi Tang, Nicholas C Mew, Sakshi Das, Justin Athey, Kirsty E McAleese, Julia K Kofler, Margaret E Flanagan, Ewa Borys, Charles L White III, et al. Deep learning from multiple experts improves identification of amyloid neuropathologies. Acta neuropathologica com...

    2022

  68. [76]

    Kvasir: A multi-class image dataset for computer aided gastrointestinal disease detection

    Konstantin Pogorelov, Kristin Ranheim Randel, Carsten Griwodz, Sigrun Losada Eskeland, Thomas de Lange, Dag Johansen, Concetto Spampinato, Duc-Tien Dang-Nguyen, Mathias Lux, Peter Thelin Schmidt, Michael Riegler, and Pål Halvorsen. Kvasir: A multi-class image dataset for compu...

    2017

  69. [77]

    Montalbo

    Francis J. Montalbo. Wce curated colon disease dataset deep learning. https://www.kaggle.com/dataset s/francismon/curated-colon-dataset-for-deep-learning , 2022. Kaggle. Accessed: 2025-06-02

    2022

  70. [78]

    The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions

    Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1):1–9, 2018

    2018

  71. [79]

    Pad-ufes-20: A skin lesion dataset composed of patient data and clinical images collected from smartphones

    Andre GC Pacheco, Gustavo R Lima, Amanda S Salomao, Breno Krohling, Igor P Biral, Gabriel G de Angelo, Fábio CR Alves Jr, José GM Esgario, Alana C Simora, Pedro BC Castro, et al. Pad-ufes-20: A skin lesion dataset composed of patient data and clinical images collected from sma...

    2020

  72. [80]

    Benign": 0,

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 25 8 Supplementary materials 8.1 Model design, data distribution, and offline feature extraction pipeline We built MMKD-CLIP on a dual-stream transformer backbone co...

    2017 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.