Pith. sign in

REVIEW 2 major objections 7 minor 106 references

MedPMC: A Systematic Framework for Scaling High-Fidelity Medical Multimodal Data for Foundation Models

T0 review · 2 major / 7 minor · reviewed 2026-07-09 · glm-5.2

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Cleaner medical data beats bigger data: 11M curated pairs top 24M

desk verdict Solid data infrastructure paper with a real attribution gap in its central experiment read the letter →

arxiv 2607.07673 v1 pith:EGQNXQP3 submitted 2026-07-08 cs.CV cs.LG

Hyunjae Kim , Dain Kim , Pan Xiao , Serina S. Applebaum , Younjoon Chung , Xuguang Ai , Yu Yin , Roy Jiang
show 20 more authors
Yuexi Du Yawen Wei Yiming Kong Tuo Guo Zhiyuan Cao Mengmeng Du Yuelei Fu Yan Hu Rui Shi Gui Yang Kevin W. Jin Yuntian Liu Yuxuan Tian Jonathan Marquez Zhen Chen Sheng Zhang Hoifung Poon Hua Xu Jaewoo Kang Qingyu Chen
This is my paper · ORCID
classification cs.CVcs.LG
keywords medicalmultimodalmodelsacrossdatamedpmcbenchmarksclinical
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the quality of image-text pairs matters more than raw quantity when training medical vision-language models. The authors built MedPMC, a five-stage automated pipeline that mines PubMed Central's open-access biomedical literature for clinically relevant images. The pipeline screens out non-medical figures (graphs, charts, molecular diagrams), decomposes compound multi-panel figures into individual subimages, aligns each subimage with its corresponding subcaption, and filters the result for medical relevance. Applied to 6.1 million articles, it produced 11 million image-text pairs with 95.3% medical relevance, versus 19.7% in a prior 24-million-pair dataset. The central claim is tested through a controlled experiment: a CLIP-style model trained on the 11M MedPMC corpus, using identical architecture and hyperparameters to a baseline trained on the 24M BIOMEDICA corpus, improved average zero-shot AUC by 7.1 percentage points across 26 benchmarks spanning 11 medical specialties, improved medical visual question-answering by up to 16.9 percentage points when used as the vision encoder in a multimodal language model, and improved morphology-to-image retrieval by 11.7 percentage points on 10,524 internal clinical dermatology photographs. The paper frames data curation not as preprocessing but as reusable, continuously updatable infrastructure, with modular components that can be individually benchmarked, swapped, and refined as new literature is published.

What carries the argument

The five-stage curation pipeline (initial screening via PubMedBERT text classifier, multi-panel figure detection via Vision Transformer, figure separation via YOLOv10, caption separation and alignment via supervised InternVL-2.5-4B, medical figure classification via Vision Transformer) is the central object. The controlled comparison design—identical CLIP architecture, initialization, hyperparameters, and training schedule, with only the training corpus changed—is the mechanism that isolates dataset quality as the causal variable. The modular benchmark suite for each pipeline stage is the infrastructure that makes the framework auditable and continuously improvable.

What would settle it

If a CLIP model trained on a random 11M subset of BIOMEDICA (without MedPMC's curation but matched for size) achieved comparable performance to MedPMC-CLIP, the curation pipeline's contribution would be confounded with corpus size effects. Alternatively, if the pipeline's synthetic GPT-4T labels were replaced with human annotations and the final CLIP model's performance changed substantially, the gains would be attributable to label quality rather than data curation per se.

Watch

Extended reading notes

Core claim

The paper's central discovery is that systematically removing non-medical images, decomposing compound figures into panel-level units, and aligning each panel with its specific subcaption yields training data that produces stronger medical AI models than datasets more than twice as large but lacking this curation. The mechanism is signal-to-noise: when 80% of a corpus consists of graphs, charts, and schematics rather than clinically relevant images, the contrastive learning objective wastes capacity on irrelevant visual content. By filtering to 95.3% medical relevance and achieving panel-level image-text correspondence, each training pair provides supervision at the level where clinical视觉特征最

Load-bearing premise

The controlled comparison attributing all performance gains to dataset quality assumes that the only meaningful difference between MedPMC's 11M pairs and BIOMEDICA's 24M pairs is the curation applied. However, three of the five curation pipeline stages use GPT-4 Turbo to generate synthetic training labels, and if those synthetic labels encode systematic biases or medical knowledge that the downstream CLIP model then absorbs, the performance gain could partly reflect distilled

Editorial extensions

If this is right

  • If data fidelity dominates data scale for medical multimodal pretraining, then the field's emphasis on building ever-larger corpora may be less productive than investing in curation pipelines that improve signal-to-noise ratio.
  • The modular, continuously updatable pipeline design means that as new medical imaging modalities, diseases, and terminology emerge in the literature, the corpus can be refreshed semiannually without rebuilding from scratch, keeping pretrained models current.
  • The finding that literature-derived images transfer to clinical dermatology retrieval suggests that curated biomedical publications can serve as a pretraining substrate that complements, rather than competes with, institution-specific clinical datasets.
  • The release of component-level benchmarks for each curation stage establishes a standardized evaluation framework that could drive competition and improvement in individual curation subtasks, much as task-specific benchmarks have advanced other areas of machine learning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The use of GPT-4 Turbo to generate synthetic training labels for three of the five pipeline stages means the curation pipeline's quality is partly bounded by GPT-4T's own medical knowledge. If GPT-4T systematically mislabels certain image types or introduces domain-specific biases, these errors propagate into the curated corpus and could be learned by downstream models. The paper does not ablate t
  • The 7.1-point AUC gain over BMC-CLIP is attributed solely to dataset quality, but the two corpora differ in composition (MedPMC has 11M pairs, BIOMEDICA has 24M), and the smaller corpus may benefit from a more favorable signal-to-noise ratio per gradient update rather than per pair. A scaling-curve comparison at matched corpus sizes would more cleanly isolate curation quality from dataset size eff
  • The embedding-space analysis showing MedPMC dermatology images overlap more with clinical photographs than existing public dermatology datasets do suggests that biomedical literature may capture visual diversity that curated clinical datasets miss, but this could also reflect publication bias toward visually distinctive or representative cases rather than the full distribution of routine clinical
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 7 minor

Summary. This manuscript introduces MedPMC, a five-stage modular pipeline for curating medical image-text pairs from permissively licensed PubMed Central articles. The pipeline performs initial screening, multi-panel figure detection, figure separation, joint caption separation and alignment, and medical figure classification, yielding 11M pairs from 6.1M articles. The authors train MedPMC-CLIP using the exact architecture and training protocol of BMC-CLIP (which trains on the 24M-pair BIOMEDICA corpus) to isolate the effect of the dataset. They report a 7.1 pp improvement in average zero-shot AUC across 26 benchmarks spanning 11 specialties, gains in two MLLM QA benchmarks when MedPMC-CLIP replaces the vision encoder in LLaVA-Med, and an 11.7 pp improvement in Recall@5 on a morphology-to-image retrieval task using 10,524 internal clinical dermatology photographs. The framework, corpus, benchmarks, and model checkpoints are publicly released.

Significance. The paper makes a substantial contribution as a resource: a reproducible, continuously updatable curation pipeline with component-level benchmarks, a large curated medical image-text corpus, and pretrained models all publicly released. The controlled CLIP experiment—matching architecture, initialization, hyperparameters, and training schedule to BMC-CLIP—is a well-designed end-to-end test of whether curation quality translates into downstream gains. The inclusion of 95% confidence intervals via paired bootstrap across benchmarks, the independent clinical dermatology evaluation on internal patient data, and the embedding-space analysis of distributional alignment with clinical images are all commendable. The component-level benchmark suite comparing against prior pipeline approaches (PMC-OA, MedICaT, GPT-4T zero-shot) adds engineering value and positions the framework for community refinement.

major comments (2)
  1. §2.4, paragraph on training details: The paper states that the head-to-head comparison with BMC-CLIP 'isolated the effect of dataset quality.' However, the two corpora differ in at least three load-bearing ways simultaneously: (1) medical image density (MedPMC is 95.3% medical vs. 19.7% for BIOMEDICA, yielding ~10.5M vs. ~4.7M medical pairs—a roughly 2.2× difference in medical training examples), (2) total pair count (11M vs. 24M), and (3) curation depth (multi-panel decomposition, joint caption alignment). The 7.1 pp AUC gain is attributed to 'high-fidelity curation,' but without an ablation that controls for the number of medical training pairs—e.g., training on a medical-filtered subset of BIOMEDICA without decomposition or alignment—the evidence supports the broad claim that curation helps but does not specifically validate that the distinctive pipeline stages (beyond filtering) are载
  2. §2.1, Caption separation and alignment: The paper states that samples where the number of generated subcaptions does not match the number of input subfigures are removed to 'prioritize pair fidelity.' The pipeline goes from 29M subfigures to 12.5M subfigure-subcaption pairs at this stage—a 57% reduction. This is a substantial filter that could introduce systematic selection bias (e.g., excluding figures with many panels or complex layouts, which may represent important clinical content). The paper does not quantify the mismatch rate, analyze what types of figures are dropped, or discuss the potential impact on dataset coverage. This information is needed to assess whether the resulting corpus is representative of the medical literature or biased toward simpler figure layouts.
minor comments (7)
  1. §2.1, Initial screening: The paper reports F1=93.2 for initial screening but does not report precision and recall separately in the main text. Given that this stage gates all downstream processing, reporting these would help readers assess the false-negative rate (medically relevant figures lost) and false-positive rate (wasted downstream computation).
  2. §4.1, Validation and model selection: The validation sets include synthetic annotations generated by the same GPT-4T procedure used for training data. While the test sets use manual labels, the use of synthetic labels in validation could bias model selection toward models that align with GPT-4T's labeling tendencies. A brief discussion of this potential bias would strengthen the methodology section.
  3. Fig. 2a: The BIOMEDICA sample size (432 images) is much smaller than the MedPMC sample (2,906 images). The confidence intervals on the BIOMEDICA composition estimates are correspondingly wider. It would be helpful to note this asymmetry or report CIs on the category proportions.
  4. §2.4, Downstream medical QA: The MMMU improvement of 1.9 pp has a 95% CI of [-4.1, 8.0], which includes zero. The text acknowledges this, but the abstract states the improvement without qualification ('improved medical visual question-answering by 1.9 and 16.9 percentage points across two benchmarks'). The abstract should note that the MMMU result is not statistically significant.
  5. Extended Data Table 2: The caption separation and alignment training set lists '1,664 (M)' from MedICaT, but §4.1 states '1,361 manually annotated samples from the MedICaT dataset.' Please reconcile these numbers.
  6. §4.4, Morphology-to-skin image retrieval: The set-based matching criterion for retrieval evaluation is described, but the specific threshold or matching rule (subset vs. exact match of concept sets) could be stated more precisely in the main text rather than requiring the reader to consult the cited references.
  7. The paper uses 'GPT-4' and 'GPT-4T' somewhat interchangeably in places (e.g., §4.1 multi-panel figure detection says 'GPT-4' while other sections say 'GPT-4T'). Standardizing the terminology would improve clarity.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the careful and constructive review. The referee raises two major comments: (1) the CLIP comparison with BMC-CLIP does not isolate which aspects of curation drive the observed gains, and (2) the caption alignment filter drops 57% of subfigures without analysis of potential selection bias. Both points are well-taken. We will address them through a combination of new analysis and revised language in the manuscript.

read point-by-point responses

  1. Referee: §2.4: The head-to-head comparison with BMC-CLIP does not isolate the effect of curation quality because the corpora differ in medical image density, total pair count, and curation depth simultaneously. An ablation controlling for the number of medical training pairs is needed.

    Authors: The referee is correct that the three differences between MedPMC and BIOMEDICA (medical pair density, total pair count, and curation depth) are confounded in our current experimental design. We agree that the phrase 'isolated the effect of dataset quality' overstates what the comparison demonstrates. The experiment isolates the effect of the training corpus as a whole, not the individual contributions of filtering, decomposition, or alignment. We will revise the language in §2.4 and the Discussion to accurately characterize the comparison as isolating the effect of the training corpus while acknowledging that the corpora differ along multiple dimensions simultaneously. Regarding the proposed ablation—training on a medical-filtered subset of BIOMEDICA without decomposition or alignment—we agree this would be informative and will attempt it as a supplementary analysis. However, we note a practical constraint: BIOMEDICA does not provide per-image medical relevance labels, so constructing such a subset requires applying our own medical figure classifier to the BIOMEDICA corpus. This means the ablation would partially use MedPMC's own curation components, making it a cleaner test of 'filtering alone vs. filtering plus decomposition and alignment' rather than a fully independent test. We will be transparent about this limitation if the ablation is included. Regardless of whether the ablation is feasible within the revision timeline, we will soften the causal claim and explicitly enumerate the confounded factors. revision: partial

  2. Referee: §2.1: The caption separation and alignment stage drops 57% of subfigures (from 29M to 12.5M) when subcaption counts do not match subfigure counts. This could introduce systematic selection bias toward simpler layouts. The paper does not quantify the mismatch rate, analyze dropped figure types, or discuss impact on coverage.

    Authors: This is a fair and important point. We will add analysis to the revised manuscript. Specifically, we plan to: (1) quantify the mismatch rate as a function of panel count, showing how the drop-off concentrates among figures with many subpanels; (2) sample and manually categorize a subset of dropped figures to characterize what types of content are disproportionately excluded; and (3) add a discussion paragraph acknowledging the potential for selection bias toward simpler layouts and its implications for dataset coverage. We agree that the current manuscript does not adequately address this trade-off between fidelity and coverage. The revised text will explicitly note that the filter prioritizes pair-level fidelity at the cost of excluding complex multi-panel figures, which may underrepresent certain clinical content types. We will also discuss potential future approaches to recovering some of the dropped pairs (e.g., partial alignment or relaxed matching criteria) as future work. revision: yes

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity found; central claims evaluated on independent external benchmarks and manual test sets.

full rationale

The paper's central claims are not circular. The CLIP comparison (MedPMC-CLIP vs BMC-CLIP) uses 26 external public benchmarks and an independent YNHHS clinical dermatology cohort, with architecture and training protocol held fixed — the improvement is measured against data not used in training. The pipeline component benchmarks do use GPT-4T-generated synthetic labels for training data, and the validation sets also include synthetic annotations from the same procedure (e.g., initial screening validation is 9,781 synthetic examples with zero manual labels per Extended Data Table 2). This creates a partial loop where GPT-4T influences both training and model selection. However, the paper explicitly states: 'Importantly, the final reported performance of each curation component was assessed on held-out real instances with manual labels,' and the test sets per Extended Data Table 2 are composed of existing benchmark data (ImageCLEF, MedICaT, DocFigure) and newly manually annotated samples (L), not synthetic labels. The MLLM evaluation uses MMMU and OmniMedVQA, both external. No self-citation chain is load-bearing: BMC-CLIP (ref 31, Lozano et al.) and LLaVA-Med (ref 29, Li et al.) are external works by different author groups. The skeptic's concern about the controlled comparison not disentangling medical-image filtering from higher-fidelity curation stages is an attribution/mechanism gap, not a circularity issue. The reader's concern about GPT-4T grading its own distillates applies to validation, not to the final test results. Score 1 reflects the minor validation-set concern that does not propagate to the reported findings.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical entities, particles, or forces. It is an engineering and data science contribution. The 'invented' elements are the pipeline models themselves, which are instantiations of existing architectures (PubMedBERT, ViT, YOLOv10, InternVL). The primary axioms are domain assumptions about the sufficiency of text for medical filtering and the reliability of GPT-4T for synthetic label generation.

free parameters (2)
  • Pipeline model hyperparameters = Various (e.g., lr=3e-5, batch=64)
    Standard hyperparameters for fine-tuning PubMedBERT, ViT, YOLOv10, InternVL-2.5-4B; not free parameters in the theoretical sense but fitted to the task.
  • CLIP training config = batch=8192, lr=1e-6
    Inherited from BMC-CLIP to isolate dataset variable; not tuned for MedPMC specifically.
assumptions (4)
  • domain assumption Biomedical figure captions and inline reference text contain sufficient signal to distinguish medical from non-medical figures before image download.
    Underlies the initial screening stage (§2.1, Methods 4.1). Validated by F1=93.2 on held-out manual labels.
  • domain assumption A supervised 4B MLLM can jointly perform caption separation and subfigure-subcaption alignment more accurately than sequential rule-based or CLIP-based methods.
    Underlies the caption separation stage (§2.1, Methods 4.1). Validated by F1=81.4 vs PMC-OA F1=48.5.
  • ad hoc to paper GPT-4 Turbo generates sufficiently accurate synthetic labels for training pipeline components.
    Used to generate 99,321 training labels (Methods 4.1). No direct ablation on the effect of GPT-4T label noise on final pipeline output quality is provided.
  • domain assumption Public benchmark datasets are largely independent of the PMC-derived pretraining corpus.
    Stated in Contamination-aware evaluation design; provenance checked but image-level overlap cannot be fully excluded.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MedPMC: A Systematic Framework for Scaling High-Fidelity Medical Multimodal Data for Foundation Models." pith.science (2026). https://pith.science/paper/EGQNXQP3

@misc{pith2026260707673,
  author       = {Pith},
  title        = {Pith review of: MedPMC: A Systematic Framework for Scaling High-Fidelity Medical Multimodal Data for Foundation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EGQNXQP3}},
  note         = {Machine review of arXiv:2607.07673}
}
read the original abstract

Medicine is inherently multimodal, requiring clinicians to synthesize information across diverse data streams. Yet the development of multimodal foundation models is constrained by limited access to large-scale, high-quality clinical data. Although PubMed Central (PMC) offers a complementary source of expert-authored image-text data, existing PMC-derived resources remain limited in fidelity, reproducibility, and clinical validation. We introduce MedPMC, an automated, continuously updatable framework that transforms permissively licensed literature into high-fidelity infrastructure for medical multimodal models. Applied to 6.1 million PMC articles, MedPMC curated 11 million medical image-text pairs. Component evaluations showed strong performance for initial screening (F1 = 93.2), multi-panel figure detection (F1 = 96.5), figure separation (mAP = 89.8), caption separation and alignment (F1 = 81.4; ROUGE-L = 85.3), and medical figure classification (F1 = 96.5). Manual review by five annotators, three with medical training, found 95.3% of MedPMC images medically relevant, versus 19.7% in a prior PMC-derived dataset. Across 26 benchmarks spanning 11 specialties, a MedPMC-trained CLIP-style model improved average zero-shot AUC by 7.1 percentage points over the strongest architecture-matched biomedical CLIP baseline despite using fewer than half as many image-text pairs. As the vision encoder in a multimodal large language model, it improved medical visual question-answering by 1.9 and 16.9 percentage points across two benchmarks. In 10,524 Yale New Haven Health System dermatology photographs, it improved morphology-to-image retrieval Recall@5 by 11.7 percentage points. These findings show that high-fidelity literature curation strengthens medical multimodal foundation models across benchmark and clinical settings. We publicly release the framework, corpus, benchmarks, and pretrained models.

Figures

Figures reproduced from arXiv: 2607.07673 by the authors.

Figure 1
Figure 1. MedPMC framework for dataset curation and its application to multimodal model training. a, Overview of the proposed five-step framework for constructing a large-scale medical image-text dataset from PMC articles. This pipeline yields 11M medical image-text pairs with improved data quality. b, Model development pipeline. A CLIP-style vision-language model is first trained on the curated dataset via contrastive learni… view at source ↗
Figure 2
Figure 2. Overview of MedPMC dataset composition, modality distribution, and temporal growth. a, Distribution of image categories in MedPMC compared to BIOMEDICA, highlighting a substantial reduction in non-medical images (e.g., graphs and charts) and increased coverage of clinically relevant modalities. b, Distribution of radiology imaging modalities within the MedPMC dataset. c, The number of publications and extracted figu… view at source ↗
Figure 3
Figure 3. Evaluation results. a, Image classification performance across 11 medical specialties, where scores are averaged over 26 benchmarks; MedPMC-CLIP consistently outperforms prior models across accuracy, F1 score, and AUC. b, Comparison of MedPMC-CLIP and BMC-CLIP on image classification performance across medical specialties; MedPMC-CLIP outperformed BMC-CLIP in 10 of 11 specialties. c, Performance comparison on multim… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Embedding-space comparison of external dermatology image sources with internal clinical dermatology images. a, Empirical cumulative distribution of nearest-neighbor cosine distances from internal clinical dermatology images to each external image source in the DINOv2 e…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

106 extracted references · 106 canonical work pages

  1. [1]

    N., Falcone, G

    Acosta, J. N., Falcone, G. J., Rajpurkar, P. & Topol, E. J. Multimodal biomedical ai.Nat. medicine 28, 1773–1784 (2022)

    work page 2022

  2. [2]

    Moor, M.et al.Foundation models for generalist medical artificial intelligence.Nature616, 259–265 (2023)

    work page 2023

  3. [3]

    Cui, C.et al.Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: a review.Prog. Biomed. Eng.5, 022001 (2023)

    work page 2023

  4. [4]

    & Lungren, M

    Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I. & Lungren, M. P. Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. NPJ digital medicine3, 136 (2020)

    work page 2020

  5. [5]

    He, Y .et al.Foundation model for advancing healthcare: challenges, opportunities and future directions.IEEE Rev. Biomed. Eng.18, 172–191 (2024)

    work page 2024

  6. [6]

    Rajendran, S.et al.Learning across diverse biomedical data modalities and cohorts: Challenges and opportunities for innovation.Patterns5(2024)

    work page 2024

  7. [7]

    image analysis105, 103621 (2025)

    Schouten, D.et al.Navigating the landscape of multimodal ai in medicine: a scoping review on technical challenges and clinical applications.Med. image analysis105, 103621 (2025)

    work page 2025

  8. [8]

    & Metaxas, D

    Zhang, S. & Metaxas, D. On the challenges and perspectives of foundation models for medical image analysis.Med. image analysis91, 102996 (2024)

    work page 2024

Show all 106 references
  1. [9]

    Acevedo, A.et al.A dataset of microscopic peripheral blood cell images for development of automatic recognition systems.Data brief30, 105474 (2020)

    2020

  2. [10]

    Sq.rs–3 (2025)

    Huang, S.-C.et al.A systematic review and implementation guidelines of multimodal foundation models in medical imaging.Res. Sq.rs–3 (2025)

    2025

  3. [11]

    & Hernandez-Boussard, T

    Koul, A., Duran, D. & Hernandez-Boussard, T. Synthetic data, synthetic trust: navigating data challenges in the digital revolution.The Lancet Digit. Heal.7(2025)

    2025

  4. [12]

    & Panagiotakos, D

    Liu, F. & Panagiotakos, D. Real-world data: a brief review of the methods, applications, challenges and opportunities.BMC Med. Res. Methodol.22, 287 (2022)

    2022

  5. [13]

    & Missier, P

    Mandreoli, F., Ferrari, D., Guidetti, V ., Motta, F. & Missier, P. Real-world data mining meets clinical practice: Research challenges and perspective.Front. big Data5, 1021621 (2022)

    2022

  6. [14]

    InProceedings of the AAAI conference on artificial intelligence, vol

    Irvin, J.et al.Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. InProceedings of the AAAI conference on artificial intelligence, vol. 33, 590–597 (2019)

    2019

  7. [15]

    Neural Inf

    Huang, S.-C.et al.Inspect: a multimodal dataset for patient outcome prediction of pulmonary embolisms.Adv. Neural Inf. Process. Syst.36, 17742–17772 (2023). 22/32

    2023

  8. [16]

    Ai-readi: rethinking ai data collection, preparation and sharing in diabetes research and beyond.Nat

    AI-READI Project Managers & NIH Program Scientists. Ai-readi: rethinking ai data collection, preparation and sharing in diabetes research and beyond.Nat. Metab.6, 2210–2212 (2024)

    2024

  9. [17]

    & V ogt, J

    Sokol, K., Fackler, J. & V ogt, J. E. Artificial intelligence should genuinely support clinical reasoning and decision making to bridge the translational gap.npj Digit. Medicine8, 345 (2025)

    2025

  10. [18]

    Jiang, R.et al.Are multimodal llms ready for clinical dermatology? a real-world evaluation in dermatology.arXiv preprint arXiv:2605.04098(2026)

    2026 arXiv

  11. [19]

    Medicine1–9 (2026)

    Bedi, S.et al.Holistic evaluation of large language models for medical tasks with medhelm.Nat. Medicine1–9 (2026)

    2026

  12. [20]

    D.et al.Addressing barriers in fair data practices for biomedical data.Sci

    Hughes, L. D.et al.Addressing barriers in fair data practices for biomedical data.Sci. Data10, 98 (2023)

    2023

  13. [21]

    & Kou, T

    Huang, X., Zhu, S., Lv, Z., Zhou, Q. & Kou, T. The open sharing operation mechanism of health data in the digital healthcare era: A study based on grounded theory and interpretative structural modeling method.Digit. Heal.11, 20552076251353694 (2025)

    2025

  14. [22]

    Cao, Z.et al.The development landscape of large language models for biomedical applications. Annu. review biomedical data science8(2025)

    2025

  15. [23]

    Zhang, X.et al.Pmc-vqa: Visual instruction tuning for medical visual question answering.arXiv preprint arXiv:2305.10415(2023)

    2023 arXiv

  16. [24]

    Chen, Z.et al.From compound figures to composite understanding: Developing a multi-modal llm from biomedical literature with medical multiple-image benchmarking and validation.arXiv preprint arXiv:2511.22232(2025)

    2025

  17. [25]

    About pmc

    National Library of Medicine. About pmc. https://pmc.ncbi.nlm.nih.gov/about/intro/. Accessed: 2026-05-31

    2026

  18. [26]

    InInternational Conference on Medical Image Computing and Computer-Assisted Intervention, 525–536 (Springer, 2023)

    Lin, W.et al.Pmc-clip: Contrastive language-image pre-training using biomedical documents. InInternational Conference on Medical Image Computing and Computer-Assisted Intervention, 525–536 (Springer, 2023)

    2023

  19. [27]

    & De Melo, G

    Eslami, S., Meinel, C. & De Melo, G. Pubmedclip: How much does clip benefit visual question answering in the medical domain? InFindings of the Association for Computational Linguistics: EACL 2023, 1181–1193 (2023)

    2023

  20. [28]

    & Sun, J

    Wang, Z., Wu, Z., Agarwal, D. & Sun, J. Medclip: Contrastive learning from unpaired medical images and text. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 3876–3887 (2022)

    2022

  21. [29]

    Li, C.et al.Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Adv. Neural Inf. Process. Syst.36, 28541–28564 (2023)

    2023

  22. [30]

    Qin, Z.et al.V olmo: Versatile and open large models for ophthalmology.arXiv preprint arXiv:2603.23953(2026)

    2026

  23. [31]

    InProceedings of the Computer Vision and Pattern Recognition Conference, 19724–19735 (2025)

    Lozano, A.et al.Biomedica: An open biomedical image-caption archive, dataset, and vision- language models derived from scientific literature. InProceedings of the Computer Vision and Pattern Recognition Conference, 19724–19735 (2025)

    2025

  24. [32]

    In International conference on machine learning, 8748–8763 (PmLR, 2021)

    Radford, A.et al.Learning transferable visual models from natural language supervision. In International conference on machine learning, 8748–8763 (PmLR, 2021). 23/32

    2021

  25. [33]

    & Katlam, A

    Karthik, K., Kamath, S., Supreetha, R. & Katlam, A. Content-based medical retrieval systems with evidence-based diagnosis for enhanced clinical decision support.Expert. Syst. with Appl.272, 126678 (2025)

    2025

  26. [34]

    & Liang, X

    Zhou, X., Han, X., Li, H., Wang, J. & Liang, X. Cross-domain image retrieval: methods and applications.Int. J. Multimed. Inf. Retr.11, 199–218 (2022)

    2022

  27. [35]

    M.et al.Multimodal generative ai for medical image interpretation.Nature639, 888–896 (2025)

    Rao, V . M.et al.Multimodal generative ai for medical image interpretation.Nature639, 888–896 (2025)

    2025

  28. [36]

    J.et al.Medical image processing, analysis and visualization in clinical research

    McAuliffe, M. J.et al.Medical image processing, analysis and visualization in clinical research. InProceedings 14th IEEE symposium on computer-based medical systems. CBMS 2001, 381–386 (IEEE, 2001)

    2001

  29. [37]

    Artificial intelligence in clinical medicine: challenges across diagnostic imaging, clinical decision support, surgery, pathology, and drug discovery.Clin

    Ogut, E. Artificial intelligence in clinical medicine: challenges across diagnostic imaging, clinical decision support, surgery, pathology, and drug discovery.Clin. practice15, 169 (2025)

    2025

  30. [38]

    & Müller, H

    García Seco de Herrera, A., Schaer, R., Bromuri, S. & Müller, H. Overview of the ImageCLEF 2016 medical task. InWorking Notes of CLEF 2016 (Cross Language Evaluation Forum)(2016)

    2016

  31. [39]

    In Findings of the Association for Computational Linguistics: EMNLP 2020, 2112–2120 (2020)

    Subramanian, S.et al.Medicat: A dataset of medical images, captions, and textual references. In Findings of the Association for Computational Linguistics: EMNLP 2020, 2112–2120 (2020)

    2020

  32. [40]

    & Jawahar, C

    Jobin, K., Mondal, A. & Jawahar, C. Docfigure: A dataset for scientific document figure classification. In2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), vol. 1, 74–79 (IEEE, 2019)

    2019

  33. [41]

    I.et al.A multimodal data processing pipeline for mimic-iv dataset.arXiv preprint arXiv:2601.11606(2026)

    Adiba, F. I.et al.A multimodal data processing pipeline for mimic-iv dataset.arXiv preprint arXiv:2601.11606(2026)

    2026

  34. [42]

    Wornow, M.et al.The shaky foundations of large language models and foundation models for electronic health records.npj digital medicine6, 135 (2023)

    2023

  35. [43]

    ACM Transactions on Comput

    Gu, Y .et al.Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Comput. for Healthc. (HEALTH)3, 1–23 (2021)

    2021

  36. [44]

    InInternational Conference on Learning Representations(2021)

    Dosovitskiy, A.et al.An image is worth 16x16 words: Transformers for image recognition at scale. InInternational Conference on Learning Representations(2021)

    2021

  37. [45]

    InProceedings of the 2024 SIAM International Conference on Data Mining (SDM), 118–126 (SIAM, 2024)

    Meng, S.et al.Yolo-ocr: End-to-end compound figure separation and label recognition of images in scientific publications. InProceedings of the 2024 SIAM International Conference on Data Mining (SDM), 118–126 (SIAM, 2024)

    2024

  38. [46]

    neural information processing systems37, 107984–108011 (2024)

    Wang, A.et al.Yolov10: Real-time end-to-end object detection.Adv. neural information processing systems37, 107984–108011 (2024). 47.Hurst, A.et al.Gpt-4o system card.arXiv preprint arXiv:2410.21276(2024). 48.Chen, Z.et al.Expanding performance boundaries of open-source multimo...

    2024 arXiv

  39. [47]

    & Lee, Y

    Liu, H., Li, C., Wu, Q. & Lee, Y . J. Visual instruction tuning.Adv. neural information processing systems36, 34892–34916 (2023)

    2023

  40. [48]

    In European Conference on Computer Vision, 304–323 (Springer, 2024)

    McKinzie, B.et al.Mm1: methods, analysis and insights from multimodal llm pre-training. In European Conference on Computer Vision, 304–323 (Springer, 2024). 24/32

    2024

  41. [49]

    M.et al.A clinically accessible small multimodal radiology model and evaluation metric for chest x-ray findings.Nat

    Zambrano Chaves, J. M.et al.A clinically accessible small multimodal radiology model and evaluation metric for chest x-ray findings.Nat. Commun.16, 3108 (2025)

    2025

  42. [50]

    & Atkins, M

    Sadeghi, M., Chilana, P., Yap, J., Tschandl, P. & Atkins, M. S. Using content-based image retrieval of dermoscopic images for interpretation and education: A pilot study.Ski. Res. Technol.26, 503–512 (2020)

    2020

  43. [51]

    Gassner, M.et al.Saliency-enhanced content-based image retrieval for diagnosis support in derma- tology consultation: Reader study.JMIR dermatology6, e42129 (2023)

    2023

  44. [52]

    & Geissbuhler, A

    Müller, H., Michoux, N., Bandon, D. & Geissbuhler, A. A review of content-based image retrieval systems in medical applications—clinical benefits and future directions.Int. journal medical informatics73, 1–23 (2004)

    2004

  45. [53]

    R., Antani, S., Deserno, T

    Long, L. R., Antani, S., Deserno, T. M. & Thoma, G. R. Content-based image retrieval in medicine: retrospective assessment, state of the art, and future directions.Int. J. Healthc. Inf. Syst. Informatics (IJHISI)4, 1–16 (2009)

    2009

  46. [54]

    InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 1820–1828 (2021)

    Groh, M.et al.Evaluating deep neural networks trained on clinical images in dermatology with the fitzpatrick 17k dataset. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 1820–1828 (2021)

    2021

  47. [55]

    Open7, e2446615 (2024)

    Ward, A.et al.Creating an empirical dermatology dataset through crowdsourcing with web search advertisements.JAMA Netw. Open7, e2446615 (2024). 58.DermNet New Zealand Trust. Dermnet. Accessed: 2026-06-17

    2024

  48. [56]

    & Friedrich, C

    Pelka, O., Koitka, S., Rückert, J., Nensa, F. & Friedrich, C. M. Radiology objects in context (roco): A multimodal image dataset. InInternational Workshop on Large-scale Annotation of Biomedical data and Expert Label Synthesis, 180–189 (Springer, 2018)

    2018

  49. [57]

    Zhang, S.et al.A multimodal biomedical foundation model trained from fifteen million image–text pairs.NEJM AI2, AIoa2400640 (2025)

    2025

  50. [58]

    S., Ashkezari, S., Dolatabadi, E

    Baghbanzadeh, N., Islam, M. S., Ashkezari, S., Dolatabadi, E. & Afkanpour, A. Open-pmc-18m: A high-fidelity large scale medical dataset for multimodal representation learning.arXiv preprint arXiv:2506.02738(2025)

    2025

  51. [59]

    InMachine learning for health (ML4H), 353–367 (PMLR, 2023)

    Moor, M.et al.Med-flamingo: a multimodal medical few-shot learner. InMachine learning for health (ML4H), 353–367 (PMLR, 2023)

    2023

  52. [60]

    medicine30, 3129–3141 (2024)

    Zhang, K.et al.A generalist vision–language foundation model for diverse biomedical tasks.Nat. medicine30, 3129–3141 (2024)

    2024

  53. [61]

    InInternational Conference on Machine Learning, 37975–37995 (PMLR, 2025)

    Lin, T.et al.Healthgpt: A medical large vision-language model for unifying comprehension and generation via heterogeneous knowledge adaptation. InInternational Conference on Machine Learning, 37975–37995 (PMLR, 2025). 65.Sellergren, A.et al.Medgemma technical report.arXiv prep...

    2025 arXiv

  54. [62]

    84, 1388–1395 (2024)

    Wang, Z.et al.Nci cancer research data commons: resources to share key cancer data.Cancer Res. 84, 1388–1395 (2024)

    2024

  55. [63]

    In2024 IEEE international conference on bioinformatics and biomedicine (BIBM), 3138–3143 (IEEE, 2024)

    Dai, D.et al.Pa-llava: A large language-vision assistant for human pathology image understanding. In2024 IEEE international conference on bioinformatics and biomedicine (BIBM), 3138–3143 (IEEE, 2024). 25/32

    2024

  56. [64]

    C., Massiris, M

    Nievas Offidani, M., Roffet, F., González Galtier, M. C., Massiris, M. & Delrieux, C. An open-source clinical case dataset for medical image classification and multimodal ai applications.Data10, 123 (2025)

    2025

  57. [65]

    Wu, D.et al.Gestaltmml: Enhancing rare genetic disease diagnosis through multimodal machine learning combining facial images and clinical texts.ArXivarXiv–2312 (2024)

    2024

  58. [66]

    InProceedings of the 31st ACM international conference on multimedia, 547–556 (2023)

    Yuan, Z.et al.Ramm: Retrieval-augmented biomedical visual question answering with multi-modal pre-training. InProceedings of the 31st ACM international conference on multimedia, 547–556 (2023)

    2023

  59. [67]

    & Wynn, R

    Nissen, T. & Wynn, R. The clinical case report: a review of its merits and limitations.BMC research notes7, 264 (2014)

    2014

  60. [68]

    InThe Twelfth International Conference on Learning Representations(2024)

    Xu, H.et al.Demystifying clip data. InThe Twelfth International Conference on Learning Representations(2024)

    2024

  61. [69]

    In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 7346– 7370 (2024)

    Chen, J.et al.Towards injecting medical visual knowledge into multimodal llms at scale. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 7346– 7370 (2024)

    2024

  62. [70]

    Bedi, S.et al.Testing and evaluation of health care applications of large language models: a systematic review.Jama333, 319–328 (2025)

    2025

  63. [71]

    communications16, 3280 (2025)

    Chen, Q.et al.Benchmarking large language models for biomedical natural language processing applications and recommendations.Nat. communications16, 3280 (2025)

    2025

  64. [72]

    & Kostakos, P

    Sheikhi, S., Loven, L. & Kostakos, P. Beyond the leaderboard: A survey of the science of evaluation, benchmarking, and methodologies for large language models.IEEE Access(2026)

    2026

  65. [73]

    Commun.(2026)

    Li, A.et al.Memorization in large language models in medicine prevalence characteristics and implications.Nat. Commun.(2026)

    2026

  66. [74]

    Chang, Y .et al.A survey on evaluation of large language models.ACM transactions on intelligent systems technology15, 1–45 (2024)

    2024

  67. [75]

    Liu, X.et al.A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis.The lancet digital health1, e271–e297 (2019)

    2019

  68. [76]

    Pytorch image models

    Wightman, R. Pytorch image models. https://github.com/rwightman/pytorch-image-models, DOI: 10.5281/zenodo.4414861 (2019)

    2019 doi

  69. [77]

    & Hutter, F

    Loshchilov, I. & Hutter, F. Decoupled weight decay regularization. InInternational Conference on Learning Representations(2017)

    2017

  70. [78]

    Yu, J.et al.Coca: Contrastive captioners are image-text foundation models.Transactions on Mach. Learn. Res.(2022)

    2022

  71. [79]

    InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2818–2829 (2023)

    Cherti, M.et al.Reproducible scaling laws for contrastive language-image learning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2818–2829 (2023)

    2023

  72. [80]

    Neural Inf

    Lozano, A.et al.Micro-bench: A microscopy benchmark for vision-language understanding.Adv. Neural Inf. Process. Syst.37, 30670–30685 (2024)

    2024

  73. [81]

    InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 9556–9567 (2024)

    Yue, X.et al.Mmmu: A massive multi-discipline multimodal understanding and reasoning bench- mark for expert agi. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 9556–9567 (2024). 26/32

    2024

  74. [82]

    InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 22170–22183 (2024)

    Hu, Y .et al.Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical lvlm. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 22170–22183 (2024)

    2024

  75. [83]

    R., Novoa, R

    Daneshjou, R., Yuksekgonul, M., Cai, Z. R., Novoa, R. & Zou, J. Y . Skincon: A skin disease dataset densely annotated by domain experts for fine-grained debugging and analysis.Adv. Neural Inf. Process. Syst.35, 18157–18167 (2022)

    2022

  76. [84]

    Agarwal, S.et al.gpt-oss-120b & gpt-oss-20b model card.arXiv preprint arXiv:2508.10925(2025)

    2025 arXiv

  77. [85]

    & Su, Y .-T

    Liu, A.-A., Nie, W.-Z., Gao, Y . & Su, Y .-T. Multi-modal clique-graph matching for view-based 3d model retrieval.IEEE Transactions on Image Process.25, 2103–2116 (2016)

    2016

  78. [86]

    & König-Ries, B

    Küster, U. & König-Ries, B. Evaluating semantic web service matchmaking effectiveness based on graded relevance. InThe 7th International Semantic Web Conference, vol. 35 (2008)

    2008

  79. [87]

    Oquab, M.et al.Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193(2023)

    2023 arXiv

  80. [88]

    & Großberger, L

    McInnes, L., Healy, J., Saul, N. & Großberger, L. Umap: Uniform manifold approximation and projection.J. Open Source Softw.3, 861 (2018)

    2018

  81. [89]

    Shih, G.et al.Augmenting the national institutes of health chest radiograph dataset with expert annotations of possible pneumonia.Radiol. Artif. Intell.1, e180041 (2019)

    2019

  82. [90]

    Yang, J., Shi, R. & Ni, B. Medmnist classification decathlon: A lightweight automl benchmark for medical image analysis. InIEEE 18th International Symposium on Biomedical Imaging (ISBI), 191–195 (2021)

    2021

  83. [91]

    Patterns3(2022)

    Liu, R.et al.Deepdrid: Diabetic retinopathy—grading and image quality estimation challenge. Patterns3(2022)

    2022

  84. [92]

    & Nyang, D

    Jung, C., Abuhamad, M., Mohaisen, D., Han, K. & Nyang, D. Wbc image classification and generative models based on convolutional neural network.BMC Med. Imaging22, 94 (2022)

    2022

  85. [93]

    B., Borah, H

    Hussain, E., Mahanta, L. B., Borah, H. & Das, C. R. Liquid based-cytology pap smear dataset for automated multi-class diagnosis of pre-cancerous and cervical cancer lesions.Data brief30, 105589 (2020)

    2020

  86. [94]

    N.et al.Multi-class texture analysis in colorectal cancer histology.Sci

    Kather, J. N.et al.Multi-class texture analysis in colorectal cancer histology.Sci. reports6, 27988 (2016)

    2016

  87. [95]

    A.et al.Lung and colon cancer histopathological image dataset (lc25000).arXiv preprint arXiv:1912.12142(2019)

    Borkowski, A. A.et al.Lung and colon cancer histopathological image dataset (lc25000).arXiv preprint arXiv:1912.12142(2019)

    1912 arXiv

  88. [96]

    Litjens, G.et al.1399 h&e-stained sentinel lymph node sections of breast cancer patients: the camelyon dataset.GigaScience7, giy065 (2018)

    2018

  89. [97]

    communications10, 2173 (2019)

    Tang, Z.et al.Interpretable classification of alzheimer’s disease pathologies with a convolutional neural network pipeline.Nat. communications10, 2173 (2019)

    2019

  90. [98]

    R.et al.Deep learning from multiple experts improves identification of amyloid neu- ropathologies.Acta neuropathologica communications10, 66 (2022)

    Wong, D. R.et al.Deep learning from multiple experts improves identification of amyloid neu- ropathologies.Acta neuropathologica communications10, 66 (2022)

    2022

  91. [99]

    J.et al.A deep-learning classifier identifies patients with clinical heart failure using whole-slide images of h&e tissue.PloS one13, e0192726 (2018)

    Nirschl, J. J.et al.A deep-learning classifier identifies patients with clinical heart failure using whole-slide images of h&e tissue.PloS one13, e0192726 (2018). 27/32

    2018

  92. [100]

    communications14, 692 (2023)

    Wu, G.-H.et al.Cryoet reveals organelle phenotypes in huntington disease patient ipsc-derived and mouse primary neurons.Nat. communications14, 692 (2023)

    2023

  93. [101]

    & Kittler, H

    Tschandl, P., Rosendahl, C. & Kittler, H. The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions.Sci. data5, 180161 (2018)

    2018

  94. [102]

    Data10, 3 (2023)

    Carstens, M.et al.The dresden surgical anatomy dataset for abdominal organ segmentation in surgical data science.Sci. Data10, 3 (2023)

    2023

  95. [103]

    communications8, 463 (2017)

    Eulenberg, P.et al.Reconstructing cell cycle and disease progression using deep learning.Nat. communications8, 463 (2017)

    2017

  96. [104]

    Commun.15, 1022 (2024)

    Burgess, J.et al.Orientation-invariant autoencoders learn robust representations for shape profiling of cells and organelles.Nat. Commun.15, 1022 (2024)

    2024

  97. [105]

    Iudin, A.et al.Empiar: the electron microscopy public image archive.Nucleic Acids Res.51, D1503–D1511 (2023)

    2023

  98. [106]

    In2020 IEEE International Conference on Image Processing (ICIP), 2456–2460 (IEEE, 2020)

    Battiato, S.et al.Pollen13k: A large scale microscope pollen grain image dataset. In2020 IEEE International Conference on Image Processing (ICIP), 2456–2460 (IEEE, 2020). Acknowledgements This study is supported by the National Institutes of Health National Library of Medicine...

    2020

Pith tools

Reviewed July 9, 2026 · model on record in the stance chip above.