Pith. sign in

REVIEW 4 major objections 4 minor 52 references

Benchmarking Foundation Models with Multimodal Public Electronic Health Records

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fusing chest X-ray and clinical-note embeddings with structured ICU data improves mortality and length-of-stay predictions without worsening group fairness, while current vision-language models lag, the paper claims.

desk verdict Useful, reproducible multimodal EHR benchmark with a clean pipeline and public code, but the abstract oversells 'consistent improvements' and the handling of missing modalities needs to be pinned down. read the letter →

arxiv 2507.14824 v1 pith:W5VOXJOF submitted 2025-07-20 cs.LG cs.AI

Kunyu Yu , Rui Yang , Jingchi Liao , Siqi Li , Huitao Li , Irene Li , Yifan Peng , Rishikesan Kamaleswaran
show 1 more author
Nan Liu
This is my paper
classification cs.LGcs.AI
keywords foundationmodelsmultimodalelectronichealthrecordsMIMIC-IVin-hospitalmortalitylengthofstayfairnessinterpretabilitylargevision-language
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper builds a public benchmark from the MIMIC-IV ICU database and asks whether embeddings from foundation models improve two clinical predictions—in-hospital mortality and an ICU stay longer than three days—when added to structured demographics and vital signs. The authors compare eight models, including domain-specific and general-purpose encoders for time series, chest X-rays, and clinical notes, plus three large vision-language models, and they evaluate fairness across age, gender, and race as well as feature importance. Their central claim is that adding image and text representations to structured data consistently improves predictive performance without worsening group fairness, and that small domain-specific encoders can match much larger general-purpose encoders on unimodal tasks. They also report that current vision-language models generalize poorly, especially to length-of-stay prediction, so modular pipelines remain the more reliable option. The contribution is a reproducible, analysis-ready multimodal EHR pipeline and a multidimensional comparison to guide model selection.

What carries the argument

The engine is the frozen-embedding linear probe. Each modality is encoded once by a foundation model—fixed-interval aggregation, a GRU, or Moment for vital signs; CXR-Foundation or Swin Transformer for chest X-rays; RadBERT or Text-Embedding-3-Large for notes—and the resulting vectors are concatenated and fed to a logistic-regression classifier, so any performance difference reflects the representation rather than the classifier. Fairness is measured with demographic parity and equalized odds computed across age, gender, and race subgroups, and interpretability uses SHAP values and logistic-regression coefficients. For the multimodal-learner arm, the machinery is a prompted evaluation that converts non-image data to text and asks closed-ended yes/no questions, with the percentage of answerable questions tracked.

What would settle it

Run the same benchmark on only complete-case ICU stays, namely patients with structured data, chest X-rays, and notes all present, and compare AUROC, AUPRC, and fairness metrics to the reported full-cohort results; if the complete-case numbers show that text-plus-image worsens mortality AUROC and that the model rankings change, the claim that multimodal fusion adds value without bias is not supported.

Watch

Extended reading notes

Core claim

The central discovery the paper argues for is that a modular two-stage design—frozen embeddings per modality concatenated into a single vector and classified by logistic regression—lets a hospital exploit multiple EHR modalities without retraining the encoders. On MIMIC-IV v2.2, the authors report that this design beats structured-data-only baselines on both tasks: the best mortality AUROC is 0.8943 with text embeddings from Text-Embedding-3-Large, and the best length-of-stay AUROC is 0.8532 with the same text encoder; adding CXR embeddings on top of text yields slightly lower numbers, which the authors attribute to missing modalities and the limits of simple concatenation. They further claim that multimodal integration does not harm group fairness, that domain-specific encoders such as CXR-Foundation and RadBERT stay competitive with general-purpose models trained at far larger scale, and that GRU-based vital-sign embeddings outperform the time-series foundation model Moment. For vision-language models, the paper claims that GPT-4o mini reaches comparable accuracy to the modular framework on mortality, but all tested LVLMs, including the medical LLaVA-Med, perform poorly on length-of-stay prediction, indicating limited task generalizability.

Load-bearing premise

The results stand on the assumption that a logistic-regression head on concatenated frozen embeddings is a fair way to compare foundation models, and that the handling of patients who lack images or notes does not distort the comparison.

Editorial extensions

If this is right

  • Hospitals can evaluate ICU risk models by combining frozen embeddings with a simple logistic head, since this setup exposes the value of each modality without encoder fine-tuning.
  • Text embeddings carry most of the predictive signal beyond structured vital signs; the reported tables show that stacking averaged CXR embeddings on top of text can slightly reduce AUROC, so modality additions should be validated per task.
  • Small, domain-specific encoders such as CXR-Foundation and RadBERT are a cost-effective substitute for much larger general-purpose encoders, at least for these two ICU tasks.
  • Current large vision-language models are not yet reliable for length-of-stay prediction; modular frameworks outperform them and should be preferred for deployment until LVLMs improve.
  • The released pipeline can be extended to additional MIMIC-IV modalities such as ICD codes and ECG signals, making the benchmark a starting point for broader multimodal ICU research.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Read carefully, the paper's numbers support 'adding a single modality to structured data helps' more strongly than 'adding every modality helps': in its mortality table, AUROC drops from 0.8943 with text alone to 0.8846 with text plus chest X-rays, so the consistency claim should be scoped to multimodal-versus-structured comparisons.
  • The fairness conclusion applies to the linear head used in the benchmark, not to arbitrary fusion architectures; expressive end-to-end models could redistribute errors across groups, so fairness should be re-audited whenever the fusion method changes.
  • Because the LVLM arm relies on a single prompt template and closed-ended answers, small template changes could shift results; a natural extension is to vary prompt wording and answer parsing and see whether the ranking of GPT-4o mini versus the modular framework survives.
  • The interpretability finding that image features become more important in the complete-case subset implies that missing-modality imputation drives apparent feature importance; reporting all headline results on the complete-case cohort would separate fusion effects from imputation effects.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents a benchmark evaluating eight foundation models on multimodal MIMIC-IV EHR data for two ICU prediction tasks (in-hospital mortality and length of stay). For unimodal encoders, the authors use a modular two-stage framework: frozen embeddings from structured vital signs, chest X-ray images, and clinical text are concatenated and fed to a logistic regression head. They also evaluate several LVLMs as multimodal learners. The benchmark includes predictive performance, fairness metrics across demographic subgroups, and interpretability via SHAP and logistic-regression coefficients. The central claim is that incorporating multiple data modalities leads to consistent improvements in predictive performance without introducing additional bias. The code is released on GitHub.

Significance. If the claims are substantiated, the paper would provide a useful standardized pipeline and a systematic reference for comparing unimodal and multimodal foundation models on a publicly available critical-care dataset. Strengths include the clear description of the data-processing pipeline, the use of bootstrap confidence intervals for the unimodal results, and the inclusion of both domain-specific and general-purpose models. The code release is a practical contribution that supports reproducibility. However, the paper's headline claims are currently weakened by an unspecified missing-data handling rule and by an abstract that overstates the consistency of the multimodal improvements.

major comments (4)
  1. [Section II-C / II-A2 and Tables II-III] The manuscript never states how patients without chest X-rays or clinical notes are handled when constructing the unimodal encoder evaluations. Section II-A2 mentions customizable cohort selection by modality availability, but it does not say whether the results in Tables II and III are computed on a common cohort, on per-row subcohorts, or with imputation. Section III-A3 explicitly says that imaging contribution 'increased substantially in the subset of patients without missing modalities,' which confirms that missingness is present and changes the measured signal. Without a fixed cohort or an explicit imputation rule, the incremental-value comparisons between rows of Tables II and III are confounded by documentation status and patient acuity, making the 'consistent improvements' claim untestable as reported.
  2. [Abstract and Section III-A1] The abstract's claim of 'consistent improvements in predictive performance' is contradicted by the paper's own results. In Table II, for in-hospital mortality, text alone (Text-Embedding-3-Large) achieves AUROC 0.8943 (95% CI 0.8863-0.9018), while adding CXR-Foundation image embeddings reduces it to 0.8846 (95% CI 0.8767-0.8931). Section III-A1 acknowledges 'a modest decline in performance was observed when both image and text modalities were combined, compared to using text alone.' The abstract should either qualify that the improvements are relative to the structured-data baseline only, or the analysis should be rerun on a single fixed cohort to support the stronger wording.
  3. [Section III-B / Figure 6] The LVLM evaluation lacks confidence intervals, even though the unimodal encoder evaluation uses 1,000 bootstrap samples and reports 95% CIs for all metrics. The comparison between GPT-4o mini and the modular framework in Figure 6 is also made across different metrics (accuracy/precision/recall/F1 for LVLMs versus AUROC/AUPRC/accuracy for the modular framework), so the statement that GPT-4o mini achieved 'comparable' performance is not well-defined. Reporting error bars or confidence intervals and aligning the metrics would be necessary to support the conclusions drawn in this section.
  4. [Section III-A2 and Figure 4b] The claim that incorporating multiple modalities 'did not introduce further bias' is based on visual inspection of Figure 4b without statistical tests or confidence intervals for demographic parity and equalized odds. Given the small sizes of some demographic subgroups and the multiple modality combinations being compared, the stability of these fairness metrics should be assessed quantitatively, for example by bootstrap intervals or by testing for significant differences across modality combinations.
minor comments (4)
  1. [Section II-C2] The paper refers to CXR-Foundation but cites ELIXR [28]; please clarify the model name and note that ELIXR was pretrained on MIMIC-CXR, which shares the same patient population as MIMIC-IV, and discuss the potential for inflated performance due to this overlap.
  2. [Section II-C1] The GRU model is trained on the same MIMIC-IV data used for evaluation, unlike the other foundation models which are frozen. This difference should be explicitly acknowledged in the comparison, since the GRU is not a foundation model in the same sense.
  3. [Figure 6] Adding confidence intervals or error bars to Figure 6 would bring the LVLM evaluation in line with the rest of the benchmark and improve interpretability.
  4. [Author list] There is a typographical issue in the author list ('Rui Y ang' with an extra space); this should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the benchmark is an empirical comparison; the overstated 'consistent improvements' claim and unstated missing-modality cohorts are correctness/reproducibility issues, not circular reasoning.

full rationale

This paper contains no derivation chain that could reduce to its inputs. It benchmarks fixed, externally pretrained foundation models on MIMIC-IV, extracts embeddings, concatenates them, and fits a logistic-regression head. The 'improvement' from adding modalities is an empirical AUROC comparison, not a quantity that is defined in terms of the claim: no parameter is fitted to the target metric, no result is imported from the authors' prior work as a forced premise, and no ansatz is smuggled in via citation. The self-citations (e.g., refs. 6, 18, 20, 31) are background or methodology references and are not load-bearing. The abstract's 'consistent improvements' is contradicted by Table II (mortality AUROC drops from 0.8943 with text alone to 0.8846 with CXR-Foundation plus Text-Embedding-3-Large), and the paper itself concedes 'a modest decline in performance was observed when both image and text modalities were combined, compared to using text alone.' This is an overstatement or internal inconsistency, but it is not circular. Likewise, the unstated handling of patients without CXRs or notes could confound the comparison across modality combinations, and pretraining CXR-Foundation on similar chest X-ray data could inflate its scores, but both are data-validity concerns, not definitional reductions. No circular step can be exhibited, so the appropriate finding is no significant circularity (0).

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claims rest on standard statistical assumptions and domain assumptions about MIMIC-IV data and the validity of frozen embeddings as task probes. No new entities, forces, or fitted constants are introduced; the only model parameters are those learned during pretraining or in the GRU, which are not free parameters of the benchmark claim.

assumptions (4)
  • domain assumption MIMIC-IV v2.2 data are accurate and complete enough for benchmarking, and the labels for in-hospital mortality and length-of-stay over 3 days are reliable.
    Used throughout the study; if label errors or documentation inconsistencies exist, all metric comparisons inherit them. The authors apply outlier removal and imputation but do not validate label accuracy.
  • domain assumption Frozen embeddings from each foundation model, when concatenated and passed to a logistic regression classifier, capture clinically relevant information in a way that permits fair comparison across models.
    Section II-C states that embeddings are extracted independently and then used as input to logistic regression; this assumes the linear probe is a valid measure of representation quality for all models.
  • standard math Bootstrap 95% confidence intervals provide correct coverage for comparing AUROC and AUPRC differences.
    Section II-E uses 1,000 bootstrap samples; this assumes the resampling distribution approximates the sampling distribution of the metrics.
  • domain assumption The GPT-4o-designed prompt template and the closed-ended yes/no format are a valid and unbiased way to elicit clinical predictions from LVLMs.
    Section II-D describes the prompt format; if the format or the table-to-text conversion biases outputs, the LVLM performance and fairness results are not generalizable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Benchmarking Foundation Models with Multimodal Public Electronic Health Records." pith.science (2026). https://pith.science/paper/W5VOXJOF

@misc{pith2026250714824,
  author       = {Pith},
  title        = {Pith review of: Benchmarking Foundation Models with Multimodal Public Electronic Health Records},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W5VOXJOF}},
  note         = {Machine review of arXiv:2507.14824}
}
read the original abstract

Foundation models have emerged as a powerful approach for processing electronic health records (EHRs), offering flexibility to handle diverse medical data modalities. In this study, we present a comprehensive benchmark that evaluates the performance, fairness, and interpretability of foundation models, both as unimodal encoders and as multimodal learners, using the publicly available MIMIC-IV database. To support consistent and reproducible evaluation, we developed a standardized data processing pipeline that harmonizes heterogeneous clinical records into an analysis-ready format. We systematically compared eight foundation models, encompassing both unimodal and multimodal models, as well as domain-specific and general-purpose variants. Our findings demonstrate that incorporating multiple data modalities leads to consistent improvements in predictive performance without introducing additional bias. Through this benchmark, we aim to support the development of effective and trustworthy multimodal artificial intelligence (AI) systems for real-world clinical applications. Our code is available at https://github.com/nliulab/MIMIC-Multimodal.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

52 extracted references · 35 canonical work pages

  1. [1]

    Harnessing ehr data for health research,

    A. S. Tang, S. R. Woldemariam, S. Miramontes, B. Norgeot, T. T. Oskotsky, and M. Sirota, “Harnessing ehr data for health research,” Nature Medicine, vol. 30, no. 7, pp. 1847–1855, 2024

    work page 2024

  2. [2]

    Personalized medicine and the power of electronic health records,

    N. S. Abul-Husn and E. E. Kenny, “Personalized medicine and the power of electronic health records,” Cell, vol. 177, no. 1, pp. 58–69, 2019

    work page 2019

  3. [3]

    Mining electronic health records: towards better research applications and clinical care,

    P. B. Jensen, L. J. Jensen, and S. Brunak, “Mining electronic health records: towards better research applications and clinical care,” Nature Reviews Genetics, vol. 13, no. 6, pp. 395–405, 2012

    work page 2012

  4. [4]

    A guide to sharing open healthcare data under the general data protection regulation,

    J. W. de Kok, M. ´A. A. de la Hoz, Y . de Jong, V . Brokke, P. W. Elbers, P. Thoral, A. Castillejo, T. Trenor, J. M. Castellano, A. E. Bronchalo et al., “A guide to sharing open healthcare data under the general data protection regulation,” Scientific data, vol. 10, no. 1, p. 404, 2023

    work page 2023

  5. [5]

    Mimic- iv, a freely accessible electronic health record dataset,

    A. E. Johnson, L. Bulgarelli, L. Shen, A. Gayles, A. Shammout, S. Horng, T. J. Pollard, S. Hao, B. Moody, B. Gow et al. , “Mimic- iv, a freely accessible electronic health record dataset,” Scientific data , vol. 10, no. 1, p. 1, 2023

    work page 2023

  6. [6]

    Comparing open-access database and traditional intensive care studies using machine learning: bibliometric analysis study,

    Y . Ke, R. Yang, and N. Liu, “Comparing open-access database and traditional intensive care studies using machine learning: bibliometric analysis study,”Journal of Medical Internet Research, vol. 26, p. e48330, 2024

    work page 2024

  7. [7]

    Multimodal biomedical ai,

    J. N. Acosta, G. J. Falcone, P. Rajpurkar, and E. J. Topol, “Multimodal biomedical ai,” Nature medicine, vol. 28, no. 9, pp. 1773–1784, 2022

    work page 2022

  8. [8]

    Foundation models for generalist medical artificial intelligence,

    M. Moor, O. Banerjee, Z. S. H. Abad, H. M. Krumholz, J. Leskovec, E. J. Topol, and P. Rajpurkar, “Foundation models for generalist medical artificial intelligence,” Nature, vol. 616, no. 7956, pp. 259–265, 2023

    2023

Show all 52 references
  1. [9]

    Towards generalist biomedical ai,

    T. Tu, S. Azizi, D. Driess, M. Schaekermann, M. Amin, P.-C. Chang, A. Carroll, C. Lau, R. Tanno, I. Ktena et al. , “Towards generalist biomedical ai,” Nejm Ai, vol. 1, no. 3, p. AIoa2300138, 2024

    2024

  2. [10]

    On the opportunities and risks of foundation models,

    R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill et al. , “On the opportunities and risks of foundation models,” arXiv preprint arXiv:2108.07258, 2021

    2021 arXiv

  3. [11]

    Multimodal data matters: Language model pre-training over structured and unstructured electronic health records,

    S. Liu, X. Wang, Y . Hou, G. Li, H. Wang, H. Xu, Y . Xiang, and B. Tang, “Multimodal data matters: Language model pre-training over structured and unstructured electronic health records,” IEEE Journal of Biomedical and Health Informatics , vol. 27, no. 1, pp. 504–514, 2022

    2022

  4. [12]

    Multimodal foundation models: From specialists to general-purpose assistants,

    C. Li, Z. Gan, Z. Yang, J. Yang, L. Li, L. Wang, J. Gao et al. , “Multimodal foundation models: From specialists to general-purpose assistants,” Foundations and Trends® in Computer Graphics and Vision, vol. 16, no. 1-2, pp. 1–214, 2024

    2024

  5. [13]

    Foundation model for advancing healthcare: challenges, opportunities and future directions,

    Y . He, F. Huang, X. Jiang, Y . Nie, M. Wang, J. Wang, and H. Chen, “Foundation model for advancing healthcare: challenges, opportunities and future directions,” IEEE Reviews in Biomedical Engineering , 2024

    2024

  6. [14]

    A framework for evaluating the efficacy of foundation embedding models in healthcare,

    S. Xu, H. Gui, V . Rotemberg, T. Wang, Y . T. Chen, and R. Daneshjou, “A framework for evaluating the efficacy of foundation embedding models in healthcare,” medRxiv, pp. 2024–04, 2024

    2024

  7. [15]

    The application of multimodal large language models in medicine,

    J. Qiu, W. Yuan, and K. Lam, “The application of multimodal large language models in medicine,” The Lancet Regional Health–Western Pacific, vol. 45, 2024

    2024

  8. [16]

    Large language models encode clinical knowledge,

    K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl et al., “Large language models encode clinical knowledge,” Nature, vol. 620, no. 7972, pp. 172– 180, 2023

    2023

  9. [17]

    Capabilities of gemini models in medicine,

    K. Saab, T. Tu, W.-H. Weng, R. Tanno, D. Stutz, E. Wulczyn, F. Zhang, T. Strother, C. Park, E. Vedadi et al., “Capabilities of gemini models in medicine,” arXiv preprint arXiv:2404.18416 , 2024

    2024 arXiv

  10. [18]

    Large language models in health care: Development, 10 IEEE TRANSACTIONS AND JOURNALS TEMPLATE applications, and challenges,

    R. Yang, T. F. Tan, W. Lu, A. J. Thirunavukarasu, D. S. W. Ting, and N. Liu, “Large language models in health care: Development, 10 IEEE TRANSACTIONS AND JOURNALS TEMPLATE applications, and challenges,” Health Care Science , vol. 2, no. 4, pp. 255–263, 2023

    2023

  11. [19]

    Multimodal large language models in health care: applications, challenges, and future outlook,

    R. AlSaad, A. Abd-Alrazaq, S. Boughorbel, A. Ahmed, M.-A. Renault, R. Damseh, and J. Sheikh, “Multimodal large language models in health care: applications, challenges, and future outlook,” Journal of medical Internet research, vol. 26, p. e59505, 2024

    2024

  12. [20]

    Disparities in clinical studies of ai enabled applications from a global perspective,

    R. Yang, S. V . Nair, Y . Ke, D. D’Agostino, M. Liu, Y . Ning, and N. Liu, “Disparities in clinical studies of ai enabled applications from a global perspective,” NPJ digital medicine , vol. 7, no. 1, p. 209, 2024

    2024

  13. [21]

    The shaky foundations of large language models and foundation models for electronic health records,

    M. Wornow, Y . Xu, R. Thapa, B. Patel, E. Steinberg, S. Fleming, M. A. Pfeffer, J. Fries, and N. H. Shah, “The shaky foundations of large language models and foundation models for electronic health records,” npj digital medicine , vol. 6, no. 1, p. 135, 2023

    2023

  14. [22]

    Gmai-mmbench: A comprehensive multimodal evaluation benchmark towards general medical ai,

    J. Ye, G. Wang, Y . Li, Z. Deng, W. Li, T. Li, H. Duan, Z. Huang, Y . Su, B. Wang et al., “Gmai-mmbench: A comprehensive multimodal evaluation benchmark towards general medical ai,” Advances in Neural Information Processing Systems , vol. 37, pp. 94 327–94 427, 2024

    2024

  15. [23]

    Multimodal clinical benchmark for emergency care (mc- bec): A comprehensive benchmark for evaluating foundation models in emergency medicine,

    E. Chen, A. Kansal, J. Chen, B. T. Jin, J. Reisler, D. E. Kim, and P. Rajpurkar, “Multimodal clinical benchmark for emergency care (mc- bec): A comprehensive benchmark for evaluating foundation models in emergency medicine,” Advances in Neural Information Processing Systems, v...

    2023

  16. [24]

    Integrated multimodal artificial intelligence framework for healthcare applications,

    L. R. Soenksen, Y . Ma, C. Zeng, L. Boussioux, K. Villalobos Carballo, L. Na, H. M. Wiberg, M. L. Li, I. Fuentes, and D. Bertsimas, “Integrated multimodal artificial intelligence framework for healthcare applications,” NPJ digital medicine , vol. 5, no. 1, p. 149, 2022

    2022

  17. [25]

    A multimodal integration pipeline for accurate diagnosis, pathogen identification, and prognosis prediction of pulmonary infections,

    J. Shao, J. Ma, Y . Yu, S. Zhang, W. Wang, W. Li, and C. Wang, “A multimodal integration pipeline for accurate diagnosis, pathogen identification, and prognosis prediction of pulmonary infections,” The Innovation, vol. 5, no. 4, 2024

    2024

  18. [26]

    Learning phrase representations using rnn encoder-decoder for statistical machine translation,

    K. Cho, B. Van Merri ¨enboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y . Bengio, “Learning phrase representations using rnn encoder-decoder for statistical machine translation,” arXiv preprint arXiv:1406.1078, 2014

    2014 arXiv

  19. [27]

    Moment: A family of open time-series foundation models,

    M. Goswami, K. Szafer, A. Choudhry, Y . Cai, S. Li, and A. Dubrawski, “Moment: A family of open time-series foundation models,” arXiv preprint arXiv:2402.03885, 2024

    2024 arXiv

  20. [28]

    Elixr: Towards a general purpose x-ray artificial intelligence system through alignment of large language models and radiology vision encoders,

    S. Xu, L. Yang, C. Kelly, M. Sieniek, T. Kohlberger, M. Ma, W.-H. Weng, A. Kiraly, S. Kazemzadeh, Z. Melamed et al. , “Elixr: Towards a general purpose x-ray artificial intelligence system through alignment of large language models and radiology vision encoders,” arXiv preprin...

    2023 arXiv

  21. [29]

    Swin transformer: Hierarchical vision transformer using shifted windows,

    Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 10 012–10 022

    2021

  22. [30]

    Improved fine-tuning of in-domain transformer model for inferring covid-19 presence in multi- institutional radiology reports,

    P. Chambon, T. S. Cook, and C. P. Langlotz, “Improved fine-tuning of in-domain transformer model for inferring covid-19 presence in multi- institutional radiology reports,” Journal of Digital Imaging , vol. 36, no. 1, pp. 164–177, 2023

    2023

  23. [31]

    Benchmarking emergency department prediction models with machine learning and public electronic health records,

    F. Xie, J. Zhou, J. W. Lee, M. Tan, S. Li, L. S. Rajnthern, M. L. Chee, B. Chakraborty, A.-K. I. Wong, A. Dagan et al., “Benchmarking emergency department prediction models with machine learning and public electronic health records,” Scientific Data, vol. 9, no. 1, p. 658, 2022

    2022

  24. [32]

    Llava-med: Training a large language-and-vision assistant for biomedicine in one day,

    C. Li, C. Wong, S. Zhang, N. Usuyama, H. Liu, J. Yang, T. Naumann, H. Poon, and J. Gao, “Llava-med: Training a large language-and-vision assistant for biomedicine in one day,” Advances in Neural Information Processing Systems, vol. 36, pp. 28 541–28 564, 2023

    2023

  25. [33]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” Advances in neural information processing systems, vol. 36, pp. 34 892– 34 916, 2023

    2023

  26. [34]

    A unified approach to interpreting model predictions,

    S. M. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” Advances in neural information processing systems, vol. 30, 2017

    2017

  27. [35]

    “everyone wants to do the model work, not the data work

    N. Sambasivan, S. Kapania, H. Highfill, D. Akrong, P. Paritosh, and L. M. Aroyo, ““everyone wants to do the model work, not the data work”: Data cascades in high-stakes ai,” in proceedings of the 2021 CHI Conference on Human Factors in Computing Systems , 2021, pp. 1–15

    2021

  28. [36]

    An ethics assessment tool for artificial intelligence implementation in healthcare: Care-ai,

    Y . Ning, X. Liu, G. S. Collins, K. G. Moons, M. McCradden, D. S. W. Ting, J. C. L. Ong, B. A. Goldstein, S. K. Wagner, P. A. Keane et al., “An ethics assessment tool for artificial intelligence implementation in healthcare: Care-ai,” Nature medicine, pp. 1–2, 2024

    2024

  29. [37]

    As artificial intelligence goes multimodal, medical appli- cations multiply,

    E. J. Topol, “As artificial intelligence goes multimodal, medical appli- cations multiply,” Science, vol. 381, no. 6663, p. eadk6139, 2023

    2023

  30. [38]

    Ascle—a python natural language processing toolkit for medical text generation: development and evaluation study,

    R. Yang, Q. Zeng, K. You, Y . Qiao, L. Huang, C.-C. Hsieh, B. Rosand, J. Goldwasser, A. Dave, T. Keenan et al. , “Ascle—a python natural language processing toolkit for medical text generation: development and evaluation study,” Journal of Medical Internet Research , vol. 26, ...

    2024

  31. [39]

    Are time series foundation models ready for vital sign forecasting in healthcare?

    X. Gu, Y . Liu, Z. Mohsin, J. Bedford, A. Thakur, P. Watkinson, L. Clifton, T. Zhu, and D. Clifton, “Are time series foundation models ready for vital sign forecasting in healthcare?” in Machine Learning for Health (ML4H). PMLR, 2025, pp. 401–419

    2025

  32. [40]

    Enhancing medical coding efficiency through domain-specific fine-tuned large language models,

    Z. Hou, H. Liu, J. Bian, X. He, and Y . Zhuang, “Enhancing medical coding efficiency through domain-specific fine-tuned large language models,” npj Health Systems , vol. 2, no. 1, p. 14, 2025

    2025

  33. [41]

    Artificial intelligence and multimodal data fusion for smart healthcare: topic modeling and bibliometrics,

    X. Chen, H. Xie, X. Tao, F. L. Wang, M. Leng, and B. Lei, “Artificial intelligence and multimodal data fusion for smart healthcare: topic modeling and bibliometrics,” Artificial Intelligence Review , vol. 57, no. 4, p. 91, 2024

    2024

  34. [42]

    Visual–language foundation models in medicine,

    C. Liu, Y . Jin, Z. Guan, T. Li, Y . Qin, B. Qian, Z. Jiang, Y . Wu, X. Wang, Y . F. Zheng et al. , “Visual–language foundation models in medicine,” The Visual Computer , pp. 1–20, 2024

    2024

  35. [43]

    Longhealth: A question answering benchmark with long clinical documents,

    L. Adams, F. Busch, T. Han, J.-B. Excoffier, M. Ortala, A. L ¨oser, H. J. Aerts, J. N. Kather, D. Truhn, and K. Bressem, “Longhealth: A question answering benchmark with long clinical documents,” arXiv preprint arXiv:2401.14490, 2024

    2024 arXiv

  36. [44]

    Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical lvlm,

    Y . Hu, T. Li, Q. Lu, W. Shao, J. He, Y . Qiao, and P. Luo, “Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical lvlm,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 22 170–22 183

    2024

  37. [45]

    Generative artificial intelligence and ethical considerations in health care: a scoping review and ethics checklist,

    Y . Ning, S. Teixayavong, Y . Shang, J. Savulescu, V . Nagaraj, D. Miao, M. Mertens, D. S. W. Ting, J. C. L. Ong, M. Liu et al. , “Generative artificial intelligence and ethical considerations in health care: a scoping review and ethics checklist,” The Lancet Digital Health , ...

    2024

  38. [46]

    Cares: A comprehensive benchmark of trustworthiness in medical vision language models,

    P. Xia, Z. Chen, J. Tian, Y . Gong, R. Hou, Y . Xu, Z. Wu, Z. Fan, Y . Zhou, K. Zhu et al. , “Cares: A comprehensive benchmark of trustworthiness in medical vision language models,” Advances in Neural Information Processing Systems, vol. 37, pp. 140 334–140 365, 2024

    2024

  39. [47]

    Radfusion: Bench- marking performance and fairness for multimodal pulmonary embolism detection from ct and ehr,

    Y . Zhou, S.-C. Huang, J. A. Fries, A. Youssef, T. J. Amrhein, M. Chang, I. Banerjee, D. Rubin, L. Xing, N. Shah et al. , “Radfusion: Bench- marking performance and fairness for multimodal pulmonary embolism detection from ct and ehr,” arXiv preprint arXiv:2111.11665 , 2021

    2021 arXiv

  40. [48]

    Learn- ing missing modal electronic health records with unified multi-modal data embedding and modality-aware attention,

    K. Lee, S. Lee, S. Hahn, H. Hyun, E. Choi, B. Ahn, and J. Lee, “Learn- ing missing modal electronic health records with unified multi-modal data embedding and modality-aware attention,” in Machine Learning for Healthcare Conference . PMLR, 2023, pp. 423–442

    2023

  41. [49]

    M3care: Learning with missing modalities in multimodal healthcare data,

    C. Zhang, X. Chu, L. Ma, Y . Zhu, Y . Wang, J. Wang, and J. Zhao, “M3care: Learning with missing modalities in multimodal healthcare data,” in Proceedings of the 28th ACM SIGKDD conference on knowl- edge discovery and data mining , 2022, pp. 2418–2428

    2022

  42. [50]

    Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: a review,

    C. Cui, H. Yang, Y . Wang, S. Zhao, Z. Asad, L. A. Coburn, K. T. Wilson, B. A. Landman, and Y . Huo, “Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: a review,” Progress in Biomedical Engineering , vol. 5, no. 2, p. 022001, 2023

    2023

  43. [51]

    Rethinking interpretability in the era of large language models,

    C. Singh, J. P. Inala, M. Galley, R. Caruana, and J. Gao, “Rethinking interpretability in the era of large language models,” arXiv preprint arXiv:2402.01761, 2024

    2024 arXiv

  44. [52]

    Explainable and interpretable multimodal large language models: A comprehensive survey,

    Y . Dang, K. Huang, J. Huo, Y . Yan, S. Huang, D. Liu, M. Gao, J. Zhang, C. Qian, K. Wang et al. , “Explainable and interpretable multimodal large language models: A comprehensive survey,” arXiv preprint arXiv:2412.02104, 2024

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.